OpenRaiser/Intern-Atlas huggingface dataset:
Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics
Venue: arXiv cs.LG 2020
Year: 2020

A central challenge in developing versatile machine learning systems is catastrophic forgetting: a model trained on tasks in sequence will suffer significant performance drops on earlier tasks. Despite the ubiquity of catastrophic forgetting, there is limited understanding of the underlying process and its causes. In this paper, we address this important knowledge gap, investigating how forgetting affects representations in neural network models. Through representational analysis techniques, we find that deeper layers are disproportionately the source of forgetting. Supporting this, a study of methods to mitigate forgetting illustrates that they act to stabilize deeper layers. These insights enable the development of an analytic argument and empirical picture relating the degree of forgetting to representational similarity between tasks. Consistent with this picture, we observe maximal forgetting occurs for task sequences with intermediate similarity. We perform empirical studies on the standard split CIFAR-10 setup and also introduce a novel CIFAR-100 based task approximating realistic input distribution shift.
Widest Paths and Global Propagation in Bounded Value Iteration for Stochastic Games
Venue: arXiv cs.LO 2020
Year: 2020

Solving stochastic games with the reachability objective is a fundamental problem, especially in quantitative verification and synthesis. For this purpose, bounded value iteration (BVI) attracts attention as an efficient iterative method. However, BVI's performance is often impeded by costly end component (EC) computation that is needed to ensure convergence. Our contribution is a novel BVI algorithm that conducts, in addition to local propagation by the Bellman update that is typical of BVI, global propagation of upper bounds that is not hindered by ECs. To conduct global propagation in a computationally tractable manner, we construct a weighted graph and solve the widest path problem in it. Our experiments show the algorithm's performance advantage over the previous BVI algorithms that rely on EC computation.
Content-based Recommendations for Radio Stations with Deep Learned Audio Fingerprints
Venue: arXiv cs.HC 2020
Year: 2020

The world of linear radio broadcasting is characterized by a wide variety of stations and played content. That is why finding stations playing the preferred content is a tough task for a potential listener, especially due to the overwhelming number of offered choices. Here, recommender systems usually step in but existing content-based approaches rely on metadata and thus are constrained by the available data quality. Other approaches leverage user behavior data and thus do not exploit any domain-specific knowledge and are furthermore disadvantageous regarding privacy concerns. Therefore, we propose a new pipeline for the generation of audio-based radio station fingerprints relying on audio stream crawling and a Deep Autoencoder. We show that the proposed fingerprints are especially useful for characterizing radio stations by their audio content and thus are an excellent representation for meaningful and reliable radio station recommendations. Furthermore, the proposed modules are part of the HRADIO Communication Platform, which enables hybrid radio features to radio stations. It is released with a flexible open source license and enables especially small- and medium-sized businesses, to provide customized and high quality radio services to potential listeners.
Coding Theorems for Noisy Permutation Channels
Venue: arXiv cs.IT 2020
Year: 2020

In this paper, we formally define and analyze the class of noisy permutation channels. The noisy permutation channel model constitutes a standard discrete memoryless channel (DMC) followed by an independent random permutation that reorders the output codeword of the DMC. While coding theoretic aspects of this model have been studied extensively, particularly in the context of reliable communication in network settings where packets undergo transpositions, and closely related models of DNA based storage systems have also been analyzed recently, we initiate an information theoretic study of this model by defining an appropriate notion of noisy permutation channel capacity. Specifically, on the achievability front, we prove a lower bound on the noisy permutation channel capacity of any DMC in terms of the rank of the stochastic matrix of the DMC. On the converse front, we establish two upper bounds on the noisy permutation channel capacity of any DMC whose stochastic matrix is strictly positive (entry-wise). Together, these bounds yield coding theorems that characterize the noisy permutation channel capacities of every strictly positive and "full rank" DMC, and our achievability proof yields a conceptually simple, computationally efficient, and capacity achieving coding scheme for such DMCs. Furthermore, we also demonstrate the relation between the output degradation preorder over channels and noisy permutation channel capacity. In fact, the proof of one of our converse bounds exploits a degradation result that constructs a symmetric channel for any DMC such that the DMC is a degraded version of the symmetric channel. Finally, we illustrate some examples such as the special cases of binary symmetric channels and (general) erasure channels. Somewhat surprisingly, our results suggest that noisy permutation channel capacities are generally quite agnostic to the parameters that define the DMCs.
On the Inclusion of Spatial Information for Spatio-Temporal Neural Networks
Venue: arXiv cs.LG 2020
Year: 2020

When confronting a spatio-temporal regression, it is sensible to feed the model with any available prior information about the spatial dimension. For example, it is common to define the architecture of neural networks based on spatial closeness, adjacency, or correlation. A common alternative, if spatial information is not available or is too costly to introduce it in the model, is to learn it as an extra step of the model. While the use of prior spatial knowledge, given or learnt, might be beneficial, in this work we question this principle by comparing spatial agnostic neural networks with state of the art models. Our results show that the typical inclusion of prior spatial information is not really needed in most cases. In order to validate this counterintuitive result, we perform thorough experiments over ten different datasets related to sustainable mobility and air quality, substantiating our conclusions on real world problems with direct implications for public health and economy.
Predicting Clinical Diagnosis from Patients Electronic Health Records Using BERT-based Neural Networks
Venue: arXiv cs.CL 2020
Year: 2020

In this paper we study the problem of predicting clinical diagnoses from textual Electronic Health Records (EHR) data. We show the importance of this problem in medical community and present comprehensive historical review of the problem and proposed methods. As the main scientific contributions we present a modification of Bidirectional Encoder Representations from Transformers (BERT) model for sequence classification that implements a novel way of Fully-Connected (FC) layer composition and a BERT model pretrained only on domain data. To empirically validate our model, we use a large-scale Russian EHR dataset consisting of about 4 million unique patient visits. This is the largest such study for the Russian language and one of the largest globally. We performed a number of comparative experiments with other text representation models on the task of multiclass classification for 265 disease subset of ICD-10. The experiments demonstrate improved performance of our models compared to other baselines, including a fine-tuned Russian BERT (RuBERT) variant. We also show comparable performance of our model with a panel of experienced medical experts. This allows us to hope that implementation of this system will reduce misdiagnosis.
Partially Observable Concurrent Kleene Algebra
Venue: arXiv cs.LO 2020
Year: 2020

We introduce partially observable concurrent Kleene algebra (POCKA), an algebraic framework to reason about concurrent programs with control structures, such as conditionals and loops. POCKA enables reasoning about programs that can access variables and values, which we illustrate through concrete examples. We prove that POCKA is a sound and complete axiomatisation of a model of partial observations, and show the semantics passes an important check for sequential consistency.
Non-Relational Databases on FPGAs: Survey, Design Decisions, Challenges
Venue: arXiv cs.DB 2020
Year: 2020

Non-relational database systems (NRDS), such as graph, document, key-value, and wide-column, have gained much attention in various trending (business) application domains like smart logistics, social network analysis, and medical applications, due to their data model variety and scalability. The broad data variety and sheer size of datasets pose unique challenges for the system design and runtime (incl. power consumption). While CPU performance scaling becomes increasingly more difficult, we argue that NRDS can benefit from adding field programmable gate arrays (FPGAs) as accelerators. However, FPGA-accelerated NRDS have not been systematically studied, yet. To facilitate understanding of this emerging domain, we explore the fit of FPGA acceleration for NRDS with a focus on data model variety. We define the term NRDS class as a group of non-relational database systems supporting the same data model. This survey describes and categorizes the inherent differences and non-trivial trade-offs of relevant NRDS classes as well as their commonalities in the context of common design decisions when building such a system with FPGAs. For example, we found in the literature that for key-value stores the FPGA should be placed into the system as a smart network interface card (SmartNIC) to benefit from direct access of the FPGA to the network. However, more complex data models and processing of other classes (e.g., graph and document) commonly require more elaborate near-data or socket accelerator placements where the FPGA respectively has the only or shared access to main memory. Across the different classes, FPGAs can be used as communication layer or for acceleration of operators and data access. We close with open research and engineering challenges to outline the future of FPGA-accelerated NRDS.
Augmented Bi-path Network for Few-shot Learning
Venue: arXiv cs.CV 2020
Year: 2020

Few-shot Learning (FSL) which aims to learn from few labeled training data is becoming a popular research topic, due to the expensive labeling cost in many real-world applications. One kind of successful FSL method learns to compare the testing (query) image and training (support) image by simply concatenating the features of two images and feeding it into the neural network. However, with few labeled data in each class, the neural network has difficulty in learning or comparing the local features of two images. Such simple image-level comparison may cause serious mis-classification. To solve this problem, we propose Augmented Bi-path Network (ABNet) for learning to compare both global and local features on multi-scales. Specifically, the salient patches are extracted and embedded as the local features for every image. Then, the model learns to augment the features for better robustness. Finally, the model learns to compare global and local features separately, i.e., in two paths, before merging the similarities. Extensive experiments show that the proposed ABNet outperforms the state-of-the-art methods. Both quantitative and visual ablation studies are provided to verify that the proposed modules lead to more precise comparison results.
Leafy Spanning Arborescences in DAGs
Venue: arXiv cs.DS 2020
Year: 2020

Broadcasting in a computer network is a method of transferring a message to all recipients simultaneously. It is common in this situation to use a tree with many leaves to perform the broadcast, as internal nodes have to forward the messages received, while leaves are only receptors. We consider the subjacent problem of, given a directed graph~$D$, finding a spanning arborescence of D, if one exists, with the maximum number of leaves. In this paper, we concentrate on the class of rooted directed acyclic graphs, for which the problem is known to be MaxSNP-hard. A 2-approximation was previously known for this problem on this class of directed graphs. We improve on this result, presenting a (3/2)-approximation. We also adapt a result for the undirected case and derive an inapproximability result for the vertex-weighted version of Maximum Leaf Spanning Arborescence on rooted directed acyclic graphs.
End-to-end training of a two-stage neural network for defect detection
Venue: arXiv cs.CV 2020
Year: 2020

Segmentation-based, two-stage neural network has shown excellent results in the surface defect detection, enabling the network to learn from a relatively small number of samples. In this work, we introduce end-to-end training of the two-stage network together with several extensions to the training process, which reduce the amount of training time and improve the results on the surface defect detection tasks. To enable end-to-end training we carefully balance the contributions of both the segmentation and the classification loss throughout the learning. We adjust the gradient flow from the classification into the segmentation network in order to prevent the unstable features from corrupting the learning. As an additional extension to the learning, we propose frequency-of-use sampling scheme of negative samples to address the issue of over- and under-sampling of images during the training, while we employ the distance transform algorithm on the region-based segmentation masks as weights for positive pixels, giving greater importance to areas with higher probability of presence of defect without requiring a detailed annotation. We demonstrate the performance of the end-to-end training scheme and the proposed extensions on three defect detection datasets - DAGM, KolektorSDD and Severstal Steel defect dataset - where we show state-of-the-art results. On the DAGM and the KolektorSDD we demonstrate 100\% detection rate, therefore completely solving the datasets. Additional ablation study performed on all three datasets quantitatively demonstrates the contribution to the overall result improvements for each of the proposed extensions.
Are Hyperbolic Representations in Graphs Created Equal?
Venue: arXiv cs.LG 2020
Year: 2020

Recently there was an increasing interest in applications of graph neural networks in non-Euclidean geometry; however, are non-Euclidean representations always useful for graph learning tasks? For different problems such as node classification and link prediction we compute hyperbolic embeddings and conclude that for tasks that require global prediction consistency it might be useful to use non-Euclidean embeddings, while for other tasks Euclidean models are superior. To do so we first fix an issue of the existing models associated with the optimization process at zero curvature. Current hyperbolic models deal with gradients at the origin in ad-hoc manner, which is inefficient and can lead to numerical instabilities. We solve the instabilities of kappa-Stereographic model at zero curvature cases and evaluate the approach of embedding graphs into the manifold in several graph representation learning tasks.
Lunar Terrain Relative Navigation Using a Convolutional Neural Network for Visual Crater Detection
Venue: arXiv cs.CV 2020
Year: 2020

Terrain relative navigation can improve the precision of a spacecraft's position estimate by detecting global features that act as supplementary measurements to correct for drift in the inertial navigation system. This paper presents a system that uses a convolutional neural network (CNN) and image processing methods to track the location of a simulated spacecraft with an extended Kalman filter (EKF). The CNN, called LunaNet, visually detects craters in the simulated camera frame and those detections are matched to known lunar craters in the region of the current estimated spacecraft position. These matched craters are treated as features that are tracked using the EKF. LunaNet enables more reliable position tracking over a simulated trajectory due to its greater robustness to changes in image brightness and more repeatable crater detections from frame to frame throughout a trajectory. LunaNet combined with an EKF produces a decrease of 60% in the average final position estimation error and a decrease of 25% in average final velocity estimation error compared to an EKF using an image processing-based crater detection method when tested on trajectories using images of standard brightness.
NERD: Neural Network for Edict of Risky Data Streams
Venue: arXiv cs.CY 2020
Year: 2020

Cyber incidents can have a wide range of cause from a simple connection loss to an insistent attack. Once a potential cyber security incidents and system failures have been identified, deciding how to proceed is often complex. Especially, if the real cause is not directly in detail determinable. Therefore, we developed the concept of a Cyber Incident Handling Support System. The developed system is enriched with information by multiple sources such as intrusion detection systems and monitoring tools. It uses over twenty key attributes like sync-package ratio to identify potential security incidents and to classify the data into different priority categories. Afterwards, the system uses artificial intelligence to support the further decision-making process and to generate corresponding reports to brief the Board of Directors. Originating from this information, appropriate and detailed suggestions are made regarding the causes and troubleshooting measures. Feedback from users regarding the problem solutions are included into future decision-making by using labelled flow data as input for the learning process. The prototype shows that the decision making can be sustainably improved and the Cyber Incident Handling process becomes much more effective.
Enabling Mixed-Precision Quantized Neural Networks in Extreme-Edge Devices
Venue: arXiv cs.AR 2020
Year: 2020

The deployment of Quantized Neural Networks (QNN) on advanced microcontrollers requires optimized software to exploit digital signal processing (DSP) extensions of modern instruction set architectures (ISA). As such, recent research proposed optimized libraries for QNNs (from 8-bit to 2-bit) such as CMSIS-NN and PULP-NN. This work presents an extension to the PULP-NN library targeting the acceleration of mixed-precision Deep Neural Networks, an emerging paradigm able to significantly shrink the memory footprint of deep neural networks with negligible accuracy loss. The library, composed of 27 kernels, one for each permutation of input feature maps, weights, and output feature maps precision (considering 8-bit, 4-bit and 2-bit), enables efficient inference of QNN on parallel ultra-low-power (PULP) clusters of RISC-V based processors, featuring the RV32IMCXpulpV2 ISA. The proposed solution, benchmarked on an 8-cores GAP-8 PULP cluster, reaches peak performance of 16 MACs/cycle on 8 cores, performing 21x to 25x faster than an STM32H7 (powered by an ARM Cortex M7 processor) with 15x to 21x better energy efficiency.
NDNTP: A Named Data Networking Time Protocol
Venue: arXiv cs.NI 2020
Year: 2020

Named Data Networking (NDN) architectural features, including multicast data delivery, stateful forwarding, and in-network data caching, have shown promise for applications such as video streaming and file sharing. However, collaborative applications, requiring a multi-producer participation introduce new NDN design challenges. In this paper, we highlight these challenges in the context of the Network Time Protocol (NTP) and one of its most widely-used deployments for NTP server discovery, the NTP pool project. We discuss the design requirements for the support of NTP and NTP pool and present general directions for the design of a time synchronization protocol over NDN, coined Named Data Networking Time Protocol (NDNTP).
Continuous Prefetch for Interactive Data Applications
Venue: arXiv cs.DB 2020
Year: 2020

Interactive data visualization and exploration (DVE) applications are often network-bottlenecked due to bursty request patterns, large response sizes, and heterogeneous deployments over a range of networks and devices. This makes it difficult to ensure consistently low response times (< 100ms). Khameleon is a framework for DVE applications that uses a novel combination of prefetching and response tuning to dynamically trade-off response quality for low latency. Khameleon exploits DVE's approximation tolerance: immediate lower-quality responses are preferable to waiting for complete results. To this end, Khameleon progressively encodes responses, and runs a server-side scheduler that proactively streams portions of responses using available bandwidth to maximize user's perceived interactivity. The scheduler involves a complex optimization based on available resources, predicted user interactions, and response quality levels; yet, decisions must also be real-time. To overcome this, Khameleon uses a fast greedy approximation which closely mimics the optimal approach. Using image exploration and visualization applications with real user interaction traces, we show that across a wide range of network and client resource conditions, Khameleon outperforms classic prefetching approaches that benefit from perfect prediction models: response latencies with Khameleon are never higher, and typically between 2 to 3 orders of magnitude lower while response quality remains within 50%-80%.
Sinhala Language Corpora and Stopwords from a Decade of Sri Lankan Facebook
Venue: arXiv cs.CL 2020
Year: 2020

This paper presents two colloquial Sinhala language corpora from the language efforts of the Data, Analysis and Policy team of LIRNEasia, as well as a list of algorithmically derived stopwords. The larger of the two corpora spans 2010 to 2020 and contains 28,825,820 to 29,549,672 words of multilingual text posted by 533 Sri Lankan Facebook pages, including politics, media, celebrities, and other categories; the smaller corpus amounts to 5,402,76 words of only Sinhala text extracted from the larger. Both corpora have markers for their date of creation, page of origin, and content type.
Compression strategies and space-conscious representations for deep neural networks
Venue: arXiv cs.LG 2020
Year: 2020

Recent advances in deep learning have made available large, powerful convolutional neural networks (CNN) with state-of-the-art performance in several real-world applications. Unfortunately, these large-sized models have millions of parameters, thus they are not deployable on resource-limited platforms (e.g. where RAM is limited). Compression of CNNs thereby becomes a critical problem to achieve memory-efficient and possibly computationally faster model representations. In this paper, we investigate the impact of lossy compression of CNNs by weight pruning and quantization, and lossless weight matrix representations based on source coding. We tested several combinations of these techniques on four benchmark datasets for classification and regression problems, achieving compression rates up to $165$ times, while preserving or improving the model performance.