OpenRaiser/Intern-Atlas huggingface dataset:
Edge-Labeling based Directed Gated Graph Network for Few-shot Learning
Venue: arXiv cs.CV 2021
Year: 2021

Existing graph-network-based few-shot learning methods obtain similarity between nodes through a convolution neural network (CNN). However, the CNN is designed for image data with spatial information rather than vector form node feature. In this paper, we proposed an edge-labeling-based directed gated graph network (DGGN) for few-shot learning, which utilizes gated recurrent units to implicitly update the similarity between nodes. DGGN is composed of a gated node aggregation module and an improved gated recurrent unit (GRU) based edge update module. Specifically, the node update module adopts a gate mechanism using activation of edge feature, making a learnable node aggregation process. Besides, improved GRU cells are employed in the edge update procedure to compute the similarity between nodes. Further, this mechanism is beneficial to gradient backpropagation through the GRU sequence across layers. Experiment results conducted on two benchmark datasets show that our DGGN achieves a comparable performance to the-state-of-art methods.
Testing in Global Software Development -- A Pattern Approach
Venue: arXiv cs.SE 2021
Year: 2021

Although testing is critical in GSD, its application in this context has not been deeply investigated so far. This work investigates testing in GSD. It provides support for test managers acting in a globally distributed environment. With this it closes a gap. The leading question is "What problems exist in testing in GSD and how can they be addressed in projects?" Decomposing this question we a) identify problems of testing in GSD projects and b) provide good practices to support practitioners in testing in GSD projects. The research is realized in the context of Capgemini Germany. Our contribution to solving the stated research problem is a collection of 16 patterns for testing in GSD projects. For practitioners the usage of the patterns is simplified by various views on the patterns. Herewith we stipulate research and support project managers and test managers in the realization of testing in GSD projects.
On Massive IoT Connectivity with Temporally-Correlated User Activity
Venue: arXiv cs.IT 2021
Year: 2021

This paper considers joint device activity detection and channel estimation in Internet of Things (IoT) networks, where a large number of IoT devices exist but merely a random subset of them become active for short-packet transmission at each time slot. In particular, we propose to leverage the temporal correlation in user activity, i.e., a device active at the previous time slot is more likely to be still active at the current moment, to improve the detection performance. Despite the temporally-correlated user activity in consecutive time slots, it is challenging to unveil the connection between the activity pattern estimated previously, which is imperfect but the only available side information (SI), and the true activity pattern at the current moment due to the unknown estimation error. In this work, we manage to tackle this challenge under the framework of approximate message passing (AMP). Specifically, thanks to the state evolution, the correlation between the activity pattern estimated by AMP at the previous time slot and the real activity pattern at the previous and current moment is quantified explicitly. Based on the well-defined temporal correlation, we further manage to embed this useful SI into the design of the minimum mean-squared error (MMSE) denoisers and log-likelihood ratio (LLR) test based activity detectors under the AMP framework. Theoretical comparison between the SI-aided AMP algorithm and its counterpart without utilizing temporal correlation is provided. Moreover, numerical results are given to show the significant gain in activity detection accuracy brought by the SI-aided algorithm.
Entropy of Mersenne-Twisters
Venue: arXiv cs.DS 2021
Year: 2021

The Mersenne-Twister is one of the most popular generators of uniform pseudo-random numbers. It is used in many numerical libraries and software. In this paper, we look at the Komolgorov entropy of the original Mersenne-Twister, as well as of more modern variations such as the 64-bit Mersenne-Twisters, the Well generators, and the Melg generators.
Compositional Semantics for Probabilistic Programs with Exact Conditioning
Venue: arXiv cs.PL 2021
Year: 2021

We define a probabilistic programming language for Gaussian random variables with a first-class exact conditioning construct. We give operational, denotational and equational semantics for this language, establishing convenient properties like exchangeability of conditions. Conditioning on equality of continuous random variables is nontrivial, as the exact observation may have probability zero; this is Borel's paradox. Using categorical formulations of conditional probability, we show that the good properties of our language are not particular to Gaussians, but can be derived from universal properties, thus generalizing to wider settings. We define the Cond construction, which internalizes conditioning as a morphism, providing general compositional semantics for probabilistic programming with exact conditioning.
An explainable Transformer-based deep learning model for the prediction of incident heart failure
Venue: arXiv cs.LG 2021
Year: 2021

Predicting the incidence of complex chronic conditions such as heart failure is challenging. Deep learning models applied to rich electronic health records may improve prediction but remain unexplainable hampering their wider use in medical practice. We developed a novel Transformer deep-learning model for more accurate and yet explainable prediction of incident heart failure involving 100,071 patients from longitudinal linked electronic health records across the UK. On internal 5-fold cross validation and held-out external validation, our model achieved 0.93 and 0.93 area under the receiver operator curve and 0.69 and 0.70 area under the precision-recall curve, respectively and outperformed existing deep learning models. Predictor groups included all community and hospital diagnoses and medications contextualised within the age and calendar year for each patient's clinical encounter. The importance of contextualised medical information was revealed in a number of sensitivity analyses, and our perturbation method provided a way of identifying factors contributing to risk. Many of the identified risk factors were consistent with existing knowledge from clinical and epidemiological research but several new associations were revealed which had not been considered in expert-driven risk prediction models.
Variational Encoders and Autoencoders : Information-theoretic Inference and Closed-form Solutions
Venue: arXiv cs.IT 2021
Year: 2021

This work develops problem statements related to encoders and autoencoders with the goal of elucidating variational formulations and establishing clear connections to information-theoretic concepts. Specifically, four problems with varying levels of input are considered : a) The data, likelihood and prior distributions are given, b) The data and likelihood are given; c) The data and prior are given; d) the data and the dimensionality of the parameters is specified. The first two problems seek encoders (or the posterior) and the latter two seek autoencoders (i.e. the posterior and the likelihood). A variational Bayesian setting is pursued, and detailed derivations are provided for the resulting optimization problem. Following this, a linear Gaussian setting is adopted, and closed form solutions are derived. Numerical experiments are also performed to verify expected behavior and assess convergence properties. Explicit connections are made to rate-distortion theory, information bottleneck theory, and the related concept of sufficiency of statistics is also explored. One of the motivations of this work is to present the theory and learning dynamics associated with variational inference and autoencoders, and to expose information theoretic concepts from a computational science perspective.
SkillNER: Mining and Mapping Soft Skills from any Text
Venue: arXiv cs.CL 2021
Year: 2021

In today's digital world, there is an increasing focus on soft skills. On the one hand, they facilitate innovation at companies, but on the other, they are unlikely to be automated soon. Researchers struggle with accurately approaching quantitatively the study of soft skills due to the lack of data-driven methods to retrieve them. This limits the possibility for psychologists and HR managers to understand the relation between humans and digitalisation. This paper presents SkillNER, a novel data-driven method for automatically extracting soft skills from text. It is a named entity recognition (NER) system trained with a support vector machine (SVM) on a corpus of more than 5000 scientific papers. We developed this system by measuring the performance of our approach against different training models and validating the results together with a team of psychologists. Finally, SkillNER was tested in a real-world case study using the job descriptions of ESCO (European Skill/Competence Qualification and Occupation) as textual source. The system enabled the detection of communities of job profiles based on their shared soft skills and communities of soft skills based on their shared job profiles. This case study demonstrates that the tool can automatically retrieve soft skills from a large corpus in an efficient way, proving useful for firms, institutions, and workers. The tool is open and available online to foster quantitative methods for the study of soft skills.
Online Extrinsic Calibration based on Per-Sensor Ego-Motion Using Dual Quaternions
Venue: arXiv cs.RO 2021
Year: 2021

In this work, we propose an approach for extrinsic sensor calibration from per-sensor ego-motion estimates. Our problem formulation is based on dual quaternions, enabling two different online capable solving approaches. We provide a certifiable globally optimal and a fast local approach along with a method to verify the globality of the local approach. Additionally, means for integrating previous knowledge, for example, a common ground plane for planar sensor motion, are described. Our algorithms are evaluated on simulated data and on a publicly available dataset containing RGB-D camera images. Further, our online calibration approach is tested on the KITTI odometry dataset, which provides data of a lidar and two stereo camera systems mounted on a vehicle. Our evaluation confirms the short run time, state-of-the-art accuracy, as well as online capability of our approach while retaining the global optimality of the solution at any time.
Adversaries in Online Learning Revisited: with applications in Robust Optimization and Adversarial training
Venue: arXiv cs.LG 2021
Year: 2021

We revisit the concept of "adversary" in online learning, motivated by solving robust optimization and adversarial training using online learning methods. While one of the classical setups in online learning deals with the "adversarial" setup, it appears that this concept is used less rigorously, causing confusion in applying results and insights from online learning. Specifically, there are two fundamentally different types of adversaries, depending on whether the "adversary" is able to anticipate the exogenous randomness of the online learning algorithms. This is particularly relevant to robust optimization and adversarial training because the adversarial sequences are often anticipative, and many online learning algorithms do not achieve diminishing regret in such a case. We then apply this to solving robust optimization problems or (equivalently) adversarial training problems via online learning and establish a general approach for a large variety of problem classes using imaginary play. Here two players play against each other, the primal player playing the decisions and the dual player playing realizations of uncertain data. When the game terminates, the primal player has obtained an approximately robust solution. This meta-game allows for solving a large variety of robust optimization and multi-objective optimization problems and generalizes the approach of arXiv:1402.6361.
Controlling by Showing: i-Mimic: A Video-based Method to Control Robotic Arms
Venue: arXiv cs.CV 2021
Year: 2021

A novel concept of vision-based intelligent control of robotic arms is developed here in this work. This work enables the controlling of robotic arms motion only with visual inputs, that is, controlling by showing the videos of correct movements. This work can broadly be sub-divided into two segments. The first part of this work is to develop an unsupervised vision-based method to control robotic arm in 2-D plane, and the second one is with deep CNN in the same task in 3-D plane. The first method is unsupervised, where our aim is to perform mimicking of human arm motion in real-time by a manipulator. We developed a network, namely the vision-to-motion optical network (DON), where the input should be a video stream containing hand movements of human, the the output would be out the velocity and torque information of the hand movements shown in the videos. The output information of the DON is then fed to the robotic arm by enabling it to generate motion according to the real hand videos. The method has been tested with both live-stream video feed as well as on recorded video obtained from a monocular camera even by intelligently predicting the trajectory of human hand hand when it gets occluded. This is why the mimicry of the arm incorporates some intelligence to it and becomes intelligent mimic (i-mimic). Alongside the unsupervised method another method has also been developed deploying the deep neural network technique with CNN (Convolutional Neural Network) to perform the mimicking, where labelled datasets are used for training. The same dataset, as used in the unsupervised DON-based method, is used in the deep CNN method, after manual annotations. Both the proposed methods are validated with off-line as well as with on-line video datasets in real-time. The entire methodology is validated with real-time 1-link and simulated n-link manipulators alongwith suitable comparisons.
Investigating Dissemination of Scientific Information on Twitter: A Study of Topic Networks in Opioid Publications
Venue: arXiv cs.IR 2021
Year: 2021

One way to assess a certain aspect of the value of scientific research is to measure the attention it receives on social media. While previous research has mostly focused on the "number of mentions" of scientific research on social media, the current study applies "topic networks" to measure public attention to scientific research on Twitter. Topic networks are the networks of co-occurring author keywords in scholarly publications and networks of co-occurring hashtags in the tweets mentioning those scholarly publications. This study investigates which topics in opioid scholarly publications have received public attention on Twitter. Additionally, it investigates whether the topic networks generated from the publications tweeted by all accounts (bot and non-bot accounts) differ from those generated by non-bot accounts. Our analysis is based on a set of opioid scholarly publications from 2011 to 2019 and the tweets associated with them. We use co-occurrence network analysis to generate topic networks. Results indicated that Twitter users have mostly used generic terms to discuss opioid publications, such as "Opioid," "Pain," "Addiction," "Treatment," "Analgesics," "Abuse," "Overdose," and "Disorders." Results confirm that topic networks provide a legitimate method to visualize public discussions of health-related scholarly publications and how Twitter users discuss health-related scientific research differently from the scientific community. There was a substantial overlap between the topic networks based on the tweets by all accounts and non-bot accounts. This result indicates that it might not be necessary to exclude bot accounts for generating topic networks as they have a negligible impact on the results.
Super-Resolution for Doubly-Dispersive Channel Estimation
Venue: arXiv cs.IT 2021
Year: 2021

In this work we consider the problem of identification and reconstruction of doubly-dispersive channel operators which are given by finite linear combinations of time-frequency shifts. Such operators arise as time-varying linear systems for example in radar and wireless communications. In particular, for information transmission in highly non-stationary environments the channel needs to be estimated quickly with identification signals of short duration and for vehicular application simultaneous high-resolution radar is desired as well. We consider the time-continuous setting and prove an exact resampling reformulation of the involved channel operator when applied to a trigonometric polynomial as identifier in terms of sparse linear combinations of real-valued atoms. Motivated by recent works of Heckel et al. we present an exact approach for off-the-grid superresolution which allows to perform the identification with realizable signals having compact support. Then we show how an alternating descent conditional gradient algorithm can be adapted to solve the reformulated problem. Numerical examples demonstrate the performance of this algorithm, in particular in comparison with a simple adaptive grid refinement strategy and an orthogonal matching pursuit algorithm.
Privacy Information Classification: A Hybrid Approach
Venue: arXiv cs.AI 2021
Year: 2021

A large amount of information has been published to online social networks every day. Individual privacy-related information is also possibly disclosed unconsciously by the end-users. Identifying privacy-related data and protecting the online social network users from privacy leakage turn out to be significant. Under such a motivation, this study aims to propose and develop a hybrid privacy classification approach to detect and classify privacy information from OSNs. The proposed hybrid approach employs both deep learning models and ontology-based models for privacy-related information extraction. Extensive experiments are conducted to validate the proposed hybrid approach, and the empirical results demonstrate its superiority in assisting online social network users against privacy leakage.
CNN with large memory layers
Venue: arXiv cs.CV 2021
Year: 2021

This work is centred around the recently proposed product key memory structure \cite{large_memory}, implemented for a number of computer vision applications. The memory structure can be regarded as a simple computation primitive suitable to be augmented to nearly all neural network architectures. The memory block allows implementing sparse access to memory with square root complexity scaling with respect to the memory capacity. The latter scaling is possible due to the incorporation of Cartesian product space decomposition of the key space for the nearest neighbour search. We have tested the memory layer on the classification, image reconstruction and relocalization problems and found that for some of those, the memory layers can provide significant speed/accuracy improvement with the high utilization of the key-value elements, while others require more careful fine-tuning and suffer from dying keys. To tackle the later problem we have introduced a simple technique of memory re-initialization which helps us to eliminate unused key-value pairs from the memory and engage them in training again. We have conducted various experiments and got improvements in speed and accuracy for classification and PoseNet relocalization models. We showed that the re-initialization has a huge impact on a toy example of randomly labeled data and observed some gains in performance on the image classification task. We have also demonstrated the generalization property perseverance of the large memory layers on the relocalization problem, while observing the spatial correlations between the images and the selected memory cells.
Dopamine: Differentially Private Federated Learning on Medical Data
Venue: arXiv cs.LG 2021
Year: 2021

While rich medical datasets are hosted in hospitals distributed across the world, concerns on patients' privacy is a barrier against using such data to train deep neural networks (DNNs) for medical diagnostics. We propose Dopamine, a system to train DNNs on distributed datasets, which employs federated learning (FL) with differentially-private stochastic gradient descent (DPSGD), and, in combination with secure aggregation, can establish a better trade-off between differential privacy (DP) guarantee and DNN's accuracy than other approaches. Results on a diabetic retinopathy~(DR) task show that Dopamine provides a DP guarantee close to the centralized training counterpart, while achieving a better classification accuracy than FL with parallel DP where DPSGD is applied without coordination. Code is available at https://github.com/ipc-lab/private-ml-for-health.
Compositionality Through Language Transmission, using Artificial Neural Networks
Venue: arXiv cs.CL 2021
Year: 2021

We propose an architecture and process for using the Iterated Learning Model ("ILM") for artificial neural networks. We show that ILM does not lead to the same clear compositionality as observed using DCGs, but does lead to a modest improvement in compositionality, as measured by holdout accuracy and topologic similarity. We show that ILM can lead to an anti-correlation between holdout accuracy and topologic rho. We demonstrate that ILM can increase compositionality when using non-symbolic high-dimensional images as input.
Information contraction in noisy binary neural networks and its implications
Venue: arXiv cs.IT 2021
Year: 2021

Neural networks have gained importance as the machine learning models that achieve state-of-the-art performance on large-scale image classification, object detection and natural language processing tasks. In this paper, we consider noisy binary neural networks, where each neuron has a non-zero probability of producing an incorrect output. These noisy models may arise from biological, physical and electronic contexts and constitute an important class of models that are relevant to the physical world. Intuitively, the number of neurons in such systems has to grow to compensate for the noise while maintaining the same level of expressive power and computation reliability. Our key finding is a lower bound for the required number of neurons in noisy neural networks, which is first of its kind. To prove this lower bound, we take an information theoretic approach and obtain a novel strong data processing inequality (SDPI), which not only generalizes the Evans-Schulman results for binary symmetric channels to general channels, but also improves the tightness drastically when applied to estimate end-to-end information contraction in networks. Our SDPI can be applied to various information processing systems, including neural networks and cellular automata. Applying the SDPI in noisy binary neural networks, we obtain our key lower bound and investigate its implications on network depth-width trade-offs, our results suggest a depth-width trade-off for noisy neural networks that is very different from the established understanding regarding noiseless neural networks. Furthermore, we apply the SDPI to study fault-tolerant cellular automata and obtain bounds on the error correction overheads and the relaxation time. This paper offers new understanding of noisy information processing systems through the lens of information theory.