OpenRaiser/Intern-Atlas huggingface dataset:
Generating Natural Questions from Images for Multimodal Assistants
Venue: arXiv cs.CV 2020
Year: 2020

Generating natural, diverse, and meaningful questions from images is an essential task for multimodal assistants as it confirms whether they have understood the object and scene in the images properly. The research in visual question answering (VQA) and visual question generation (VQG) is a great step. However, this research does not capture questions that a visually-abled person would ask multimodal assistants. Recently published datasets such as KB-VQA, FVQA, and OK-VQA try to collect questions that look for external knowledge which makes them appropriate for multimodal assistants. However, they still contain many obvious and common-sense questions that humans would not usually ask a digital assistant. In this paper, we provide a new benchmark dataset that contains questions generated by human annotators keeping in mind what they would ask multimodal digital assistants. Large scale annotations for several hundred thousand images are expensive and time-consuming, so we also present an effective way of automatically generating questions from unseen images. In this paper, we present an approach for generating diverse and meaningful questions that consider image content and metadata of image (e.g., location, associated keyword). We evaluate our approach using standard evaluation metrics such as BLEU, METEOR, ROUGE, and CIDEr to show the relevance of generated questions with human-provided questions. We also measure the diversity of generated questions using generative strength and inventiveness metrics. We report new state-of-the-art results on the public and our datasets.
UNOC: Understanding Occlusion for Embodied Presence in Virtual Reality
Venue: arXiv cs.CV 2020
Year: 2020

Tracking body and hand motions in the 3D space is essential for social and self-presence in augmented and virtual environments. Unlike the popular 3D pose estimation setting, the problem is often formulated as inside-out tracking based on embodied perception (e.g., egocentric cameras, handheld sensors). In this paper, we propose a new data-driven framework for inside-out body tracking, targeting challenges of omnipresent occlusions in optimization-based methods (e.g., inverse kinematics solvers). We first collect a large-scale motion capture dataset with both body and finger motions using optical markers and inertial sensors. This dataset focuses on social scenarios and captures ground truth poses under self-occlusions and body-hand interactions. We then simulate the occlusion patterns in head-mounted camera views on the captured ground truth using a ray casting algorithm and learn a deep neural network to infer the occluded body parts. In the experiments, we show that our method is able to generate high-fidelity embodied poses by applying the proposed method on the task of real-time inside-out body tracking, finger motion synthesis, and 3-point inverse kinematics.
Conversational Browsing
Venue: arXiv cs.IR 2020
Year: 2020

How can we better understand the mechanisms behind multi-turn information seeking dialogues? How can we use these insights to design a dialogue system that does not require explicit query formulation upfront as in question answering? To answer these questions, we collected observations of human participants performing a similar task to obtain inspiration for the system design. Then, we studied the structure of conversations that occurred in these settings and used the resulting insights to develop a grounded theory, design and evaluate a first system prototype. Evaluation results show that our approach is effective and can complement query-based information retrieval approaches. We contribute new insights about information-seeking behavior by analyzing and providing automated support for a type of information-seeking strategy that is effective when the clarity of the information need and familiarity with the collection content are low.
Pricing Security in Proof-of-Work Systems
Venue: arXiv cs.CR 2020
Year: 2020

A key component of security in decentralized blockchains is proof of opportunity cost among block producers. In the case of proof-of-work (PoW), currently used by the most prominent systems, the cost is due to spent computation. In this paper, we characterize the security investment of miners in terms of its cost in fiat money. This enables comparison of security allocations across PoW blockchains that generally use different PoW algorithms and reward miners in different cryptocurrency units. We prove that there exists a unique allocation equilibrium, depending on market prices only, that is achieved by both strategic miners (who contemplate the actions of others) and by miners seeking only short-term profit. In fact, the latter will unknowingly compensate for any attempt to deliberately shift security allocation away from equilibrium. Our conclusions are supported analytically through the development of a Markov decision process, game theoretical analysis, and derivation of no arbitrage conditions. We corroborate those results with empirical evidence from more than two years of blockchain and price data. Overall agreement is strong. We show that between January 1, 2018 and August 1, 2020, market prices predicted security allocation between Bitcoin and Bitcoin Cash with error less than 0.6%. And from the beginning of October 2019, until August 1, 2020, market prices predicted security allocation between Bitcoin and Litecoin with error of 0.45%. These results are further corroborated by our establishment of Granger-causality between change in market prices and change in security allocation. To demonstrate the practicality of our results, we describe a trustless oracle that leverages the equilibrium to estimate the price ratios of PoW cryptocurrencies from on-chain information only.
Combining Deep Transfer Learning with Signal-image Encoding for Multi-Modal Mental Wellbeing Classification
Venue: arXiv cs.CV 2020
Year: 2020

The quantification of emotional states is an important step to understanding wellbeing. Time series data from multiple modalities such as physiological and motion sensor data have proven to be integral for measuring and quantifying emotions. Monitoring emotional trajectories over long periods of time inherits some critical limitations in relation to the size of the training data. This shortcoming may hinder the development of reliable and accurate machine learning models. To address this problem, this paper proposes a framework to tackle the limitation in performing emotional state recognition on multiple multimodal datasets: 1) encoding multivariate time series data into coloured images; 2) leveraging pre-trained object recognition models to apply a Transfer Learning (TL) approach using the images from step 1; 3) utilising a 1D Convolutional Neural Network (CNN) to perform emotion classification from physiological data; 4) concatenating the pre-trained TL model with the 1D CNN. Furthermore, the possibility of performing TL to infer stress from physiological data is explored by initially training a 1D CNN using a large physical activity dataset and then applying the learned knowledge to the target dataset. We demonstrate that model performance when inferring real-world wellbeing rated on a 5-point Likert scale can be enhanced using our framework, resulting in up to 98.5% accuracy, outperforming a conventional CNN by 4.5%. Subject-independent models using the same approach resulted in an average of 72.3% accuracy (SD 0.038). The proposed CNN-TL-based methodology may overcome problems with small training datasets, thus improving on the performance of conventional deep learning methods.
Sequential Resource Access: Theory and Algorithm
Venue: arXiv cs.NI 2020
Year: 2020

We formulate and analyze a generic sequential resource access problem arising in a variety of engineering fields, where a user disposes a number of heterogeneous computing, communication, or storage resources, each characterized by the probability of successfully executing the user's task and the related access delay and cost, and seeks an optimal access strategy to maximize her utility within a given time horizon, defined as the expected reward minus the access cost. We develop an algorithmic framework on the (near-)optimal sequential resource access strategy. We first prove that the problem of finding an optimal strategy is NP-hard in general. Given the hardness result, we present a greedy strategy implementable in linear time, and establish the closed-form sufficient condition for its optimality. We then develop a series of polynomial-time approximation algorithms achieving $(ε,δ)$-optimality, with the key component being a pruning process eliminating dominated strategies and, thus maintaining polynomial time and space overhead.
Patterns, anticipation and participatory futures
Venue: arXiv cs.DC 2020
Year: 2020

Patterns embody repeating phenomena, and, as such, they are partly but not fully detachable from their context. 'Design patterns' and 'pattern languages' are established methods for working with patterns. They have been applied in architecture, software engineering, and other design fields, but have so far seen little application in the field of future studies. We reimagine futures discourse and anticipatory practices using pattern methods. We focus specifically on processes for coordinating distributed projects, integrating multiple voices, and on play that builds capability to face what's yet to come. One of the advantages of the method as a whole is that it deals with local knowledge and does not subsume everything within one overall 'global' strategy, while nevertheless offering a way to communicate between contexts and disciplines.
Deep Learning Methods for Credit Card Fraud Detection
Venue: arXiv cs.LG 2020
Year: 2020

Credit card frauds are at an ever-increasing rate and have become a major problem in the financial sector. Because of these frauds, card users are hesitant in making purchases and both the merchants and financial institutions bear heavy losses. Some major challenges in credit card frauds involve the availability of public data, high class imbalance in data, changing nature of frauds and the high number of false alarms. Machine learning techniques have been used to detect credit card frauds but no fraud detection systems have been able to offer great efficiency to date. Recent development of deep learning has been applied to solve complex problems in various areas. This paper presents a thorough study of deep learning methods for the credit card fraud detection problem and compare their performance with various machine learning algorithms on three different financial datasets. Experimental results show great performance of the proposed deep learning methods against traditional machine learning models and imply that the proposed approaches can be implemented effectively for real-world credit card fraud detection systems.
Dynamic Clustering in Federated Learning
Venue: arXiv cs.LG 2020
Year: 2020

In the resource management of wireless networks, Federated Learning has been used to predict handovers. However, non-independent and identically distributed data degrade the accuracy performance of such predictions. To overcome the problem, Federated Learning can leverage data clustering algorithms and build a machine learning model for each cluster. However, traditional data clustering algorithms, when applied to the handover prediction, exhibit three main limitations: the risk of data privacy breach, the fixed shape of clusters, and the non-adaptive number of clusters. To overcome these limitations, in this paper, we propose a three-phased data clustering algorithm, namely: generative adversarial network-based clustering, cluster calibration, and cluster division. We show that the generative adversarial network-based clustering preserves privacy. The cluster calibration deals with dynamic environments by modifying clusters. Moreover, the divisive clustering explores the different number of clusters by repeatedly selecting and dividing a cluster into multiple clusters. A baseline algorithm and our algorithm are tested on a time series forecasting task. We show that our algorithm improves the performance of forecasting models, including cellular network handover, by 43%.
Perspectives on Sim2Real Transfer for Robotics: A Summary of the R:SS 2020 Workshop
Venue: arXiv cs.RO 2020
Year: 2020

This report presents the debates, posters, and discussions of the Sim2Real workshop held in conjunction with the 2020 edition of the "Robotics: Science and System" conference. Twelve leaders of the field took competing debate positions on the definition, viability, and importance of transferring skills from simulation to the real world in the context of robotics problems. The debaters also joined a large panel discussion, answering audience questions and outlining the future of Sim2Real in robotics. Furthermore, we invited extended abstracts to this workshop which are summarized in this report. Based on the workshop, this report concludes with directions for practitioners exploiting this technology and for researchers further exploring open problems in this area.
Vulnerability Forecasting: In theory and practice
Venue: arXiv cs.CR 2020
Year: 2020

Why wait for zero-days when you could predict them in advance? It is possible to predict the volume of CVEs released in the NVD as much as a year in advance. This can be done within 3 percent of the actual value, and different predictive algorithms perform well at different lookahead values. It is also possible to estimate the proportions of that total volumn belonging to specific vendors, software, CVSS scores, or vulnerability types. Strategic patch management should become much easier, with this uncertainty reduction.
Correct block-design experiments mitigate temporal correlation bias in EEG classification
Venue: arXiv cs.CV 2020
Year: 2020

It is argued in [1] that [2] was able to classify EEG responses to visual stimuli solely because of the temporal correlation that exists in all EEG data and the use of a block design. We here show that the main claim in [1] is drastically overstated and their other analyses are seriously flawed by wrong methodological choices. To validate our counter-claims, we evaluate the performance of state-of-the-art methods on the dataset in [2] reaching about 50% classification accuracy over 40 classes, lower than in [2], but still significant. We then investigate the influence of EEG temporal correlation on classification accuracy by testing the same models in two additional experimental settings: one that replicates [1]'s rapid-design experiment, and another one that examines the data between blocks while subjects are shown a blank screen. In both cases, classification accuracy is at or near chance, in contrast to what [1] reports, indicating a negligible contribution of temporal correlation to classification accuracy. We, instead, are able to replicate the results in [1] only when intentionally contaminating our data by inducing a temporal correlation. This suggests that what Li et al. [1] demonstrate is that their data are strongly contaminated by temporal correlation and low signal-to-noise ratio. We argue that the reason why Li et al. [1] observe such high correlation in EEG data is their unconventional experimental design and settings that violate the basic cognitive neuroscience design recommendations, first and foremost the one of limiting the experiments' duration, as instead done in [2]. Our analyses in this paper refute the claims of the "perils and pitfalls of block-design" in [1]. Finally, we conclude the paper by examining a number of other oversimplistic statements, inconsistencies, misinterpretation of machine learning concepts, speculations and misleading claims in [1].
Hierarchical Deep Recurrent Neural Network based Method for Fault Detection and Diagnosis
Venue: arXiv cs.LG 2020
Year: 2020

A Deep Neural Network (DNN) based algorithm is proposed for the detection and classification of faults in industrial plants. The proposed algorithm has the ability to classify faults, especially incipient faults that are difficult to detect and diagnose with traditional threshold based statistical methods or by conventional Artificial Neural Networks (ANNs). The algorithm is based on a Supervised Deep Recurrent Autoencoder Neural Network (Supervised DRAE-NN) that uses dynamic information of the process along the time horizon. Based on this network a hierarchical structure is formulated by grouping faults based on their similarity into subsets of faults for detection and diagnosis. Further, an external pseudo-random binary signal (PRBS) is designed and injected into the system to identify incipient faults. The hierarchical structure based strategy improves the detection and classification accuracy significantly for both incipient and non-incipient faults. The proposed approach is tested on the benchmark Tennessee Eastman Process resulting in significant improvements in classification as compared to both multivariate linear model-based strategies and non-hierarchical nonlinear model-based strategies.
COVIDScholar: An automated COVID-19 research aggregation and analysis platform
Venue: arXiv cs.DL 2020
Year: 2020

The ongoing COVID-19 pandemic has had far-reaching effects throughout society, and science is no exception. The scale, speed, and breadth of the scientific community's COVID-19 response has lead to the emergence of new research literature on a remarkable scale -- as of October 2020, over 81,000 COVID-19 related scientific papers have been released, at a rate of over 250 per day. This has created a challenge to traditional methods of engagement with the research literature; the volume of new research is far beyond the ability of any human to read, and the urgency of response has lead to an increasingly prominent role for pre-print servers and a diffusion of relevant research across sources. These factors have created a need for new tools to change the way scientific literature is disseminated. COVIDScholar is a knowledge portal designed with the unique needs of the COVID-19 research community in mind, utilizing NLP to aid researchers in synthesizing the information spread across thousands of emergent research articles, patents, and clinical trials into actionable insights and new knowledge. The search interface for this corpus, https://covidscholar.org, now serves over 2000 unique users weekly. We present also an analysis of trends in COVID-19 research over the course of 2020.
Centrality of nodes in Federated Byzantine Agreement Systems
Venue: arXiv cs.DC 2020
Year: 2020

The federated Byzantine agreement system (FBAS) is a consensus model introduced by Mazières in 2016 where the participating nodes conceptually form a network, with links between them being established by each node individually and thus in a decentralized way. An important question is whether these decentralized decisions lead to an overall decentralized network. The level of (de-)centralization in a network can be assessed using centrality measures. In this paper we consider three different approaches for obtaining centrality measures for the nodes in an FBAS. Two of them are based on adapting well-known measures based on graphs and hypergraphs to the FBAS context. Since the network structure of an FBAS can be more complex than (usual) graphs or hypergraphs, we also develop a new, problem-adapted centrality measure. This new measure is based on the intactness of nodes, which is an important ingredient of the FBAS model. We illustrate advantages and disadvantages of the three approaches on several computed examples. We have implemented all centrality measures and performed all computations in the Python package Stellar Observatory.
The friendship paradox in real and model networks
Venue: arXiv cs.SI 2020
Year: 2020

The friendship paradox is the observation that the degrees of the neighbors of a node in any network will, on average, be greater than the degree of the node itself. In common parlance, your friends have more friends than you do. In this paper we develop the mathematical theory of the friendship paradox, both in general as well as for specific model networks, focusing not only on average behavior but also on variation about the average and using generating function methods to calculate full distributions of quantities of interest. We compare the predictions of our theory with measurements on a large number of real-world network data sets and find remarkably good agreement. We also develop equivalent theory for the generalized friendship paradox, which compares characteristics of nodes other than degree to those of their neighbors.
Design and Analysis of Uplink and Downlink Communications for Federated Learning
Venue: arXiv cs.IT 2020
Year: 2020

Communication has been known to be one of the primary bottlenecks of federated learning (FL), and yet existing studies have not addressed the efficient communication design, particularly in wireless FL where both uplink and downlink communications have to be considered. In this paper, we focus on the design and analysis of physical layer quantization and transmission methods for wireless FL. We answer the question of what and how to communicate between clients and the parameter server and evaluate the impact of the various quantization and transmission options of the updated model on the learning performance. We provide new convergence analysis of the well-known FedAvg under non-i.i.d. dataset distributions, partial clients participation, and finite-precision quantization in uplink and downlink communications. These analyses reveal that, in order to achieve an O(1/T) convergence rate with quantization, transmitting the weight requires increasing the quantization level at a logarithmic rate, while transmitting the weight differential can keep a constant quantization level. Comprehensive numerical evaluation on various real-world datasets reveals that the benefit of a FL-tailored uplink and downlink communication design is enormous - a carefully designed quantization and transmission achieves more than 98% of the floating-point baseline accuracy with fewer than 10% of the baseline bandwidth, for majority of the experiments on both i.i.d. and non-i.i.d. datasets. In particular, 1-bit quantization (3.1% of the floating-point baseline bandwidth) achieves 99.8% of the floating-point baseline accuracy at almost the same convergence rate on MNIST, representing the best known bandwidth-accuracy tradeoff to the best of the authors' knowledge.
Frame-level SpecAugment for Deep Convolutional Neural Networks in Hybrid ASR Systems
Venue: arXiv cs.CL 2020
Year: 2020

Inspired by SpecAugment -- a data augmentation method for end-to-end ASR systems, we propose a frame-level SpecAugment method (f-SpecAugment) to improve the performance of deep convolutional neural networks (CNN) for hybrid HMM based ASR systems. Similar to the utterance level SpecAugment, f-SpecAugment performs three transformations: time warping, frequency masking, and time masking. Instead of applying the transformations at the utterance level, f-SpecAugment applies them to each convolution window independently during training. We demonstrate that f-SpecAugment is more effective than the utterance level SpecAugment for deep CNN based hybrid models. We evaluate the proposed f-SpecAugment on 50-layer Self-Normalizing Deep CNN (SNDCNN) acoustic models trained with up to 25000 hours of training data. We observe f-SpecAugment reduces WER by 0.5-4.5% relatively across different ASR tasks for four languages. As the benefits of augmentation techniques tend to diminish as training data size increases, the large scale training reported is important in understanding the effectiveness of f-SpecAugment. Our experiments demonstrate that even with 25k training data, f-SpecAugment is still effective. We also demonstrate that f-SpecAugment has benefits approximately equivalent to doubling the amount of training data for deep CNNs.