OpenRaiser/Intern-Atlas huggingface dataset:
Human Action Recognition Using Deep Multilevel Multimodal (M2) Fusion of Depth and Inertial Sensors
Venue: arXiv cs.LG 2019
Year: 2019

Multimodal fusion frameworks for Human Action Recognition (HAR) using depth and inertial sensor data have been proposed over the years. In most of the existing works, fusion is performed at a single level (feature level or decision level), missing the opportunity to fuse rich mid-level features necessary for better classification. To address this shortcoming, in this paper, we propose three novel deep multilevel multimodal fusion frameworks to capitalize on different fusion strategies at various stages and to leverage the superiority of multilevel fusion. At input, we transform the depth data into depth images called sequential front view images (SFIs) and inertial sensor data into signal images. Each input modality, depth and inertial, is further made multimodal by taking convolution with the Prewitt filter. Creating "modality within modality" enables further complementary and discriminative feature extraction through Convolutional Neural Networks (CNNs). CNNs are trained on input images of each modality to learn low-level, high-level and complex features. Learned features are extracted and fused at different stages of the proposed frameworks to combine discriminative and complementary information. These highly informative features are served as input to a multi-class Support Vector Machine (SVM). We evaluate the proposed frameworks on three publicly available multimodal HAR datasets, namely, UTD Multimodal Human Action Dataset (MHAD), Berkeley MHAD, and UTD-MHAD Kinect V2. Experimental results show the supremacy of the proposed fusion frameworks over existing methods.
RhythmNet: End-to-end Heart Rate Estimation from Face via Spatial-temporal Representation
Venue: arXiv cs.CV 2019
Year: 2019

Heart rate (HR) is an important physiological signal that reflects the physical and emotional status of a person. Traditional HR measurements usually rely on contact monitors, which may cause inconvenience and discomfort. Recently, some methods have been proposed for remote HR estimation from face videos; however, most of them focus on well-controlled scenarios, their generalization ability into less-constrained scenarios (e.g., with head movement, and bad illumination) are not known. At the same time, lacking large-scale HR databases has limited the use of deep models for remote HR estimation. In this paper, we propose an end-to-end RhythmNet for remote HR estimation from the face. In RyhthmNet, we use a spatial-temporal representation encoding the HR signals from multiple ROI volumes as its input. Then the spatial-temporal representations are fed into a convolutional network for HR estimation. We also take into account the relationship of adjacent HR measurements from a video sequence via Gated Recurrent Unit (GRU) and achieves efficient HR measurement. In addition, we build a large-scale multi-modal HR database (named as VIPL-HR, available at 'http://vipl.ict.ac.cn/view_database.php?id=15'), which contains 2,378 visible light videos (VIS) and 752 near-infrared (NIR) videos of 107 subjects. Our VIPL-HR database contains various variations such as head movements, illumination variations, and acquisition device changes, replicating a less-constrained scenario for HR estimation. The proposed approach outperforms the state-of-the-art methods on both the public-domain and our VIPL-HR databases.
CrevNet: Conditionally Reversible Video Prediction
Venue: arXiv cs.CV 2019
Year: 2019

Applying resolution-preserving blocks is a common practice to maximize information preservation in video prediction, yet their high memory consumption greatly limits their application scenarios. We propose CrevNet, a Conditionally Reversible Network that uses reversible architectures to build a bijective two-way autoencoder and its complementary recurrent predictor. Our model enjoys the theoretically guaranteed property of no information loss during the feature extraction, much lower memory consumption and computational efficiency.
An End-to-End HW/SW Co-Design Methodology to Design Efficient Deep Neural Network Systems using Virtual Models
Venue: arXiv cs.LG 2019
Year: 2019

End-to-end performance estimation and measurement of deep neural network (DNN) systems become more important with increasing complexity of DNN systems consisting of hardware and software components. The methodology proposed in this paper aims at a reduced turn-around time for evaluating different design choices of hardware and software components of DNN systems. This reduction is achieved by moving the performance estimation from the implementation phase to the concept phase by employing virtual hardware models instead of gathering measurement results from physical prototypes. Deep learning compilers introduce hardware-specific transformations and are, therefore, considered a part of the design flow of virtual system models to extract end-to-end performance estimations. To validate the run-time accuracy of the proposed methodology, a system processing the DilatedVGG DNN is realized both as virtual system model and as hardware implementation. The results show that up to 92 % accuracy can be reached in predicting the processing time of the DNN inference.
Exploring Multilingual Syntactic Sentence Representations
Venue: arXiv cs.CL 2019
Year: 2019

We study methods for learning sentence embeddings with syntactic structure. We focus on methods of learning syntactic sentence-embeddings by using a multilingual parallel-corpus augmented by Universal Parts-of-Speech tags. We evaluate the quality of the learned embeddings by examining sentence-level nearest neighbours and functional dissimilarity in the embedding space. We also evaluate the ability of the method to learn syntactic sentence-embeddings for low-resource languages and demonstrate strong evidence for transfer learning. Our results show that syntactic sentence-embeddings can be learned while using less training data, fewer model parameters, and resulting in better evaluation metrics than state-of-the-art language models.
Control of Flying Robotic Insects: A Perspective and Unifying Approach
Venue: arXiv cs.RO 2019
Year: 2019

We discuss the problem of designing and implementing controllers for insect-scale flapping-wing micro air vehicles (FWMAVs), from a unifying perspective and employing two different experimental platforms; namely, a Harvard RoboBee-like two-winged robot and the four-winged USC Bee+. Through experiments, we demonstrate that a method that employs quaternion coordinates for attitude control, developed to control quadrotors, can be applied to drive both robotic insects considered in this work. The proposed notion that a generic strategy can be used to control several types of artificial insects with some common characteristics was preliminarily tested and validated using a set of experiments, which include position- and attitude-controlled flights. We believe that the presented results are interesting and valuable from both the research and educational perspectives.
Network Characteristics Control Social Dilemmas in a Public Good Game: A Mechanical Framework
Venue: arXiv cs.GT 2019
Year: 2019

Purpose: We propose a model to present a possible mechanism for obtaining sizeable behavioural structures by simulating an agent based on the evolutionary public good game with available social learning. Methods: The model considered a population with a fixed number of players. For each round, the chosen players may contribute part of their value to a common pool. Then each player may imitate the strategy of another player, based on relative payoffs (whoever has the lower payoff adopts the strategy of the other player) and change strategy using different exploratory variables. Results: Relative payoffs are subject to incentive, including participation costs, but are also subject to mutation, the rate of which is sensitized by network characteristics (social ties). Values: The process covered by this study is of interest and is relevant across a broad range of disciplines that use game theory, including the framework of cultural evolutionary dynamics.
Stochastic Geometry Analysis of Hybrid Aerial Terrestrial Networks with mmWave Backhauling
Venue: arXiv cs.IT 2019
Year: 2019

To meet increasing data demands, service providers are considering the use of Unmanned aerial vehicles (UAVs) for delivering connectivity as aerial base stations (BSs). UAVs are especially important to provide connectivity in case of disasters and accidents which may cripple completely the existing terrestrial networks. However, in order to maintain the communication of UAVs with the core network, it is essential to provide them with wireless backhaul connection to terrestrial BSs. In this work, we use stochastic geometry to study the impact of millimeter-wave (mmWave) backhauling of UAVs in a hybrid aerial-terrestrial cellular network where the UAVs are added to assist terrestrial BSs in delivering reliable service to users (UEs). In the proposed model, the UE can associate to either a terrestrial BS or a UAV connected to a BS to get backhaul support. The performance of the proposed model is evaluated in terms of coverage probability and validated against intensive simulations. The obtained results unveil that the quality of the UAVs' mmWave backhaul link has a significant impact on the UE 's experience and the deployment of UAVs must be adjusted accordingly.
Finding Mixed Strategy Nash Equilibrium for Continuous Games through Deep Learning
Venue: arXiv cs.GT 2019
Year: 2019

Nash equilibrium has long been a desired solution concept in multi-player games, especially for those on continuous strategy spaces, which have attracted a rapidly growing amount of interests due to advances in research applications such as the generative adversarial networks. Despite the fact that several deep learning based approaches are designed to obtain pure strategy Nash equilibrium, it is rather luxurious to assume the existence of such an equilibrium. In this paper, we present a new method to approximate mixed strategy Nash equilibria in multi-player continuous games, which always exist and include the pure ones as a special case. We remedy the pure strategy weakness by adopting the pushforward measure technique to represent a mixed strategy in continuous spaces. That allows us to generalize the Gradient-based Nikaido-Isoda (GNI) function to measure the distance between the players' joint strategy profile and a Nash equilibrium. Applying the gradient descent algorithm, our approach is shown to converge to a stationary Nash equilibrium under the convexity assumption on payoff functions, the same popular setting as in previous studies. In numerical experiments, our method consistently and significantly outperforms recent works on approximating Nash equilibrium for quadratic games, general blotto games, and GAMUT games.
A holistic approach to polyphonic music transcription with neural networks
Venue: arXiv cs.SD 2019
Year: 2019

We present a framework based on neural networks to extract music scores directly from polyphonic audio in an end-to-end fashion. Most previous Automatic Music Transcription (AMT) methods seek a piano-roll representation of the pitches, that can be further transformed into a score by incorporating tempo estimation, beat tracking, key estimation or rhythm quantization. Unlike these methods, our approach generates music notation directly from the input audio in a single stage. For this, we use a Convolutional Recurrent Neural Network (CRNN) with Connectionist Temporal Classification (CTC) loss function which does not require annotated alignments of audio frames with the score rhythmic information. We trained our model using as input Haydn, Mozart, and Beethoven string quartets and Bach chorales synthesized with different tempos and expressive performances. The output is a textual representation of four-voice music scores based on **kern format. Although the proposed approach is evaluated in a simplified scenario, results show that this model can learn to transcribe scores directly from audio signals, opening a promising avenue towards complete AMT.
Near-Optimal Bounds for Online Caching with Machine Learned Advice
Venue: arXiv cs.DS 2019
Year: 2019

In the model of online caching with machine learned advice, introduced by Lykouris and Vassilvitskii, the goal is to solve the caching problem with an online algorithm that has access to next-arrival predictions: when each input element arrives, the algorithm is given a prediction of the next time when the element will reappear. The traditional model for online caching suffers from an $Ω(\log k)$ competitive ratio lower bound (on a cache of size $k$). In contrast, the augmented model admits algorithms which beat this lower bound when the predictions have low error, and asymptotically match the lower bound when the predictions have high error, even if the algorithms are oblivious to the prediction error. In particular, Lykouris and Vassilvitskii showed that there is a prediction-augmented caching algorithm with a competitive ratio of $O(1+\min(\sqrt{η/OPT}, \log k))$ when the overall $\ell_1$ prediction error is bounded by $η$, and $OPT$ is the cost of the optimal offline algorithm. The dependence on $k$ in the competitive ratio is optimal, but the dependence on $η/OPT$ may be far from optimal. In this work, we make progress towards closing this gap. Our contributions are twofold. First, we provide an improved algorithm with a competitive ratio of $O(1 + \min((η/OPT)/k, 1) \log k)$. Second, we provide a lower bound of $Ω(\log \min((η/OPT)/(k \log k), k))$.
An Adaptive and Momental Bound Method for Stochastic Learning
Venue: arXiv cs.LG 2019
Year: 2019

Training deep neural networks requires intricate initialization and careful selection of learning rates. The emergence of stochastic gradient optimization methods that use adaptive learning rates based on squared past gradients, e.g., AdaGrad, AdaDelta, and Adam, eases the job slightly. However, such methods have also been proven problematic in recent studies with their own pitfalls including non-convergence issues and so on. Alternative variants have been proposed for enhancement, such as AMSGrad, AdaShift and AdaBound. In this work, we identify a new problem of adaptive learning rate methods that exhibits at the beginning of learning where Adam produces extremely large learning rates that inhibit the start of learning. We propose the Adaptive and Momental Bound (AdaMod) method to restrict the adaptive learning rates with adaptive and momental upper bounds. The dynamic learning rate bounds are based on the exponential moving averages of the adaptive learning rates themselves, which smooth out unexpected large learning rates and stabilize the training of deep neural networks. Our experiments verify that AdaMod eliminates the extremely large learning rates throughout the training and brings significant improvements especially on complex networks such as DenseNet and Transformer, compared to Adam. Our implementation is available at: https://github.com/lancopku/AdaMod
Exploring 3 R's of Long-term Tracking: Re-detection, Recovery and Reliability
Venue: arXiv cs.CV 2019
Year: 2019

Recent works have proposed several long term tracking benchmarks and highlight the importance of moving towards long-duration tracking to bridge the gap with application requirements. The current evaluation methodologies, however, do not focus on several aspects that are crucial in a long term perspective like Re-detection, Recovery, and Reliability. In this paper, we propose novel evaluation strategies for a more in-depth analysis of trackers from a long-term perspective. More specifically, (a) we test re-detection capability of the trackers in the wild by simulating virtual cuts, (b) we investigate the role of chance in the recovery of tracker after failure and (c) we propose a novel metric allowing visual inference on the ability of a tracker to track contiguously (without any failure) at a given accuracy. We present several original insights derived from an extensive set of quantitative and qualitative experiments.
A memory enhanced LSTM for modeling complex temporal dependencies
Venue: arXiv cs.LG 2019
Year: 2019

In this paper, we present Gamma-LSTM, an enhanced long short term memory (LSTM) unit, to enable learning of hierarchical representations through multiple stages of temporal abstractions. Gamma memory, a hierarchical memory unit, forms the central memory of Gamma-LSTM with gates to regulate the information flow into various levels of hierarchy, thus providing the unit with a control to pick the appropriate level of hierarchy to process the input at a given instant of time. We demonstrate better performance of Gamma-LSTM model regular and stacked LSTMs in two settings (pixel-by-pixel MNIST digit classification and natural language inference) placing emphasis on the ability to generalize over long sequences.
What does BERT Learn from Multiple-Choice Reading Comprehension Datasets?
Venue: arXiv cs.CL 2019
Year: 2019

Multiple-Choice Reading Comprehension (MCRC) requires the model to read the passage and question, and select the correct answer among the given options. Recent state-of-the-art models have achieved impressive performance on multiple MCRC datasets. However, such performance may not reflect the model's true ability of language understanding and reasoning. In this work, we adopt two approaches to investigate what BERT learns from MCRC datasets: 1) an un-readable data attack, in which we add keywords to confuse BERT, leading to a significant performance drop; and 2) an un-answerable data training, in which we train BERT on partial or shuffled input. Under un-answerable data training, BERT achieves unexpectedly high performance. Based on our experiments on the 5 key MCRC datasets - RACE, MCTest, MCScript, MCScript2.0, DREAM - we observe that 1) fine-tuned BERT mainly learns how keywords lead to correct prediction, instead of learning semantic understanding and reasoning; and 2) BERT does not need correct syntactic information to solve the task; 3) there exists artifacts in these datasets such that they can be solved even without the full context.
Effectiveness of random deep feature selection for securing image manipulation detectors against adversarial examples
Venue: arXiv cs.CR 2019
Year: 2019

We investigate if the random feature selection approach proposed in [1] to improve the robustness of forensic detectors to targeted attacks, can be extended to detectors based on deep learning features. In particular, we study the transferability of adversarial examples targeting an original CNN image manipulation detector to other detectors (a fully connected neural network and a linear SVM) that rely on a random subset of the features extracted from the flatten layer of the original network. The results we got by considering three image manipulation detection tasks (resizing, median filtering and adaptive histogram equalization), two original network architectures and three classes of attacks, show that feature randomization helps to hinder attack transferability, even if, in some cases, simply changing the architecture of the detector, or even retraining the detector is enough to prevent the transferability of the attacks.
Information Seeking and Information Processing Behaviors Among Type 2 Diabetics
Venue: arXiv cs.HC 2019
Year: 2019

Effective patient education is critical for managing Type 2 Diabetes Mellitus (T2DM), one of the most common chronic diseases in the United States. While some studies focus on the information-seeking behavior of T2DM patients, other self-education behaviors including information processing and utilization are rarely explored in the context of T2DM. This study sought to assess two self-education behaviors of type 2 diabetics, namely, information seeking and information processing, to understand more about how these behaviors affect the self-management of this common chronic disease. Semi-structured interviews were conducted with 8 English speaking T2DM patients and qualitative content analysis techniques were performed to analyze their responses. The information seeking and processing behaviors vary across individuals based on their prognosis of T2DM, information needs, and personal preferences. Patients are often dissatisfied with information from official sources, have difficulty evaluating the trustworthiness of information sources, and desire information that is more personally relevant to them. Several participants identified a lack of personalized information as a key factor in the inability to adhere to T2DM management guidelines, which led them to experience increased glucose levels, difficulty managing A1C levels, frustration, and anxiety. They mentioned that they followed trial and error based approaches to tailor information according to their needs and physiological conditions. Many participants identified conflicting or inconsistent information from different sources as a major barrier to information processing. The results of this study indicate a need for authentic, consistent, and individualized information for type 2 diabetics.
Towards Successful Social Media Advertising: Predicting the Influence of Commercial Tweets
Venue: arXiv cs.SI 2019
Year: 2019

Businesses communicate using Twitter for a variety of reasons -- to raise awareness of their brands, to market new products, to respond to community comments, and to connect with their customers and potential customers in a targeted manner. For businesses to do this effectively, they need to understand which content and structural elements about a tweet make it influential, that is, widely liked, followed, and retweeted. This paper presents a systematic methodology for analyzing commercial tweets, and predicting the influence on their readers. Our model, which use a combination of decoration and meta features, outperforms the prediction ability of the baseline model as well as the tweet embedding model. Further, in order to demonstrate a practical use of this work, we show how an unsuccessful tweet may be engineered (for example, reworded) to increase its potential for success.
Asynchronous Methods for Model-Based Reinforcement Learning
Venue: arXiv cs.LG 2019
Year: 2019

Significant progress has been made in the area of model-based reinforcement learning. State-of-the-art algorithms are now able to match the asymptotic performance of model-free methods while being significantly more data efficient. However, this success has come at a price: state-of-the-art model-based methods require significant computation interleaved with data collection, resulting in run times that take days, even if the amount of agent interaction might be just hours or even minutes. When considering the goal of learning in real-time on real robots, this means these state-of-the-art model-based algorithms still remain impractical. In this work, we propose an asynchronous framework for model-based reinforcement learning methods that brings down the run time of these algorithms to be just the data collection time. We evaluate our asynchronous framework on a range of standard MuJoCo benchmarks. We also evaluate our asynchronous framework on three real-world robotic manipulation tasks. We show how asynchronous learning not only speeds up learning w.r.t wall-clock time through parallelization, but also further reduces the sample complexity of model-based approaches by means of improving the exploration and by means of effectively avoiding the policy overfitting to the deficiencies of learned dynamics models.