Reviewerly/exHarmony huggingface dataset:
Deep Generative Probabilistic Graph Neural Networks for Scene Graph Generation

We propose a new algorithm, called Deep Generative Probabilistic Graph Neural Networks (DG-PGNN), to generate a scene graph for an image. The input to DG-PGNN is an image, together with a set of region-grounded captions and object bounding-box proposals for the image. To generate the scene graph, DG-PGNN constructs and updates a new model, called a Probabilistic Graph Network (PGN). A PGN can be thought of as a scene graph with uncertainty: it represents each node and each edge by a CNN feature vector and defines a probability mass function (PMF) for node-type (object category) of each node and edge-type (predicate class) of each edge. The DG-PGNN sequentially adds a new node to the current PGN by learning the optimal ordering in a Deep Q-learning framework, where states are partial PGNs, actions choose a new node, and rewards are defined based on the ground-truth. After adding a node, DG-PGNN uses message passing to update the feature vectors of the current PGN by leveraging contextual relationship information, object co-occurrences, and language priors from captions. The updated features are then used to fine-tune the PMFs. Our experiments show that the proposed algorithm significantly outperforms the state-of-the-art results on the Visual Genome dataset for scene graph generation. We also show that the scene graphs constructed by DG-PGNN improve performance on the visual question answering task, for questions that need reasoning about objects and their interactions in the scene context.
Cross-Lingual Pre-Training Based Transfer for Zero-Shot Neural Machine Translation

Transfer learning between different language pairs has shown its effectiveness for Neural Machine Translation (NMT) in low-resource scenario. However, existing transfer methods involving a common target language are far from success in the extreme scenario of zero-shot translation, due to the language space mismatch problem between transferor (the parent model) and transferee (the child model) on the source side. To address this challenge, we propose an effective transfer learning approach based on cross-lingual pre-training. Our key idea is to make all source languages share the same feature space and thus enable a smooth transition for zero-shot translation. To this end, we introduce one monolingual pre-training method and two bilingual pre-training methods to obtain a universal encoder for different languages. Once the universal encoder is constructed, the parent model built on such encoder is trained with large-scale annotated data and then directly applied in zero-shot translation scenario. Experiments on two public datasets show that our approach significantly outperforms strong pivot-based baseline and various multilingual NMT approaches.
Uncertainty-Aware Deep Classifiers Using Generative Models

Deep neural networks are often ignorant about what they do not know and overconfident when they make uninformed predictions. Some recent approaches quantify classification uncertainty directly by training the model to output high uncertainty for the data samples close to class boundaries or from the outside of the training distribution. These approaches use an auxiliary data set during training to represent out-of-distribution samples. However, selection or creation of such an auxiliary data set is non-trivial, especially for high dimensional data such as images. In this work we develop a novel neural network model that is able to express both aleatoric and epistemic uncertainty to distinguish decision boundary and out-of-distribution regions of the feature space. To this end, variational autoencoders and generative adversarial networks are incorporated to automatically generate out-of-distribution exemplars for training. Through extensive analysis, we demonstrate that the proposed approach provides better estimates of uncertainty for in- and out-of-distribution samples, and adversarial examples on well-known data sets against state-of-the-art approaches including recent Bayesian approaches for neural networks and anomaly detection methods.
KnowIT VQA: Answering Knowledge-Based Questions about Videos

We propose a novel video understanding task by fusing knowledge-based and video question answering. First, we introduce KnowIT VQA, a video dataset with 24,282 human-generated question-answer pairs about a popular sitcom. The dataset combines visual, textual and temporal coherence reasoning together with knowledge-based questions, which need of the experience obtained from the viewing of the series to be answered. Second, we propose a video understanding model by combining the visual and textual video content with specific knowledge about the show. Our main findings are: (i) the incorporation of knowledge produces outstanding improvements for VQA in video, and (ii) the performance on KnowIT VQA still lags well behind human accuracy, indicating its usefulness for studying current video modelling limitations.
Task-Aware Monocular Depth Estimation for 3D Object Detection

Monocular depth estimation enables 3D perception from a single 2D image, thus attracting much research attention for years. Almost all methods treat foreground and background regions (“things and stuff”) in an image equally. However, not all pixels are equal. Depth of foreground objects plays a crucial role in 3D object recognition and localization. To date how to boost the depth prediction accuracy of foreground objects is rarely discussed. In this paper, we first analyze the data distributions and interaction of foreground and background, then propose the foreground-background separated monocular depth estimation (ForeSeE) method, to estimate the foreground and background depth using separate optimization objectives and decoders. Our method significantly improves the depth estimation performance on foreground objects. Applying ForeSeE to 3D object detection, we achieve 7.5 AP gains and set new state-of-the-art results among other monocular methods. Code will be available at: https://github.com/WXinlong/ForeSeE.
A model-free fuzzy adaptive trajectory tracking control algorithm based on dynamic surface control

According to the robot’s dynamics, a high performance algorithm based on dynamic surface control is introduced to track desired trajectory, and simulations are conducted on a selective compliance assembly robot arm-type manipulator to verify the algorithm. The traditional dynamic surface control is designed based on dynamic model, which requires exact model information. Due to the model uncertainty and complex environments, the tracking performance of the controller can be significantly decreased. Therefore, a model-free fuzzy adaptive dynamic surface controller is designed, by adopting a fuzzy system with Lyapunov self-adaptation law. The new controller efficiently improves the dynamic quality. The simulation results prove that the designed model-free controller ensures that all the states and signals of the closed-loop system are bounded, the system has a faster response speed and smaller steady-state error comparing with the traditional dynamic surface control using the selective compliance assembly robot arm model, and the tracking error converge to a very small scale. Besides, the proposed algorithm can track the desired trajectory with high performance without the prior knowledge of specific parameters from the experimental manipulator, which simplifies the complexity of building the control system.
Developing a Model to Predict Hospital Encounters for Asthma in Asthmatic Patients: Secondary Analysis

Background As a major chronic disease, asthma causes many emergency department (ED) visits and hospitalizations each year. Predictive modeling is a key technology to prospectively identify high-risk asthmatic patients and enroll them in care management for preventive care to reduce future hospital encounters, including inpatient stays and ED visits. However, existing models for predicting hospital encounters in asthmatic patients are inaccurate. Usually, they miss over half of the patients who will incur future hospital encounters and incorrectly classify many others who will not. This makes it difficult to match the limited resources of care management to the patients who will incur future hospital encounters, increasing health care costs and degrading patient outcomes. Objective The goal of this study was to develop a more accurate model for predicting hospital encounters in asthmatic patients. Methods Secondary analysis of 334,564 data instances from Intermountain Healthcare from 2005 to 2018 was conducted to build a machine learning classification model to predict the hospital encounters for asthma in the following year in asthmatic patients. The patient cohort included all asthmatic patients who resided in Utah or Idaho and visited Intermountain Healthcare facilities during 2005 to 2018. A total of 235 candidate features were considered for model building. Results The model achieved an area under the receiver operating characteristic curve of 0.859 (95% CI 0.846-0.871). When the cutoff threshold for conducting binary classification was set at the top 10.00% (1926/19,256) of asthmatic patients with the highest predicted risk, the model reached an accuracy of 90.31% (17,391/19,256; 95% CI 89.86-90.70), a sensitivity of 53.7% (436/812; 95% CI 50.12-57.18), and a specificity of 91.93% (16,955/18,444; 95% CI 91.54-92.31). To steer future research on this topic, we pinpointed several potential improvements to our model. Conclusions Our model improves the state of the art for predicting hospital encounters for asthma in asthmatic patients. After further refinement, the model could be integrated into a decision support tool to guide asthma care management allocation. International Registered Report Identifier (IRRID) RR2-10.2196/resprot.5039
New Suppression Technology for Low-Frequency Noise in Desert Region: The Improved Robust Principal Component Analysis Based on Prediction of Neural Network

Lots of low-frequency noise including random noise and surface waves seriously reduces the quality of desert seismic data. However, the suppression for desert low-frequency noise faces three main problems: nonstationary and non-Gaussian of random noise; strong energy of low-frequency noise; a more serious frequency-band overlap between effective signals and low-frequency noise. Robust principal component analysis (RPCA) is a classical low-rank matrix (LM) recovery method which is very suitable for processing nonlinear noise. It can decompose noisy data to the optimal LM and sparse matrix (SM), which include most effective signals and noise, respectively. Therefore, the RPCA is introduced to suppress desert low-frequency noise. However, due to the low signal-to-noise ratio (SNR) and serious frequency-band overlap, much low-frequency noise still remains in the LM of desert seismic data after the decomposition of RPCA. Meanwhile, some nonnegligible effective signals are decomposed into the SM of desert seismic data. To solve this problem, the convolutional neural network (CNN) is introduced to extract effective signals from SM and LM. By constructing suitable training sets to guide the CNN's training, the CNN denoising models after training are used to predict the effective signals from these two matrices, respectively. In this article, to approach real desert seismic data, we use a variety of seismic wavelets to simulate different types of seismic events, and then use these synthetic seismic events and real desert low-frequency noise to construct training set. In experiments, our method can raise the SNR of synthetic noisy data from -8.69 to 9.63 dB.
Dynamic Searchable Encryption with Small Client Storage

We study the problem of dynamic searchable encryption (DSE) with forward-and-backward privacy.Many DSE schemes have been proposed recently but the most efficient ones have one limitation: they require maintaining one operation counter for each unique keyword, either stored locally at the client or stored encrypted at the server and accessed obliviously (e.g., with an oblivious map) during every operation.We propose three new schemes that overcome the above limitation and achieve constant permanent client storage with improved performance, both asymptotically and experimentally, compared to prior stateof-the-art works.In particular, our first two schemes adopt a "static-to-dynamic" transformation which eliminates the need for oblivious accesses during searches.Due to this, they are the first practical schemes with minimal client storage and non-interactive search.Our third scheme is the first quasi-optimal forward-andbackward DSE scheme with only a logarithmic overhead for retrieving the query result (independently of previous deletions).While it does require an oblivious access during search in order to keep permanent client storage minimal, its practical performance is up to four orders of magnitude better than the best existing scheme with quasi-optimal search.
Hybrid Lightweight Proxy Re-Encryption Scheme for Secure Fog-to-Things Environment

Fog computing is a promising paradigm that can mitigate the heavy burden on cloud-central processing of the vast amount of IoT data. Although fog computing has the advantages of low latency, storage, and computing resources that serve IoT applications and things, it hardly suffers from security and privacy challenges. Proxy re-encryption (PRE) is an effective cryptographic solution to ensure the security of fog-to-things communication. However, in PRE schemes, the problem of a processing delay caused by offloading significant computational load to the proxy for re-encryption, and the heavy computation operations of data owner encryption and user decryption due to asymmetric cryptographic use, are still unresolved in the literature. In this paper, we propose a hybrid proxy re-encryption scheme that combines lightweight symmetric and asymmetric encryption algorithms to establish secure communications in fog-to-things computing. In the proposed scheme, the computational cost incurred by fog nodes to carry out the re-encryption process is highly efficient. Meanwhile, the scheme can reduce the encryption and decryption overheads for end-users with resource-constrained devices. Security and performance analyses are conducted, and the results indicate that our scheme is secure, highly efficient, and lightweight.
Speech Sentiment Analysis via Pre-Trained Features from End-to-End ASR Models

In this paper, we propose to use pre-trained features from end-to-end ASR models to solve speech sentiment analysis as a down-stream task. We show that end-to-end ASR features, which integrate both acoustic and text information from speech, achieve promising results. We use RNN with self-attention as the sentiment classifier, which also provides an easy visualization through attention weights to help interpret model predictions. We use well benchmarked IEMOCAP dataset and a new large-scale speech sentiment dataset SWBD-sentiment for evaluation. Our approach improves the-state-of-the-art accuracy on IEMOCAP from 66.6% to 71.7%, and achieves an accuracy of 70.10% on SWBD-sentiment with more than 49,500 utterances.
Optimal distributed quantum sensing using Gaussian states

The authors find the optimal setup of distributed quantum sensing using multimode Gaussian states for estimating the average value of phases encoded in the distributed modes. The results demonstrate that multimode entanglement plays an important role in the precise estimation of a global parameter.
A learnheuristic approach for the team orienteering problem with aerial drone motion constraints

This work proposes a learnheuristic approach (combination of heuristics with machine learning) to solve an aerial-drone team orienteering problem. The goal is to maximise the total reward collected from information gathering or surveillance observations of a set of known targets within a fixed amount of time. The aerial drone team orienteering problem has the complicating feature that the travel times between targets depend on a drone’s flight path between previous targets. This path-dependence is caused by the aerial surveillance drones flying under the influence of air-resistance, gravity, and the laws of motion. Sharp turns slow drones down and the angle of ascent and air-resistance influence the acceleration a drone is capable of. The route dependence of inter-target travel times motivates the consideration of a learnheuristic approach, in which the prediction of travel times is outsourced to a machine learning algorithm. This work proposes an instance-based learning algorithm with interpolated predictions as the learning module. We show that a learnheuristic approach can lead to higher quality solutions in a shorter amount of time than those generated from an equivalent metaheuristic algorithm, an effect attributed to the search-diversity enhancing consequence of the online learning process.
Deep transductive network for generalized zero shot learning

Zero Shot Learning (ZSL) aims to learn projective functions on labeled seen data and transfer the learned functions to unseen classes by discovering their relationship with semantic embeddings. However, the mapping process often suffers from the domain shift problem caused by only using the labeled seen data. In this paper, we propose a novel explainable Deep Transductive Network (DTN) for the task of Generalized ZSL (GZSL) by training on both labeled seen data and unlabeled unseen data, with subsequent testing on both seen classes and unseen classes. The proposed network exploits a KL Divergence constraint to iteratively refine the probability of classifying unlabeled instances by learning from their high confidence assignments with the assistance of an auxiliary target distribution. Besides, to avoid the meaningless ascription assumption of unseen data on GZSL, we also propose an experimental paradigm by splitting the unseen data into two equivalent parts for training and testing respectively. Extensive experiments and detailed analysis demonstrate that our DTN can efficiently handle the problems and achieve the state-of-the-art performance on four popular datasets.
Privacy Preservation in Location-Based Services: A Novel Metric and Attack Model

Recent years have seen rising needs for location-based services in our everyday life.Aside from the many advantages provided by these services, they have caused serious concerns regarding the location privacy of users.Adversaries can monitor the queried locations by users to infer sensitive information, such as home addresses and shopping habits.To address this issue, dummy-based algorithms have been developed to increase the anonymity of users, and thus, protecting their privacy.Unfortunately, the existing algorithms only assume a limited amount of side information known by adversaries, which may face more severe challenges in practice.In this paper, we develop an attack model termed as Viterbi attack, which represents a realistic privacy threat on user trajectories.Moreover, we propose a metric called transition entropy that enables the evaluation of dummy-based algorithms, followed by developing a robust algorithm that can defend users against the Viterbi attack while maintaining significantly high performance in terms of the traditional metrics.We compare and evaluate our proposed algorithm and metric on a publicly available dataset published by Microsoft, i.e., Geolife dataset.
Urdu-Text Detection and Recognition in Natural Scene Images Using Deep Learning

Urdu text is a cursive script and belongs to a non-Latin family of other cursive scripts like Arabic, Chinese, and Hindi. Urdu text poses a challenge for detection/localization from natural scene images, and consequently recognition of individual ligatures in scene images. In this paper, a methodology is proposed that covers detection, orientation prediction, and recognition of Urdu ligatures in outdoor images. As a first step, the custom FasterRCNN algorithm has been used in conjunction with well-known CNNs like Squeezenet, Googlenet, Resnet18, and Resnet50 for detection and localization purposes for images of size 320 × 240 pixels. For ligature Orientation prediction, a custom Regression Residual Neural Network (RRNN) is trained/tested on datasets containing randomly oriented ligatures. Recognition of ligatures was done using Two Stream Deep Neural Network (TSDNN). In our experiments, five-set of datasets, containing 4.2K and 51K Urdu-text-embedded synthetic images were generated using the CLE annotation text to evaluate different tasks of detection, orientation prediction, and recognition of ligatures. These synthetic images contain 132, and 1600 unique ligatures corresponding to 4.2K and 51K images respectively, with 32 variations of each ligature (4-backgrounds and font 8-color variations). Also, 1094 real-world images containing more than 12k Urdu characters were used for TSDNN's evaluation. Finally, all four detectors were evaluated and used to compare them for their ability to detect/localize Urdu-text using average-precision (AP). Resnet50 features based FasterRCNN was found to be the winner detector with AP of.98. While Squeeznet, Googlenet, Resnet18 based detectors had testing AP of.65, .88, and .87 respectively. RRNN achieved and accuracy of 79% and 99% for 4k and 51K images respectively. Similarly, for characters classification in ligatures, TSDNN attained a partial sequence recognition rate of 94.90% and 95.20% for 4k and 51K images respectively. Similarly, a partial sequence recognition rate of 76.60% attained for real world-images.
Composition-Aware Image Aesthetics Assessment

Automatic image aesthetics assessment is important for a wide variety of applications such as on-line photo suggestion, photo album management and image retrieval. Previous methods have focused on mapping the holistic image content to a high or low aesthetics rating. However, the composition information of an image characterizes the harmony of its visual elements according to the principles of art, and provides richer information for learning aesthetics. In this work, we propose to model the image composition information as the mutual dependency of its local regions, and design a novel architecture to leverage such information to boost the performance of aesthetics assessment. To achieve this, we densely partition an image into local regions and compute aesthetics-preserving features over the regions to characterize the aesthetics properties of image content. With the feature representation of local regions, we build a region composition graph in which each node denotes one region and any two nodes are connected by an edge weighted by the similarity of the region features. We perform reasoning on this graph via graph convolution, in which the activation of each node is determined by its highly correlated neighbors. Our method naturally uncovers the mutual dependency of local regions in the network training procedure, and achieves the state-of-the-art performance on the benchmark visual aesthetics datasets.