Reviewerly/exHarmony huggingface dataset:
Multimodal Sentiment Analysis via RNN variants

Multimodal sentiment analysis involves the classification of sentiment by using different forms of data together, namely, text, audio, and video. Previously sentiment analysis was implemented only on textual data. Multimodal sentiment analysis relies mainly on identifying the utterances present in the video and using them as a basis for sentiment classification. In this paper, we proposed four different variants of RNN, namely, GRNN, LRNN, GLRNN and UGRNN for analyzing the utterances of the speakers from the videos. Experimental results on CMI-MOSI dataset demonstrates that our approach is able to achieve better sentiment classification accuracy on individual modality (text, audio, and video) than existing approaches on the same dataset. Moreover, our method also gave decent results after fusing the individual modality using Attention networks for multimodal sentiment analysis.
Not All Frames Are Equal: Weakly-Supervised Video Grounding With Contextual Similarity and Visual Clustering Losses

We invest the problem of weakly-supervised video grounding, where only video-level sentences are provided. This is a challenging task, and previous Multi-Instance Learning (MIL) based image grounding methods turn to fail in the video domain. Recent work attempts to decompose the video-level MIL into frame-level MIL by applying weighted sentence-frame ranking loss over frames, but it is not robust and does not exploit the rich temporal information in videos. In this work, we address these issues by extending frame-level MIL with a false positive frame-bag constraint and modeling the visual feature consistency in the video. In specific, we design a contextual similarity between semantic and visual features to deal with sparse objects association across frames. Furthermore, we leverage temporal coherence by strengthening the clustering effect of similar features in the visual space. We conduct an extensive evaluation on YouCookII and RoboWatch datasets, and demonstrate our method significantly outperforms prior state-of-the-art methods.
Velody

Biometrics have been widely adopted for enhancing user authentication, benefiting usability by exploiting pervasive and collectible unique characteristics from physiological or behavioral traits of human. However, successful attacks on "static" biometrics such as fingerprints have been reported where an adversary acquires users' biometrics stealthily and compromises non-resilient biometrics.
Anatomical context protects deep learning from adversarial perturbations in medical imaging

Deep learning has achieved impressive performance across a variety of tasks, including medical image processing. However, recent research has shown that deep neural networks are susceptible to small adversarial perturbations in the image. We study the impact of such adversarial perturbations in medical image processing where the goal is to predict an individual’s age based on a 3D MRI brain image. We consider two models: a conventional deep neural network, and a hybrid deep learning model which additionally uses features informed by anatomical context. We find that we can introduce significant errors in predicted age by adding imperceptible noise to an image, can accomplish this even for large batches of images using a single perturbation, and that the hybrid model is much more robust to adversarial perturbations than the conventional deep neural network. Our work highlights limitations of current deep learning techniques in clinical applications, and suggests a path forward.
Normalized Wasserstein for Mixture Distributions With Applications in Adversarial Learning and Domain Adaptation

Understanding proper distance measures between distributions is at the core of several learning tasks such as generative models, domain adaptation, clustering, etc. In this work, we focus on mixture distributions that arise naturally in several application domains where the data contains different sub-populations. For mixture distributions, established distance measures such as the Wasserstein distance do not take into account imbalanced mixture proportions. Thus, even if two mixture distributions have identical mixture components but different mixture proportions, the Wasserstein distance between them will be large. This often leads to undesired results in distance-based learning methods for mixture distributions. In this paper, we resolve this issue by introducing the Normalized Wasserstein measure. The key idea is to introduce mixture proportions as optimization variables, effectively normalizing mixture proportions in the Wasserstein formulation. Using the proposed normalized Wasserstein measure leads to significant performance gains for mixture distributions with imbalanced mixture proportions compared to the vanilla Wasserstein distance. We demonstrate the effectiveness of the proposed measure in GANs, domain adaptation and adversarial clustering in several benchmark datasets.
AFD-Net: Aggregated Feature Difference Learning for Cross-Spectral Image Patch Matching

Image patch matching across different spectral domains is more challenging than in a single spectral domain. We consider the reason is twofold: 1. the weaker discriminative feature learned by conventional methods; 2. the significant appearance difference between two images domains. To tackle these problems, we propose an aggregated feature difference learning network (AFD-Net). Unlike other methods that merely rely on the high-level features, we find the feature differences in other levels also provide useful learning information. Thus, the multi-level feature differences are aggregated to enhance the discrimination. To make features invariant across different domains, we introduce a domain invariant feature extraction network based on instance normalization (IN). In order to optimize the AFD-Net, we borrow the large margin cosine loss which can minimize intra-class distance and maximize inter-class distance between matching and non-matching samples. Extensive experiments show that AFD-Net largely outperforms the state-of-the-arts on the cross-spectral dataset, meanwhile, demonstrates a considerable generalizability on a single spectral dataset.
Efficient approach for incremental weighted erasable pattern mining with list structure

Erasable pattern mining is one of the important fields of frequent pattern mining. It diagnoses and solves the economic problems that arise in the manufacturing industry. The real-world database is continually accumulated over time, and each item has a different importance. Therefore, if we use conventional erasable pattern mining without considering the characteristics of the real-world database, less meaningful patterns can be extracted. Also, when mining a real-world database, the algorithm must be able to process operations quickly and efficiently. In this paper, in order to meet these requirements, we propose an algorithm which is implemented as a list structure for mining erasable patterns in an incremental database with weighted condition. Compared to existing state-of-the-art mining algorithms, the proposed algorithm performs pattern pruning by applying weighted condition to a dynamic database, so it extracts fewer candidate patterns and shows fast performance. We test our algorithms and the algorithms previously presented with various real datasets and synthetic datasets and obtained results such as run time, memory usage, scalability, and accuracy tests. By analyzing and comparing these experimental results, we show that the proposed algorithm has outstanding performance.
Super-Resolution of Remote Sensing Images via a Dense Residual Generative Adversarial Network

Single image super-resolution (SISR) has been widely studied in recent years as a crucial technique for remote sensing applications. In this paper, a dense residual generative adversarial network (DRGAN)-based SISR method is proposed to promote the resolution of remote sensing images. Different from previous super-resolution (SR) approaches based on generative adversarial networks (GANs), the novelty of our method mainly lies in the following factors. First, we made a breakthrough in terms of network architecture to improve performance. We designed a dense residual network as the generative network in GAN, which can make full use of the hierarchical features from low-resolution (LR) images. We also introduced a contiguous memory mechanism into the network to take advantage of the dense residual block. Second, we modified the loss function and altered the model of the discriminative network according to the Wasserstein GAN with a gradient penalty (WGAN-GP) for stable training. Extensive experiments were performed using the NWPU-RESISC45 dataset, and the results demonstrated that the proposed method outperforms state-of-the-art methods in terms of both objective evaluation and subjective perspective.
A Local Adaptive Minority Selection and Oversampling Method for Class-Imbalanced Fault Diagnostics in Industrial Systems

Data-driven fault diagnostics of industrial systems suffer from class-imbalanced problems, which is a common challenge for machine learning algorithms as it is difficult to learn the features of the minority class samples. Synthetic oversampling methods are commonly used to tackle these problems by generating minority class samples to balance the majority and minority classes. Two major issues will influence the performance of oversampling methods which are how to choose the most appropriate existing minority seed samples, and how to synthesize new samples from seed samples effectively. However, many existing oversampling methods are not accurate and effective enough to generate new samples when dealing with high-dimensional faulty samples with different imbalanced ratios, since they do not take these two factors into consideration at the same time. This article develops a novel adaptive oversampling technique: expectation maximization (EM)-based local-weighted minority oversampling technique for industrial fault diagnostics. This method uses a local-weighted minority oversampling strategy to identify hard-to-learn informative minority fault samples and an EM-based imputation algorithm to generate fault samples based on the distribution of minority samples. To validate the performance of the developed method, experiments were conducted on two real-world datasets. The results show that the developed method can achieve better performances, in terms of F-measure, Matthews correlation coefficient (MCC), and Mean (average of F-measure and MCC) values, on multiclass imbalanced fault diagnostics in different imbalance ratios than state-of-arts' baseline sampling techniques.
Addressing Application Latency Requirements through Edge Scheduling

Abstract Latency-sensitive and data-intensive applications, such as IoT or mobile services, are leveraged by Edge computing, which extends the cloud ecosystem with distributed computational resources in proximity to data providers and consumers. This brings significant benefits in terms of lower latency and higher bandwidth. However, by definition, edge computing has limited resources with respect to cloud counterparts; thus, there exists a trade-off between proximity to users and resource utilization. Moreover, service availability is a significant concern at the edge of the network, where extensive support systems as in cloud data centers are not usually present. To overcome these limitations, we propose a score-based edge service scheduling algorithm that evaluates network, compute, and reliability capabilities of edge nodes. The algorithm outputs the maximum scoring mapping between resources and services with regard to four critical aspects of service quality. Our simulation-based experiments on live video streaming services demonstrate significant improvements in both network delay and service time. Moreover, we compare edge computing with cloud computing and content delivery networks within the context of latency-sensitive and data-intensive applications. The results suggest that our edge-based scheduling algorithm is a viable solution for high service quality and responsiveness in deploying such applications.
Reconfigurable Pavement Sweeping Robot and Pedestrian Cohabitant Framework by Vision Techniques

The periodic cleaning pavement is a requirement for a clean environment in urban areas. The robot named Panthera introduced in this paper has the ability to change the width of the frame to help the cleaning tasks become suitable for different pavement types and friendly with the activities of the pedestrian. The Panthera cleaning operations which allow pedestrian walks freely on the pavement is modeled as a pedestrian robot cohabitant framework. To this end, the mask based deep convolutional neural network (DCNN) is used to archive segmented label maps of pedestrians in color image, then the distance from detected objects to the robot is estimated and tracked by averaging filtered depth values in the corresponding region in the refined depth image. The width and the distance from the robot to the approaching pedestrians are used to adjust the robot width. The enlarging and squeezing operations of the Panthera width are conducted by rotating one motor to change the length of the lead screw rod and the angle linkage hinges from the information of encoders sensors. The experiments carried out on real environments demonstrated the autonomous avoiding pedestrians ability by the kinematic model of Panthera.
Fully Convolutional Pixel Adaptive Image Denoiser

We propose a new image denoising algorithm, dubbed as Fully Convolutional Adaptive Image DEnoiser (FC-AIDE), that can learn from an offline supervised training set with a fully convolutional neural network as well as adaptively fine-tune the supervised model for each given noisy image. We significantly extend the framework of the recently proposed Neural AIDE, which formulates the denoiser to be context-based pixelwise mappings and utilizes the unbiased estimator of MSE for such denoisers. The two main contributions we make are; 1) implementing a novel fully convolutional architecture that boosts the base supervised model, and 2) introducing regularization methods for the adaptive fine-tuning such that a stronger and more robust adaptivity can be attained. As a result, FC-AIDE is shown to possess many desirable features; it outperforms the recent CNN-based state-of-the-art denoisers on all of the benchmark datasets we tested, and gets particularly strong for various challenging scenarios, e.g., with mismatched image/noise characteristics or with scarce supervised training data. The source code our algorithm is available at https://github.com/csm9493/FC-AIDE-Keras.
SOINN+, a Self-Organizing Incremental Neural Network for Unsupervised Learning from Noisy Data Streams

The goal of continuous learning is to acquire and fine-tune knowledge incrementally without erasing already existing knowledge. How to mitigate this erasure, known as catastrophic forgetting, is a grand challenge for machine learning, specifically when systems are trained on evolving data streams. Self-organizing incremental neural networks (SOINN) are a class of neural networks that are designed for continuous learning from non-stationary data. Here, we propose a novel method, SOINN+, which is fundamentally different from the previous generations of SOINN with respect to how "forgetting" is modeled. To achieve a more graceful forgetting, we developed three new concepts: idle time, trustworthiness, and unutility of a node. SOINN+ learns a topology-preserving mapping of the input data to a network structure, which reveals clusters of arbitrary shapes in streams of noisy data. Our experiments with synthetic and real-world data sets showed that SOINN+ can maintain discovered structures even under sudden and recurring concept drifts.
Why Does a Visual Question Have Different Answers?

Visual question answering is the task of returning the answer to a question about an image. A challenge is that different people often provide different answers to the same visual question. To our knowledge, this is the first work that aims to understand why. We propose a taxonomy of nine plausible reasons, and create two labelled datasets consisting of ~45,000 visual questions indicating which reasons led to answer differences. We then propose a novel problem of predicting directly from a visual question which reasons will cause answer differences as well as a novel algorithm for this purpose. Experiments demonstrate the advantage of our approach over several related baselines on two diverse datasets. We publicly share the datasets and code at https://vizwiz.org.
Monocular person tracking and identification with on-line deep feature selection for person following robots

This paper presents a new person tracking and identification framework based on solely a monocular camera. In this framework, we first track persons in the robot coordinate space using Unscented Kalman filter with the ground plane information and human height estimation. Then, we identify the target person to be followed with the combination of Convolutional Channel Features (CCF) and online boosting. It allows us to take advantage of deep neural network-based feature representation while adapting the person classifier to a specific target person depending on the circumstances. The entire system can be run on a recent embedded computation board with a GPU (NVIDIA Jetson TX2), and it can easily be reproduced and reused on a new mobile robot platform. Through evaluations, we validated that the proposed method outperforms existing person identification methods for mobile robots. We applied the proposed method to a real person following robot, and it has been shown that CCF-based person identification realizes robust person following in both indoor and outdoor environments.
An optimal interval type-2 fuzzy logic control based closed-loop drug administration to regulate the mean arterial blood pressure

The main aim of this work is to present an optimal and robust controller design in order to improve the drug infusion to the automatic control of mean arterial blood pressure in conditions like critically-ill or post-operative or anaesthesia administration. The physiological systems also have uncertainty issues such as parameter variations with time or external disturbances and noise. Therefore, a controlled drug administration is necessary to regulate the mean arterial blood pressure of a person during surgery/observation. Over the years, the proportional-integral-derivative (PID) controller is the most commonly used controller in industries due to its easy structure and simplicity. However, this controller does not meet the desired performance with the complex and uncertain plants. Therefore, a robust controller is required to regulate the physiological variables that are uncertain in nature and can affect the human life.In this work, a hybrid control scheme consisting of an interval type-2-fuzzy logic controller which acts as pre-compensator to the traditional PID controller is presented, to regulate the mean arterial blood pressure of a patient by administering the drug sodium nitroprusside in a controlled manner. An effective and well-established nature-inspired optimization technique namely cuckoo search algorithm is employed for obtaining the optimal parameters for the presented scheme.Simulation results are presented to show the effectiveness and robustness of proposed interval type-2-fuzzy logic controller based PID controller scheme, for maintaining the mean arterial pressure to 100 mmHg within considerable limit through SNP infusion. The results are further compared with other two controllers namely type-1 fuzzy logic based PID and traditional PID controllers for the parameter variations and external noise.In this study, the proposed interval type-2-fuzzy logic controller pre-compensator based PID controller provides an effective control than traditional type-1 fuzzy logic based control scheme and PID controller in terms of overshoot, settling-time and error which are the prime performance objectives of the closed-loop controlled drug delivery of human blood pressure. The presented study provides a firm base for initial design considerations for development of a low-cost closed-loop drug delivery system for blood pressure regulation.
Breast Ultrasound Image Synthesis using Deep Convolutional Generative Adversarial Networks

Deep convolutional generative adversarial networks (DCGANs) are newly developed tools for generating synthesized images. To determine the clinical utility of synthesized images, we generated breast ultrasound images and assessed their quality and clinical value. After retrospectively collecting 528 images of 144 benign masses and 529 images of 216 malignant masses in the breasts, synthesized images were generated using a DCGAN with 50, 100, 200, 500, and 1000 epochs. The synthesized (n = 20) and original (n = 40) images were evaluated by two radiologists, who scored them for overall quality, definition of anatomic structures, and visualization of the masses on a five-point scale. They also scored the possibility of images being original. Although there was no significant difference between the images synthesized with 1000 and 500 epochs, the latter were evaluated as being of higher quality than all other images. Moreover, 2.5%, 0%, 12.5%, 37.5%, and 22.5% of the images synthesized with 50, 100, 200, 500, and 1000 epochs, respectively, and 14% of the original images were indistinguishable from one another. Interobserver agreement was very good (|r| = 0.708-0.825, p < 0.001). Therefore, DCGAN can generate high-quality and realistic synthesized breast ultrasound images that are indistinguishable from the original images.
Delay-Based Reservoir Computing Using Multimode Semiconductor Lasers: Exploiting the Rich Carrier Dynamics

Delay-based reservoir computing is a neuromorphic-computing technique employing a single nonlinear node, in this case a semiconductor laser subjected to a delayed optical self-feedback. This implementation of reservoir computing, while being versatile and highly performant, is limited in processing speed by the delay length. In an effort to reduce the required delay line length, while maintaining the same strong performance, we investigate if it is possible to distribute the processing power over more than one longitudinal mode. We therefore have numerically analysed a rate equation model of a semiconductor laser with two longitudinal modes, delayed optical feedback and optical injection. Our modelling takes into account the carrier inversion moments that represent carrier gratings formed in the cavity during operation. This allows us to study the importance of modal interaction via carrier gratings and how this effects the computational power. We show that the setup with a dual-mode laser and an optimized reservoir parameter space, can reduce the delay-line fourfold in comparison with a single-mode laser setup.
Comparison of Haar-like, HOG and LBP approaches for face detection in video sequences

Face detection is an essential part of any face recognition system as a first step to detect faces. This paper presents a comparative study of three commonly used approaches for face detection, namely Haar-like cascade, Histogram of Oriented Gradients with Support Vector Machine and Linear Binary Pattern cascade. For this aim, video sequences from the Database for Emotion Analysis using Physiological Signals (DEAP) were explored. The proposed methods were developed using Python language with OpenCV and Dlib libraries. The obtained results show that HOG+SVM approach is more robust and accurate than LBP and Haar approaches with an average detection rate of 92.68%.