Reviewerly/exHarmony huggingface dataset:
SATSal: A Multi-Level Self-Attention Based Architecture for Visual Saliency Prediction

Human visual Attention modelling is a persistent interdisciplinary research challenge, gaining new interest in recent years mainly due to the latest developments in deep learning. That is particularly evident in saliency benchmarks. Novel deep learning-based visual saliency models show promising results in capturing high-level (top-down) human visual attention processes. Therefore, they strongly differ from the earlier approaches, mainly characterised by low-level (bottom-up) visual features. These developments account for innate human selectivity mechanisms that are reliant on both high- and low-level factors. Moreover, the two factors interact with each other. Motivated by the importance of these interactions, in this project, we tackle visual saliency modelling holistically, examining if we could consider both high- and low-level features that govern human attention. Specifically, we propose a novel method SAtSal (Self-Attention Saliency). SAtSal leverages both high and low-level features using a multilevel merging of skip connections during the decoding stage. Consequently, we incorporate convolutional self-attention modules on skip connection from the encoder to the decoder network to properly integrate the valuable signals from multilevel spatial features. Thus, the self-attention modules learn to filter out the latent representation of the salient regions from the other irrelevant information in an embedded and joint manner with the main encoder-decoder model backbone. Finally, we evaluate SAtSal against various existing solutions to validate our approach, using the well-known standard saliency benchmark MIT300. To further examine SAtSal’s robustness on other image types, we also evaluate it on the Le-Meur saliency painting benchmark.
Multi-channel fusion LSTM for medical event prediction using EHRs

Automatic medical event prediction (MEP), e.g. diagnosis prediction, medication prediction, using electronic health records (EHRs) is a popular research direction in health informatics. In many cases, MEP relies on the determinations from different types of medical events, which demonstrates the heterogeneous nature of EHRs. However, most existing methods for MEP fail to distinguishingly model the type of event that is highly associated with the prediction task, i.e. task-wise event, which usually plays a more significant role than other events. In this paper, we proposed a Long Short-Term Memory network (LSTM)-based method for MEP, named Multi-Channel Fusion LSTM (MCF-LSTM), which models the correlations between different types of medical events using multiple network channels. To this end, we designed a task-wise fusion module, in which a gated network is applied to select how much information can be transferred between events. Furthermore, the irregular temporal interval between adjacent medical visits is also modeled in an individual channel, which is combined with other events in a unified manner. We compared MCF-LSTM with state-of-the-art methods on four MEP tasks on two public datasets: MIMIC-III and eICU. Experimental results show that MCF-LSTM achieves promising results on AUC(receiver operating characteristic curve), AUPR (area under the precision-recall curve), and top-k recall, and outperforms other methods with high stability.
Meta Approach to Data Augmentation Optimization

Data augmentation policies drastically improve the performance of image recognition tasks, especially when the policies are optimized for the target data and tasks. In this paper, we propose to optimize image recognition models and data augmentation policies simultaneously to improve the performance using gradient descent. Unlike prior methods, our approach avoids using proxy tasks or reducing search space, and can directly improve the validation performance. Our method achieves efficient and scalable training by approximating the gradient of policies by implicit gradient with Neumann series approximation. We demonstrate that our approach can improve the performance of various image classification tasks, including fine-grained image recognition, without using dataset-specific hyperparameter tuning.
Proactive Anomaly Detection for Robot Navigation With Multi-Sensor Fusion

Despite the rapid advancement of navigation algorithms, mobile robots often produce anomalous behaviors that can lead to navigation failures. The ability to detect such anomalous behaviors is a key component in modern robots to achieve high-levels of autonomy. Reactive anomaly detection methods identify anomalous task executions based on the current robot state and thus lack the ability to alert the robot before an actual failure occurs. Such an alert delay is undesirable due to the potential damage to both the robot and the surrounding objects. We propose a proactive anomaly detection network (PAAD) for robot navigation in unstructured and uncertain environments. PAAD predicts the probability of future failure based on the planned motions from the predictive controller and the current observation from the perception module. Multi-sensor signals are fused effectively to provide robust anomaly detection in the presence of sensor occlusion as seen in field environments. Our experiments on field robot data demonstrates superior failure identification performance than previous methods, and that our model can capture anomalous behaviors in real-time while maintaining a low false detection rate in cluttered fields. Code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/tianchenji/PAAD</uri> .
When do GANs replicate? On the choice of dataset size

Do GANs replicate training images? Previous studies have shown that GANs do not seem to replicate training data without significant change in the training procedure. This leads to a series of research on the exact condition needed for GANs to overfit to the training data. Although a number of factors has been theoretically or empirically identified, the effect of dataset size and complexity on GANs replication is still unknown. With empirical evidence from BigGAN and StyleGAN2, on datasets CelebA, Flower and LSUN-bedroom, we show that dataset size and its complexity play an important role in GANs replication and perceptual quality of the generated images. We further quantify this relationship, discovering that replication percentage decays exponentially with respect to dataset size and complexity, with a shared decaying factor across GAN-dataset combinations. Meanwhile, the perceptual image quality follows a U-shape trend w.r.t dataset size. This finding leads to a practical tool for one-shot estimation on minimal dataset size to prevent GAN replication which can be used to guide datasets construction and selection.
Auto Graph Encoder-Decoder for Neural Network Pruning

Model compression aims to deploy deep neural networks (DNN) on mobile devices with limited computing and storage resources. However, most of the existing model compression methods rely on manually defined rules, which require domain expertise. DNNs are essentially computational graphs, which contain rich structural information. In this paper, we aim to find a suitable compression policy from DNNs’ structural information. We propose an automatic graph encoder-decoder model compression (AGMC) method combined with graph neural networks (GNN) and reinforcement learning (RL). We model the target DNN as a graph and use GNN to learn the DNN’s embeddings automatically. We compared our method with rule-based DNN embedding model compression methods to show the effectiveness of our method. Results show that our learning-based DNN embedding achieves better performance and a higher compression ratio with fewer search steps. We evaluated our method on over-parameterized and mobile-friendly DNNs and compared our method with handcrafted and learning-based model compression approaches. On over parameterized DNNs, such as ResNet-56, our method outperformed handcrafted and learning-based methods with 4.36% and 2.56% higher accuracy, respectively. Furthermore, on MobileNet-v2, we achieved a higher compression ratio than state-of-the-art methods with just 0.93% accuracy loss.
SIGNET: Efficient Neural Representation for Light Fields

We present a novel neural representation for light field content that enables compact storage and easy local reconstruction with high fidelity. We use a fully-connected neural network to learn the mapping function between each light field pixel’s coordinates and its corresponding color values. Since neural networks that simply take in raw coordinates are unable to accurately learn data containing fine details, we present an input transformation strategy based on the Gegenbauer polynomials, which previously showed theoretical advantages over the Fourier basis. We conduct experiments that show our Gegenbauer-based design combined with sinusoidal activation functions leads to a better light field reconstruction quality than a variety of network designs, including those with Fourier-inspired techniques introduced by prior works. Moreover, our SInusoidal Gegenbauer NETwork, or SIGNET, can represent light field scenes more compactly than the state-of-the-art compression methods while maintaining a comparable reconstruction quality. SIGNET also innately allows random access to encoded light field pixels due to its functional design. We further demonstrate that SIGNET’s super-resolution capability without any additional training.
Fuzziness based semi-supervised multimodal learning for patient’s activity recognition using RGBDT videos

Automatic recognition of bedridden patients’ physical activity has important applications in the clinical process. Such recognition tasks are usually accomplished on visual data captured by RGB, depth, and/or thermal cameras by utilizing supervised machine learning. However, supervised machine learning requires a large amount of labeled data and the accuracy depends on extracting appropriate features based on the domain knowledge. A plausible solution to these issues is using semi-supervised learning that focuses less on labeled data and domain knowledge. In this paper, a novel fuzziness-based semi-supervised multimodal learning algorithm, called (FSSL-PAR) is proposed for bedridden patient activity recognition. We use a synergistic interaction on RGB, Depth, and Thermal videos to assess the effect of visual multimodality for the first time in this semi-supervised learning setting. Experiments are conducted on a dataset collected by mimicking the patients with Acute Brain Injury (ABI) from a neurorehabilitation center. The results exhibit the superiority of the proposed algorithm over the existing supervised learning algorithms. We also see a positive correlation between the performance and the size of the labeled data in the proposed FSSL-PAR.
Asynchronous generative adversarial network for asymmetric unpaired image-to-image translation

The unpaired image-to-image translation aims to translate input images from one source domain to some desired outputs in a target domain by learning from unpaired training data. Cycle-consistency constraint provides a general principle to estimate and measure forward and backward mapping functions between two domains. In many cases, the information entropy of images from the two domains is not equal, resulting in an information-rich domain and an information-poor domain. However, existing solutions based on cycle-consistency either completely discard the information asymmetry between the two domains (a common choice), which leads to inferior translation performance for the asymmetric unpaired image-to-image translation, or have to rely on special task-specific designs and introduce extra loss components. These elaborative designs especially for the relatively harder translation direction from the information-poor domain to the information-rich domain (poor-to-rich translation) require extra labor and are limited to some specific tasks. In this paper, we propose a novel asynchronous generative adversarial network named Async-GAN, which provides a model-agnostic framework for easily turning symmetrical models into powerful asymmetric counterparts that can handle asymmetric unpaired image-to-image translation much better. The key innovation is to iteratively build gradually improving intermediate domains for generating pseudo paired training samples, which provide stronger full supervision for assisting the poor-to-rich translation. Extensive experiments on various asymmetric unpaired translation tasks demonstrate the superiority of the proposal. Furthermore, the proposed training framework could be extended to various Cycle-GAN solutions and achieve a performance gain.
Self-Supervised Real-to-Sim Scene Generation

Synthetic data is emerging as a promising solution to the scalability issue of supervised deep learning, especially when real data are difficult to acquire or hard to annotate. Synthetic data generation, however, can itself be prohibitively expensive when domain experts have to manually and painstakingly oversee the process. More-over, neural networks trained on synthetic data often do not perform well on real data because of the domain gap. To solve these challenges, we propose Sim2SG, a self-supervised automatic scene generation technique for matching the distribution of real data. Importantly, Sim2SG does not require supervision from the real-world dataset, thus making it applicable in situations for which such annotations are difficult to obtain. Sim2SG is designed to bridge both the content and appearance gaps, by matching the content of real data, and by matching the features in the source and target domains. We select scene graph (SG) generation as the downstream task, due to the limited availability of labeled datasets. Experiments demonstrate significant improvements over leading baselines in reducing the domain gap both qualitatively and quantitatively, on several synthetic datasets as well as the real-world KITTI dataset.
Gravity-Aware Monocular 3D Human-Object Reconstruction

This paper proposes GraviCap, i.e., a new approach for joint markerless 3D human motion capture and object trajectory estimation from monocular RGB videos. We focus on scenes with objects partially observed during a free flight. In contrast to existing monocular methods, we can recover scale, object trajectories as well as human bone lengths in meters and the ground plane's orientation, thanks to the awareness of the gravity constraining object motions. Our objective function is parametrised by the object's initial velocity and position, gravity direction and focal length, and jointly optimised for one or several free flight episodes. The proposed human-object interaction constraints ensure geometric consistency of the 3D reconstructions and improved physical plausibility of human poses compared to the unconstrained case. We evaluate GraviCap on a new dataset with ground-truth annotations for persons and different objects undergoing free flights. In the experiments, our approach achieves state-of-the-art accuracy in 3D human motion capture on various metrics. We urge the reader to watch our supplementary video. Both the source code and the dataset are released; see http://4dqv.mpi-inf.mpg.de/GraviCap/.
Exploration and Estimation for Model Compression

Deep neural networks achieve great success in many visual recognition tasks. However, the model deployment is usually subject to some computational resources. Model pruning under computational budget has attracted growing attention. In this paper, we focus on the discrimination-aware compression of Convolutional Neural Networks (CNNs). In prior arts, directly searching the optimal sub-network is an integer programming problem, which is non-smooth, non-convex, and NP-hard. Meanwhile, the heuristic pruning criterion lacks clear interpretability and doesn’t generalize well in applications. To address this problem, we formulate sub-networks as samples from a multivariate Bernoulli distribution and resort to the approximation of continuous problem. We propose a new flexible search scheme via alternating exploration and estimation. In the exploration step, we employ stochastic gradient Hamiltonian Monte Carlo with budget-awareness to generate sub-networks, which allows large search space with efficient computation. In the estimation step, we deduce the sub-network sampler to a near-optimal point, to promote the generation of high-quality sub-networks. Unifying the exploration and estimation, our approach avoids early falling into local minimum via a fast gradient-based search in a larger space. Extensive experiments on CIFAR-10 and ImageNet show that our method achieves state-of-the-art performances on pruning several popular CNNs.
Detect Larger at Once: Large-Area Remote-Sensing Image Arbitrary-Oriented Ship Detection

Ship detection is one of the main problems of satellite image analysis. Since ships are scattered on the sea and major ports, large-area remote-sensing images need to be processed in order to realize the detection of ships. In addition, since the satellite is a top-down view, the ship with aspect ratios cannot be covered in complex backgrounds by a horizontal bounding box very well and need a rotating bounding box to achieve this task. Although considerable progress has been made in object detection techniques, there are still challenges for fast detection of ships in large-area remote-sensing images. In this letter, an arbitrary-oriented detector for large-area remote-sensing images is proposed to quickly locate ship positions. A new feature extraction network DCNDarknet25 based on you only look once (YOLO) is designed by reducing paraments and adding deformable convolution (DCN) to improve the speed and accuracy. And the rotation detection capability without angle regression is added to the YOLO detection algorithm for the first time. Finally, thanks to the advantages of our fully convolutional lightweight network, a method for detecting large-area remote-sensing images at once is proposed. In the public dataset HRSC2016 and our own large-area remote-sensing (LARS) image dataset, it has achieved very good accuracy and several times the speed of other algorithms.
ProFlip: Targeted Trojan Attack with Progressive Bit Flips

The security of Deep Neural Networks (DNNs) is of great importance due to their employment in various safety-critical applications. DNNs are shown to be vulnerable against the Trojan attack that manipulates model parameters via poisoned training and gets activated by the pre-defined trigger during inference. In this work, we present ProFlip, the first targeted Trojan attack framework that can divert the prediction of the DNN to the target class by progressively identifying and flipping a small set of bits in model parameters. At its core, ProFlip consists of three key phases: (i) Determining significant neurons in the last layer; (ii) Generating an effective trigger pattern for the tar-get class; (iii) Identifying a sequence of susceptible bits of DNN parameters stored in the main memory (e.g., DRAM). After model deployment, the adversary can insert the Trojan by flipping the critical bits found by ProFlip using bit flip techniques such as Row Hammer or laser beams. As the result, the altered DNN predicts the target class when the trigger pattern is present in any inputs. We perform extensive evaluations of ProFlip on CIFAR10, SVHN, and ImageNet datasets with ResNet-18 and VGG-16 architectures. Empirical results show that, to reach an attack success rate (ASR) of over 94%, ProFlip requires only 12 bit flips out of 88 million parameter bits for ResNet-18 with CIFAR-10, and 15 bit flips for ResNet-18 with ImageNet. Compared to the SOTA, ProFlip reduces the number of required bits flips by 28× ∼ 34× while reaching the same or higher ASR.
Sentiment Analysis on IMDB Movie Reviews using Machine Learning and Deep Learning Algorithms

Sentiment analysis is the study, to classify the text based on customer reviews which can provide valuable information to improve business. Previously the analysis was carried out based on the information provided by the customers using natural language processing and machine learning. In this paper, sentiment analysis on IMDB movie reviews dataset is implemented using Machine Learning (ML) and Deep Learning (DL) approaches to measure the accuracy of the model. ML algorithms are the traditional algorithms that work in a single layer while deep learning algorithms work on multilayers and gives better output. This paper helps the researchers to identify the best algorithm for sentiment analysis. The comparison of the machine learning and deep learning approaches shows that DL algorithms provide accurate and efficient results.
Cloud Affected Solar UV Prediction With Three-Phase Wavelet Hybrid Convolutional Long Short-Term Memory Network Multi-Step Forecast System

Harmful exposure to erythemally-effective ultraviolet radiation (UVR) poses high health risks such as malignant keratinocyte cancers and eye-related diseases.Delivering short-term forecasts of the solar ultraviolet index (UVI) is an effective way to advise UVR exposure information to the public at risk.This research reports on a novel framework built to forecast UVI, integrating antecedent lagged memory of cloud statistical properties and the solar zenith angle (SZA).To produce the forecasts at multi-step horizon we design a 3-phase hybrid convolutional long short-term memory network (W-O-convLSTM) model, validated with Queensland-based datasets in near real-time (i.e., 10-minute, 20-minute, 30-minute and 1 hour forecast horizon).Our approach in optimizing the performance also entails a robust selective filtering method using the BorutaShap algorithm, data decomposition with stationary wavelet transformation and hyperparameter optimization using the Optuna algorithm.We assess the performance of the proposed W-O-convLSTM model alongside the baseline and benchmark models.The captured results, through statistical metrics and visual infographics, elucidate the superior performance of the objective model in short-term UVI forecasting.For instance, at a 10-minute forecast horizon, our objective model yields a relatively high correlation coefficient of ∼0.961 in the autumn, 0.909 in the summer, 0.926 in the spring and 0.936 in the winter season.Overall, the proposed O-convLSTM model outperforms its competing counterpart models for all forecast horizons with the lowest absolute forecast error.The robustness of our newly proposed model avers its practical utility in delivering sun-protection behavior recommendations that can mitigate UV-exposure-related public health risk.We also recommend that future integration of aerosol and ozone effects with cloud cover data can enhance our forecasting framework for wider applications in solar energy or skin health monitoring systems.
Deep retinex decomposition network for underwater image enhancement

This paper introduces a deep retinex decomposition network for underwater image enhancement to conquer the color imbalance, blurring, low contrast, etc. Specifically, we first designed a novel convolutional neural network to estimate the illumination and get reflectance. Then we changed the general idea of processing low light enhancement based on retinex, we perform color balance and illumination correction on the decomposed reflectance and illumination respectively. Finally, the fused reflectance image and illumination image are produced by post-processing to get over blurring, etc. The experiments confirm that the proposed method can retain more details and edge information. Meanwhile, compared with other underwater image enhancement methods, the proposed method performs better in terms of visual effects with nearly 20% improvement in objective evaluation.