Reviewerly/exHarmony huggingface dataset:
A Universal Ship Detection Method With Domain-Invariant Representations

Although ship detection methods based on deep learning have achieved remarkable progress, the design of the universal ship detection (USD) system is rarely studied. The main challenge of USD lies in the notorious domain bias and shift problem across multiple domains. This article implements USD based on domain-invariant representations to alleviate this issue. Specifically, the proposed method integrates a multilevel domain classification network (MDCN) and a domain-centric cut-paste module (DCM). First, the backbone network is facilitated to learn domain-independent image features from multiple domains through MDCN, thereby reducing the disturbance of domain-specific features to universal detector. Furthermore, the proposed method combines the domain-related synthetic samples generated by DCM to provide MDCN with strong supervision information, which further motivates the network to be more attentive to the domain-invariant representations at the instance level. Finally, we conduct experiments on multiple ship datasets in the synthetic aperture radar (SAR) and optical domains to verify the effectiveness of the method. The results show that the proposed method outperforms baseline by around 2.95% average precision (AP50), which achieves an effective USD system by complementing the information between domain-invariant representations.
True Black-Box Explanation in Facial Analysis

When explaining a recognition approach that can be used in facial analysis, e.g. face verification, face detection, attribute recognition, etc., the task is to answer: how relevant are the parts of a given image to establish the recognition. In many cases, however, the trained models cannot be manipulated and must be treated as "black-boxes". In this paper, we present a saliency map methodology, called MinPlus, that can be used to explain any facial analysis approach with no manipulation inside of the recognition model, because it only needs the input-output function of the black-box ‘f <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">x</inf> ’. The key idea of the method is based on how the probability of recognition of the given image changes when it is perturbed. Our method removes and aggregates different parts of the image, and measures contributions of these parts individually and in-collaboration as well. We test and compare our method in four different scenarios: face verification (with ArcFace), face expression recognition (with Xception), face detection (with MTCNN) and masked face detection (with YOLOv5s). We conclude that MinPlus achieves saliency maps that are stable and interpretable to humans. In addition, our method shows promising results in comparison with other state-of-the-art methods like AVG, LIME and RISE. This paper presents good insights into any facial analysis approach. It can be used to highlight the most relevant areas that an algorithm takes into account to carry out the recognition process.
Interaction Classification with Key Actor Detection in Multi-Person Sports Videos

Interaction recognition from multi-person videos is a challenging yet essential task in computer vision. Often the videos depict actions with multiple actors involved, some of whom participate in the main event, and the rest are present in the scene without being part of the actual event. This paper proposes a model to tackle the problem of interaction recognition from multi-person videos. Our model consists of a Recurrent Neural Network (RNN) equipped with a time-varying attention mechanism. It receives scene features and localized actors features to predict the interaction class. Additionally, the attention model identifies the people responsible for the main event. We chose penalty classification from ice hockey broadcast videos as our application. These videos are multi-persons and depict complex interactions between players in a non-laboratory recording setup. We evaluate our model on a new dataset of ice hockey penalty videos and report 93.93% classification accuracy. We include a qualitative analysis of the attention mechanism by visualizing the attention weights. Our code is publicly available <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
On the Exploitation of Deepfake Model Recognition

Despite recent advances in Generative Adversarial Networks (GANs), with special focus to the Deepfake phenomenon there is no a clear understanding neither in terms of explainability nor of recognition of the involved models. In particular, the recognition of a specific GAN model that generated the deepfake image compared to many other possible models created by the same generative architecture (e.g. StyleGAN) is a task not yet completely addressed in the state-of-the-art. In this work, a robust processing pipeline to evaluate the possibility to point-out analytic fingerprints for Deepfake model recognition is presented. After exploiting the latent space of 50 slightly different models through an in-depth analysis on the generated images, a proper encoder was trained to discriminate among these models obtaining a classification accuracy of over 96%. Once demonstrated the possibility to discriminate extremely similar images, a dedicated metric exploiting the insights discovered in the latent space was introduced. By achieving a final accuracy of more than 94% for the Model Recognition task on images generated by models not employed in the training phase, this study takes an important step in countering the Deepfake phenomenon introducing a sort of signature in some sense similar to those employed in the multimedia forensics field (e.g. for camera source identification task, image ballistics task, etc).
IMDeception: Grouped Information Distilling Super-Resolution Network

Single-Image-Super-Resolution (SISR) is a classical computer vision problem that has benefited from the recent advancements in deep learning methods, especially the advancements of convolutional neural networks (CNN). Although state-of-the-art methods improve the performance of SISR on several datasets, direct application of these networks for practical use is still an issue due to heavy computational load. For this purpose, recently, researchers have focused on more efficient and high-performing network structures. Information multi-distilling network (IMDN) is one of the highly efficient SISR networks with high performance and low computational load. IMDN achieves this efficiency with various mechanisms such as Intermediate Information Collection (IIC), working in a global setting, Progressive Refinement Module (PRM), and Contrast Aware Channel Attention (CCA), employed in a local setting. These mechanisms, however, do not equally contribute to the efficiency and performance of IMDN. In this work, we propose the Global Progressive Refinement Module (GPRM) as a less parameter-demanding alternative to the IIC module for feature aggregation. To further decrease the number of parameters and floating point operations per second (FLOPS), we also propose Grouped Information Distilling Blocks (GIDB). Using the proposed structures, we design an efficient SISR network called IMDeception. Experiments reveal that the proposed network performs on par with state-of-the-art models despite having a limited number of parameters and FLOPS. Furthermore, using grouped convolutions as a building block of GIDB increases room for further optimization during deployment. To show its potential, the proposed model was deployed on NVIDIA Jetson Xavier AGX and it has been shown that it can run in real-time on this edge device.
LAN: Lightweight Attention-based Network for RAW-to-RGB Smartphone Image Processing

The number of pictures taken by smartphones is growing exponentially. However, the smartphones’ limitations both in size and cost negatively impact on the quality of the implemented sensors. At the same time, their computing power has also been steadily improving, allowing the usage of more complex processing methods to enhance images. In prior works, deep neural networks trained with matched sensor outputs and DSLR images have shown to bring substantial improvements to the images, compared to classical and handcrafted methods. We propose a lightweight attention-based network (LAN) that employs a convolutional layer to learn the input mosaic and an unsupervised pre-training strategy. Our method is validated on standard benchmarks and shown to improve over the state-of-the-art in both perceptual and fidelity terms without hindering GPU inference time on smartphone devices. Our code is available at: github.com/draimundo/LAN
A Computational Intelligence Hybrid Algorithm Based on Population Evolutionary and Neural Network Learning for the Crude Oil Spot Price Prediction

Abstract This research attempts to reinforce the cultivating expression of radial basis function neural network (RBFnet) through computational intelligence (CI) and swarm intelligence (SI) learning methods. Consequently, the artificial immune system (AIS) and ant colony optimization (ACO) approaches are utilized to cultivate RBFnet for function approximation issue. The proposed hybridization of AIS and ACO approaches optimization (HIAO) algorithm combines the complementarity of exploitation and exploration to realize problem solving. It allows the solution domain having the advantages of intensification and diversification, which further avoids the situation of immature convergence. In addition, the empirical achievements have confirmed that the HIAO algorithm not only obtained the best accurate function approximation for theoretically standard nonlinear problems, it can be further applied on the instance solving for practical crude oil spot price prediction.
Secure transmission of secret data using optimization based embedding techniques in Blockchain

Secure transmission of data is achieved through the steganography images, where the secret data are embedded and transmitted so that the prohibited users couldn’t able to access the data. In this research, the data are embedded in the optimal location of the network using the proposed FIEO Optimal Location Enabled Stegochain. The algorithm effectively determines the location for the embedding of secret data in the finest region of the image with faster convergence. The two different steps utilized in the research is the determination of the optimal location for embedding the image using the proposed algorithm and the embedded image transformation using the discrete wavelet transform while transferring the image is the first step. And the reconstruction of the original image at the receiver side with the help of the Inverse discrete wavelet transform is performed (IDWT). The performance of the proposed method is evaluated in terms of the performance metrics such as correlation coefficient, and PSNR with the salt and pepper noise is 0.777, and 52.588 respectively. Similarly, for the random noise the correlation coefficient and PSNR obtained by the proposed method are 0.796, and 52.397 respectively.
Co-occurrence histogram based ensemble of classifiers for classification of cervical cancer cells

To explore the potential of conventional image processing techniques in the classification of cervical cancer cells, in this work, a co-occurrence histogram method was employed for image feature extraction and an ensemble classifier was developed by combining the base classifiers, namely, the artificial neural network (ANN), random forest (RF), and support vector machine (SVM), for image classification. The segmented pap-smear cell image dataset was constructed by the k-means clustering technique and used to evaluate the performance of the ensemble classifier which was formed by the combination of the above considered base classifiers. The result was also compared with that achieved by the individual base classifiers as well as that trained with color, texture, and shape features. The maximum average classification accuracy of 93.44% was obtained when the ensemble classifier was applied and trained with co-occurrence histogram features, which indicates that the ensemble classifier trained with the co-occurrence histogram features is more suitable for the classification of cervical cancer cells.
Efficient Homomorphic Convolution Designs on FPGA for Secure Inference

Recently, secure neural network (NN) inference, a combination of homomorphic encryption (HE) and NN, has attracted much attention. Nevertheless, a large number of computations, mainly brought by the HE scheme, form the bottleneck in real-time applications. In this article, we present a hardware accelerator on a field-programmable gate array (FPGA) for the homomorphic convolution layer (HomConvL), which is the most computation-intensive part of the HE-based secure inference. First, we propose a new HomConvL algorithm called packed rotations at inputs (PaRotI), which is suitable for hardware implementation for its inherent high parallelism and low complexity with acceptable noise growth and moderate resource consumption. Then, we present three highly parallel architectures for different parameter sets and application scenarios of state-of-the-art HomConvL algorithms. The new architectures are implemented on a Xilinx VCU110 FPGA board, and the experimental results demonstrate that our designs can achieve 15.31– <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$19.46\times $ </tex-math></inline-formula> speedups compared with the software implementations.
Y-Shaped Net-Based Signal Detection for OFDM-IM Systems

This letter presents the power of deep learning for signal detection in orthogonal frequency division multiplexing with index modulation (OFDM-IM) systems. In this letter, we propose the Y-shaped net-based scheme with fully connected layers (Y-FC) and with bidirectional long short-term memory units (Y-BLSTM) to further optimize the data reception. In the Y-shaped scheme, the active indices and the constellation demapping are learned by two parallel sub-neural networks respectively, the output features are then concatenated and flattened to obtain binary bits. The performance is evaluated under various system parameters, such as modulation methods, number of hidden neural nodes and channel states. Simulation results show that compared with the data-driven signal detection with direct input-output, the proposed scheme achieves several dB of bit-error-rate performance improvement with almost the same running time.
Unsupervised Anomaly Detection from Time-of-Flight Depth Images

Video anomaly detection (VAD) addresses the problem of automatically finding anomalous events in video data. The primary data modalities on which current VAD systems work on are monochrome or RGB images. Using depth data in this context instead is still hardly explored in spite of depth images being a popular choice in many other computer vision research areas and the increasing availability of inexpensive depth camera hardware. We evaluate the application of existing autoencoder-based methods on depth video and propose how the advantages of using depth data can be leveraged by integration into the loss function. Training is done unsupervised using normal sequences without need for any additional annotations. We show that depth allows easy extraction of auxiliary information for scene analysis in the form of a foreground mask and demonstrate its beneficial effect on the anomaly detection performance through evaluation on a large public dataset, for which we are also the first ones to present results on.
SaR: Self-adaptive Refinement on Pseudo Labels for Multiclass-Imbalanced Semi-supervised Learning

Class-imbalanced datasets can severely deteriorate the performance of semi-supervised learning (SSL). This is due to the confirmation bias especially when the pseudo labels are highly biased towards the majority classes. Traditional resampling or reweighting techniques may not be directly applicable when the unlabeled data distribution is unknown. Inspired by the threshold-moving method that performs well in supervised learning-based binary classification tasks, we provide a simple yet effective scheme to address the multiclass imbalance issue of SSL. This scheme, named SaR, is a Self-adaptive Refinement of soft labels before generating pseudo labels. The pseudo labels generated post-SaR will be less biased, resulting in higher quality data for training the classifier. We show that SaR can consistently improve recent consistency-based SSL algorithms on various image classification problems across different imbalanced ratios. We also show that SaR is robust to the situations where unlabeled data have different distributions as labeled data. Hence, SaR does not rely on the assumptions that unlabeled data share the same distribution as the labeled data.
Cluster-to-adapt: Few Shot Domain Adaptation for Semantic Segmentation across Disjoint Labels

Domain adaptation for semantic segmentation across datasets consisting of the same categories has seen several recent successes. However, a more general scenario is when the source and target datasets correspond to non-overlapping label spaces. For example, categories in segmentation datasets change vastly depending on the type of environment or application, yet share many valuable semantic relations. Existing approaches based on feature alignment or discrepancy minimization do not take such category shift into account. In this work, we present Cluster-to-Adapt (C2A), a computationally efficient clustering-based approach for domain adaptation across segmentation datasets with completely different, but possibly related categories. We show that such a clustering objective enforced in a transformed feature space serves to automatically select categories across source and target domains that can be aligned for improving the target performance, while preventing negative transfer for unrelated categories. We demonstrate the effectiveness of our approach through experiments on the challenging problem of outdoor to indoor adaptation for semantic segmentation in few-shot as well as zero-shot settings, with consistent improvements in performance over existing approaches and baselines in all cases.
PseudoProp: Robust Pseudo-Label Generation for Semi-Supervised Object Detection in Autonomous Driving Systems

Semi-supervised object detection methods are widely used in autonomous driving systems, where only a fraction of objects are labeled. To propagate information from the labeled objects to the unlabeled ones, pseudo-labels for unlabeled objects must be generated. Although pseudo-labels have proven to improve the performance of semi-supervised object detection significantly, the applications of image-based methods to video frames result in numerous miss or false detections using such generated pseudo-labels. In this paper, we propose a new approach, Pseudo-Prop, to generate robust pseudo-labels by leveraging motion continuity in video frames. Specifically, PseudoProp uses a novel bidirectional pseudo-label propagation approach to compensate for misdetection. A feature-based fusion technique is also used to suppress inference noise. Extensive experiments on the large-scale Cityscapes dataset demonstrate that our method outperforms the state-of-the-art semi-supervised object detection methods by 7.4% on mAP <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">75</sup> .
An Efficient Domain-Incremental Learning Approach to Drive in All Weather Conditions

Although deep neural networks enable impressive visual perception performance for autonomous driving, their robustness to varying weather conditions still requires attention. When adapting these models for changed environments, such as different weather conditions, they are prone to forgetting previously learned information. This catastrophic forgetting is typically addressed via incremental learning approaches which usually re-train the model by either keeping a memory bank of training samples or keeping a copy of the entire model or model parameters for each scenario. While these approaches show impressive results, they can be prone to scalability issues and their applicability for autonomous driving in all weather conditions has not been shown. In this paper we propose DISC – Domain Incremental through Statistical Correction – a simple online zero-forgetting approach which can incrementally learn new tasks (i.e. weather conditions) without requiring re-training or expensive memory banks. The only information we store for each task are the statistical parameters as we categorize each domain by the change in first and second order statistics. Thus, as each task arrives, we simply ‘plug and play’ the statistical vectors for the corresponding task into the model and it immediately starts to perform well on that task. We show the efficacy of our approach by testing it for object detection in a challenging domain-incremental autonomous driving scenario where we encounter different adverse weather conditions, such as heavy rain, fog, and snow.
Complete and temporally consistent video outpainting

We describe a novel method for video outpainting. The goal of outpainting is to fill in missing regions at the edges of video frames. Our focus lies on converting portrait (9:16) to landscape (16:9) video. In contrast, most video completion research is focused on inpainting: filling a masked section within the frame based on the remaining, known pixels.Our proposed method consists of three main aspects: (1) We form a background estimation using video object segmentation and video inpainting methods, (2) we use optical flow to form temporal consistency, and (3) we propose image shifting to improve individual frame completions. Our method is able to successfully broaden the aspect ratio of a video. On most videos, we achieve realistic results. Only on videos with complex camera motion and foreground objects leaving the frame, the quality is less.In contrast to other state-of-the-art methods, our method is able to reconstruct the full frame, including unseen image parts. Moreover, it is temporally consistent. We evaluate our method on the DAVIS and YouTube-VOS datasets. The code is publicly available <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
An Ensemble Learning and Slice Fusion Strategy for Three-Dimensional Nuclei Instance Segmentation

Automated microscopy image analysis is a fundamental step for digital pathology and computer aided diagnosis. Most existing deep learning methods typically require post-processing to achieve instance segmentation and are computationally expensive when directly used with 3D microscopy volumes. Supervised learning methods generally need large amounts of ground truth annotations for training whereas manually annotating ground truth masks is laborious especially for a 3D volume. To address these issues, we propose an ensemble learning and slice fusion strategy for 3D nuclei instance segmentation that we call Ensemble Mask R-CNN (EMR-CNN) which uses different object detectors to generate nuclei segmentation masks for each 2D slice of a volume and propose a 2D ensemble fusion and a 2D to 3D slice fusion to merge these 2D segmentation masks into a 3D segmentation mask. Our method does not need any ground truth annotations for training and can inference on any large size volumes. Our proposed method was tested on a variety of microscopy volumes collected from multiple regions of organ tissues. The execution time and robustness analyses show that our method is practical and effective.