Reviewerly/exHarmony huggingface dataset:
Mutually Supervised Learning via Interactive Consistency for Geographic Object Segmentation from Weakly Labeled Remote Sensing Imagery

Geographic object segmentation from weakly annotated remote sensing images has become a research hotspot, since it can greatly reduce the costly annotation burden. Recently, it has made remarkable progress by dividing it into two sequential steps, which first produces pseudo labels (PLs) from a localization model, then uses PLs to train a segmentation network for final results. The one-way knowledge transfer in the above schemes, however, lacks the feedback from the segmentation to localization model which may result in suboptimal performance. In this paper, we develop a mutually supervised learning (MSL) framework for geographic object segmentation under image-wise annotations. First, MSL learns the localization and segmentation model concurrently and employs the output from each of the two models as pseudo supervision for the other one by formulating an interactive consistency loss, which encourages each model to provide positive feedback and guidance to the other. Then, a variance-based uncertainty estimation strategy is introduced to explicitly approximate the uncertainty of the PLs, which helps to alleviate the detrimental effect caused by learning from noisy PLs. Finally, we design a multi-scale activation integration-based localization model to produce high-quality localization maps. Comprehensive evaluations and ablation studies validate the superiority of the MSL framework.
USURP: Universal Single-Source Adversarial Perturbations on Multimodal Emotion Recognition

The field of affective computing has progressed from traditional unimodal analysis to more complex multimodal analysis due to the proliferation of videos posted online. Multimodal learning has shown remarkable performance in emotion recognition tasks, but its robustness in an adversarial setting remains unknown. This paper investigates the robustness of multimodal emotion recognition models against worst-case adversarial perturbations on a single modality. We found that standard multimodal models are susceptible to single-source adversaries and can be easily fooled by perturbations on any single modality. We draw some key observations that serve as guidelines for designing universal adversarial attacks on multimodal emotion recognition models. Motivated by these findings, we propose a novel universal single-source adversarial perturbations framework on multimodal emotion recognition models: USURP. Through our analysis of adversarial robustness, we demonstrate the necessity of studying adversarial attacks on multimodal models. Our experimental results show that the proposed USURP method achieves high attack success rates and significantly improves adversarial transferability in multimodal settings. The observations and novel attack methods presented in this paper provide a new understanding of the adversarial robustness of multimodal models, contributing to their safe and reliable deployment in more real- world scenarios.
Estimated Depth Based Progressive Interactive Framework for RGB Salient Object Detection in Images

Use of depth information is often beneficial for salient object detection (SOD) in color images, and hence, RGB-D SOD has gained popularity. However, the availability of depth data related to a color image can not be assumed, which is when RGB SOD is needed. RGB SOD can still leverage the benefits of depth information if the depth is estimated from the RGB data and suitably employed. To this end, we propose a novel estimated depth based interactive (EDINet) framework that performs SOD in images using only RGB data as its input. EDINet consists of a depth attentive feature extractor (DAFE) to capture features from depth-attentive regions and an RGB-estimated depth interaction module (RGB-EDI) that fuses hierarchical features from DAFE and the input RGB data. The salient object map is obtained after its progressive enhancement through its iterative use as a feedback into RGB-EDI. Quantitative and qualitative performance comparison with the state-of-the-art (SOTA) on benchmark RGB datasets shows the effectiveness of our framework. Considering RGB-D benchmark datasets and RGB-D SOD SOTA, the usefulness of EDINet in RGB-D SOD is also demonstrated by employing the actual depth instead of the estimated one.
A Novel Weakly Supervised Segmentation Approach for Rapid Left Ventricle Annotation

In the field of medical image segmentation, convolutional neural networks stand out as a successful method. However, in order to perform well, several labeled images are required. Manual pixel-level annotation of medical images requires the presence of a well-trained expert, is time-consuming, and is expensive. Weakly supervised learning approaches aim to address these challenges.In this work, we propose a novel weakly supervised segmentation approach specifically designed for the left ventricle. By utilizing the circular shape of the left ventricle, we introduce a weak annotation framework based on concentric circles, representing pixels inside and outside the ventricle. Our weakly supervised learning approach incorporates a loss function that ignores unannotated pixels and incorporates total variation regularization for smooth predictions. Our method significantly reduces annotation time from several minutes to 5-10 seconds per scan and eliminates the need for expert presence. We validated our method on the Sunnybrook dataset and our method reached around 0.96 of the accuracy of the networks trained in fully supervised manner (with pixel-level annotations). The implementation of our work is available at "https://github.com/behnam-rahmati/LV-weaklysupervised"
Multimodadl Graph Signal Denoising With Simultaneous Graph Learning using Deep Algorithm Unrolling

We propose a simultaneous method of multimodal graph signal denoising and graph learning. Since sensor networks distributed in space can capture multiple modalities of data, referred to as modalities, they are assumed to have an underlying structure or correlations both in space and modality. Such multimodal data are regarded as graph signals on a twofold graph. Like regular signals, multimodal graph signals can be corrupted by noise during their sensing process. Furthermore, their spatial/modality relationships are not given a priori: We need to estimate twofold graphs during denoising. In this paper, we propose a signal denoising method on twofold graphs where graphs are learned simultaneously. Specifically, we formulate an optimization problem for that, and an iterative algorithm for solving it is unrolled with deep algorithm unrolling (DAU). In the proposed method, the parameters in iterations are learned from training data that results in faster convergence and denoising quality improvements. Experimental results demonstrate that the proposed method outperforms existing graph signal denoising methods.
Sar Target Extraction Based On Saliency-Guided Cross-Domain Discrepancy Alignment Strategy

Target extraction based on deep learning approaches is a significant task in Synthetic Aperture Radar (SAR) image processing. However, the lack of SAR image samples brings great challenges to the data-driven method. In this paper, a Saliency-guided Cross-domain Discrepancy Alignment strategy is proposed to solve this problem. Firstly, we propose a saliency-guided attention module, which utilizes the context-aware saliency knowledge to guide the feature extraction and improve the training efficiency. Secondly, we train the Saliency-guided Cross-domain Alignment Network (SCANet) by large-scale natural optical image dataset and tiny-scale SAR image dataset. Thirdly, based on the guidance of saliency attention, cross-domain representation alignment strategy is proposed to learn a latent representation which aligns feature distribution between the source and target domain. Finally, SCANet is more adaptive for SAR images and extracts targets more accurately. Comparison with state-of-the-arts and ablation experiments demonstrate the efficiency of our method, especially in complex conditions.
DP-Net: Learning Discriminative Parts for Image Recognition

This paper <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> presents Discriminative Part Network (DP-Net), a deep architecture with strong interpretation capabilities, which exploits a pretrained Convolutional Neural Network (CNN) combined with a part-based recognition module. This system learns and detects parts in the images that are discriminative among categories, without the need for fine-tuning the CNN, making it more scalable than other part-based models. While part-based approaches naturally offer interpretable representations, we propose explanations at image and category levels and introduce specific constraints on the part learning process to make them more discrimative.
FD_YOLOX: An improved YOLOX object detection algorithm based on dilated convolution

In autonomous driving perception, vehicle object detection based on deep learning has been a major research topic. However, detecting vehicles has been a complex issue in computer vision. To address this challenge, we present the FD_YOLOX algorithm with three effective modules. The Fusion Dilated Convolution Module (FDCM) is combined dilated convolution feature fusion with an attention mechanism, which is allowed the network to obtain richer semantic information while adaptively detecting targets. The Dilated Channel-Adjusted Convolution (DCAC) is also proposed to address the small receptive field of high feature layers and adjust the number of channels in the input feature layer. Finally, the Dilated Spatial Pyramid Pooling (DSSP) is built by introducing dilated convolution in SPP, enhancing the receptive field of the network and preserving more information about small targets and their locations. Through experiments on the SODA10M data set, the FD_YOLOX_s model showed an increased mean average precision of 2.05%(640× 640) and 1.15%(1280 × 1280) compared to YOLOX_s. Moreover, the FD_YOLOX model achieved competitive performance against other advanced object detection algorithms in the detection of small target vehicles.
Improve Unsupervised Deep Hashing Via Masked Contrastive Learning

Unsupervised hashing method aims to generate compact binary hash codes for images without label supervision. Existing unsupervised hashing methods usually learn binary hash codes by reconstructing input data or preserving similarity structures. However, these methods will either force the hash code to retain a large amount of redundant information or will learn a similarity structure with noise due to biased prior knowledge, resulting in poor retrieval performance. In this paper, we introduce a novel unsupervised hashing method called Masked Contrastive Hashing (MCH). Specifically, to maximally preserve meaningful semantic information into the binary hash code, MCH adopts an encoder-decoder structure and extracts the binary representation from the random masked image to reconstruct the original image. Furthermore, MCH maximizes the consistency of the enhanced views of the same image while minimizing the consistency of different images to establish the similarity relationship between images, which is helpful to generate hash codes that are more suitable for retrieval tasks. Extensive experiments show that the proposed MCH significantly outperforms existing state-of-the-art methods on several benchmark datasets.
TAMM: A Task-Adaptive Multi-Modal Fusion Network for Facial-Related Health Assessments on 3D Facial Images

Previous studies showed that facial appearance is an important phenotypic indicator of human diseases or biological conditions. Recent advancements in deep learning have shown great potential in facial image analysis, including health status assessments. However, prior methods mainly focused on single modality analysis of either the 2D texture images or 3D facial meshes, which are limited in their ability to fully capture the relationships between biometric measurements and diseases. To address these issues, we propose a task-adaptive multi-modal fusion network, TAMM, for facial-related health assessments. Our model leverages both the geometric and texture features of 3D facial images by a task-adaptive Transformer (TAFormer), which can dynamically extract features from different modalities and scales for various tasks via spatial attention and cross modal multi-scale attention, effectively capture intra- and inter-modal relationships between features. Experimental results on a dataset of 19,775 patients demonstrate that TAMM achieves the state-of-the-art performance on various regression and classification tasks including age, BMI, and fatty liver disease predictions. Ablation studies shows the importance of multi-modal fusion and task-specific adaptability of our model in achieving optimal performance.
Self-Compensating Learning for Few-Shot Segmentation

Few-shot segmentation (FSS) has witnessed rapid development. Most existing approaches extract prototypes from support images to segment query images. However, the integrity and validity of these support prototypes cannot be guaranteed. To solve the above drawbacks, we propose a self-compensating strategy, aiming to provide query-aware support information, to build more effective matching between support information and query images. Specifically, we design a prototype compensating module to mine useful information from the query prediction, to update original support prototypes as new query-aware support prototypes. Then the updated prototypes are utilized to perform the second matching with query features. In addition, we also compensate the information of original prior masks on the second matching phase, to improve the quality of prior masks. With improved prototype representations and prior knowledge, our approach can directly improve the performance of different approaches with new state-of-the-art performances.
Coupling Spatial and Channel Transformer for Single Image Deraining

Single image deraining is a fundamental low-level vision task, and has evolved remarkable progress with the deep learning technique. Recently, benefiting from the powerful modeling ability of long-range dependence, transformer as an alternative architecture of the dominant convolutional neural network has demonstrated large margin performance improvement in various high-level vision tasks, and has begun to be applied for low-level vision tasks. The benchmark transformer block captures long dependence via incorporating the self-attention among the spatial points of the learned feature map, and causes heavy computational workload and memory footprint quadratically increased with spatial resolutions, making it impossible to handle high-resolution images. This study proposes a novel spatial and channel coupled Transformer to jointly explore long-range dependence and correlation in both spatial and channel domains, and results in a lightweight deraining transformer model for potentially processing high-resolution images.
Intelligent Painter: Picture Composition with Resampling Diffusion Model

Have you ever thought that you can be an intelligent painter? This means that you can paint a picture with a few expected objects in mind, or with a desirable scene. This is different from normal inpainting approaches for which the location of specific objects cannot be determined. In this paper, we present an intelligent painter that generate a person's imaginary scene in one go, given explicit hints. We propose a resampling strategy for Denoising Diffusion Probabilistic Model (DDPM) to intelligently compose unconditional harmonized pictures according to the input subjects at specific locations. By exploiting the diffusion property, we resample efficiently to produce realistic pictures. Experimental results show that our resampling method favors the semantic meaning of the generated output efficiently and generates less blurry output. Quantitative analysis of image quality assessment shows that our method produces higher perceptual quality images compared with the state-of-the-art methods.
Training-Free Location-Aware Text-to-Image Synthesis

Current large-scale generative models have impressive efficiency in generating high-quality images based on text prompts. However, they lack the ability to precisely control the size and position of objects in the generated image. In this study <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> , we analyze the generative mechanism of the stable diffusion model and propose a new interactive generation paradigm that allows users to specify the position of generated objects without additional training. Moreover, we propose an object detection-based evaluation metric to assess the control capability of location aware generation task. Our experimental results show that our method outperforms state-of-the-art methods on both control capacity and image quality.
LGSQE: Lightweight Generated Sample Quality Evaluation

Despite prolific work on evaluating generative models, little research has been done on the quality evaluation of an individual generated sample. To address this problem, a lightweight generated sample quality evaluation (LGSQE) method is proposed in this work. In the training stage of LGSQE, a binary classifier is trained on real and synthetic samples, where real and synthetic data are labeled by 0 and 1, respectively. In the inference stage, the classifier assigns soft labels (ranging from 0 to 1) to each generated sample. The value of the soft label indicates the quality level; namely, the quality is better if its soft label is closer to 0. LGSQE can serve as a post-processing module for quality control. Furthermore, LGSQE can be used to evaluate the performance of generative models, such as accuracy, AUC, precision, and recall, by aggregating sample-level quality. Experiments are conducted on several datasets and generative models to demonstrate that LGSQE can preserve the same performance rank order as that predicted by the Fréchet Inception Distance (FID) but with significantly lower complexity.
Complexity Reduction of Graph Signal Denoising Based on Fast Graph Fourier Transform

Denoising is one of the most fundamental and important problems in signal processing, and graph signal denoising methods have been actively studied. Several graph signal denoising methods based on mathematical programming require solving linear equations involving Laplacian matrix, which creates problem with computational accuracy and running time. This study proposes a fast and accurate solution of linear equations for denoising based on the fast graph Fourier transform method. Moreover, the proposed method can perform denoising not only on graphs for which the fast graph Fourier transform can be performed, but also on a wide class of graphs with more relaxed conditions, without loss of accuracy. Experiments demonstrate the efficiency of the proposed method and confirm that denoising can be performed up to 167.3 times faster without loss of accuracy.
It Wasn’t Me: Irregular Identity in Deepfake Videos

With the rapid development in media generation technologies, the creation of DeepFake videos is within everyone’s reach. As the widespread diffusion of DeepFakes can lead to severe consequences (e.g., defamation, fake news spreading, etc.), detecting DeepFakes is becoming a crucial task within the forensic community. However, most of the existing DeepFake detectors suffer from two issues: i) they are hardly explainable as they build upon black-box data-driven techniques rather than interpretable features; ii) they are often tailored to low-level texture features, failing to generalize on low-quality DeepFake videos. In this work we propose a video DeepFake detector that aims at solving these issues. The proposed detector relies on the fact that most DeepFake generators work on a frame-by-frame basis, thus breaking the temporal consistency of facial features across frames. In particular, we noticed that facial identity features tend to be less stable in time on DeepFake videos than original ones. We therefore propose a framework trained on time series of facial identity features. The use of high-level semantic features makes the detector interpretable and robust against low-quality DeepFake videos. Extensive experiments show that our method achieves outstanding performance on low-quality DeepFake video and obtains promising results on unseen dataset evaluation. The code is available at https://github.com/HongguLiu/Identity-Inconsistency-DeepFake-Detection
Efficient Pruning Method for Learned Lossy Image Compression Models Based on Side Information

In recent years, deep learning-based lossy image compression have achieved great success. However, the problem of their huge overhead in terms of computational and parametric costs has still not been adequately addressed. Inspired by the classical image compression methods, deep learning based models are usually combined with an entropy model to maintain the compression performance. Existing methods also introduce side information to serve as a prior on the parameters of the entropy model, which have achieved better rate-distortion performance. Based on the role of side information in learned image compression models, we propose an efficient pruning method for such models. In particular, the proposed pruning approach automatically searches for the optimal decoder architecture based on the extent to which each hidden layer in the decoder utilizes side information. The experiment results demonstrate the effectiveness of the proposed method and show that it outperforms all existing related studies in terms of compression performance.
Enhancing Targeted Transferability Via Suppressing High-Confidence Labels

While extensive studies have pushed the limit of the transferability of untargeted attacks, transferable targeted attacks remain extremely challenging. This paper finds that the labels with high confidence in the source model are also likely to retain high confidence in the target model. This simple and intuitive observation inspires us to carefully deal with the high-confidence labels in generating targeted adversarial examples for better transferability. Specifically, we integrate the untargeted loss function into the targeted attack to push the adversarial examples away from the original label while approaching the target label. Furthermore, we suppress other high-confidence labels in the source model with an orthogonal gradient. We validate the proposed scheme by mounting targeted attacks on the ImageNet dataset. Experiments on various scenarios show that our proposed scheme improves the state-of-the-art targeted attacks in transferability. Our code is available at: https://github.com/zengh5/Transferable_targeted_attack.
DENSECL: Haze Mitigation Using Dense Blocks and Contrastive Loss Regularization

Haze, which occurs as a result of the scattering of light in the atmosphere by small particles, diminishes the visibility of scene objects, inflicting important image applications such as object detection. To address the problem, this paper introduces a new physics-based end-to-end deep learning approach to haze mitigation in outdoor scenes, including those in airborne images. The proposed model named DenseCL is designed with dense blocks and adopts a contrastive loss function as an additional regularization. The model also maintains the cycle consistency by remapping the dehazed outputs into a hazy image using the physics-based light scattering function. DenseCL has been trained with publicly available outdoor images and demonstrates outstanding performance on outdoor, indoor, and remotely sensed nonhomogeneous haze satellite images.