Reviewerly/exHarmony huggingface dataset:
DAUT: Underwater Image Enhancement Using Depth Aware U-shape Transformer

Images captured underwater are subject to different complex effects including absorption and scattering. Recovery of the original images is not a trivial task. The underwater image formation varies according to many dependencies. Also, underwater image enhancement models have been affected by the lack of large-scale underwater datasets with reference-enhanced images. Therefore, in this paper, we use cycleGAN to generate a new underwater dataset from existing in-air datasets. We train our data with a U-shape Transformer, which is one of the state-of-the-art models. We add a depth estimation module as object depth is one of the most vital dependencies to the underwater image formation model. The estimated depth and the underwater image are the input to our depth-aware U-transformer. Our experiments show that our model achieves state-of-the-art results in objective and subjective evaluations compared with the original U-shape Transformer and other state-of-the-art methods. the code and dataset are available at https://github.com/MBadran2000/Depth-Aware-U-shape-Transformer.git
Revolutionizing Thermal Imaging: GAN-Based Vision Transformers for Image Enhancement

In this paper, we propose a new architecture for thermal image enhancement in which we exploit the strengths of both architecture-based vision transformers and generative adversarial networks. Our approach includes the introduction of a thermal loss function, which is specifically employed to produce high quality images. In addition, we consider fine-tuning based on visible images for thermal image restoration, resulting in an overall improvement in image quality. The performance of our proposed architecture is evaluated using visual quality metrics. The results show significant improvements over the original thermal images and over other established enhancement methods on a subset of the KAIST dataset. The performance of the proposed enhancement architecture is also verified on the detection results by obtaining better performance with a considerable margin considering different versions of the YOLO detector.
DNC-Net:Dual-neighbourhood Consensus Network for Feature Matching

Local feature matching is the core of many computer vision tasks. While dectetor-based methods lack repeatability and only consider small image regions, sparse feature matching methods is challenging in textureless scenes. This paper proposes a novel feature matching method based on dual-neighbourhood consensus, termed as DNC-Net. In DNC-Net, we build the connection between keypoints and feature maps, so that the matching can comprehensively consider the small and large image regions. Further, dual-neighbourhood consensus fully learn the consensus between keypoints and feature maps, which improve matching robustness especially in textureless scenes. Experiments on homography estimation, outdoor pose estimation and image matching show that the model is superior to other methods, and has achieved the most advanced results.
Half of an Image is Enough for Quality Assessment

Deep networks show promising performance in image quality assessment (IQA), whereas few studies have investigated how a deep model works. In this work, a positional masked transformer for IQA is first developed, based on which we observe that half of an image might contribute trivially to image quality, whereas the other half is crucial. Such observation is generalized to that half of the image regions can dominate image quality in several CNN-based IQA models. Motivated by this observation, three semantic measures (saliency, frequency, objectness) are then derived, showing high accordance with importance degree of image regions in IQA.
Object Detection and Counting Challenges in Real Street Monitoring: Case Study of Homeless Encampments

Wide area urban street monitoring is highly demanding in various smart city applications. Manual monitoring is both laborious and time-consuming, hence automatic vision-based monitoring is a more feasible alternative. An essential part of vision-based street monitoring is detecting and counting objects of interest. However, these tasks are not straightforward due to various challenges, i.e., noisy conditions in a real environment, such as occlusion and high illumination. This study investigates the impact of these challenges on object detection and counting accuracy, then provides an empirical study to address the challenges with respect to video-based street monitoring. The selected case study demonstrates detecting and counting of homeless encampments in Los Angeles streets using street-level videos collected from a moving vehicle.
Skeleton Action Recognition Based on Spatio-Temporal Features

Graph convolutional networks have performed well in action recognition tasks using skeleton data, but topology extraction remains a challenge. Most methods fail to consider the correlations between channels in skeleton topology graphs. To address this, we propose a multi-channel optimal graph convolutional network (MOGCN) that combines multi-channel optimal graph convolution and multi-scale temporal convolution techniques. MOGCN exhibits improved capability in processing spatio-temporal information and node relationships, generating optimal skeleton topology graphs that model correlations between all nodes in the sequence. We also use multi-scale temporal convolution to extract temporal features, improving the modeling of long-term correlations and global correlations of spatio-temporal joints. We further improve accuracy using a multi-stream fusion network model. Experiments on multiple datasets show that the proposed MOGCN module is effective and improves the classification accuracy of skeleton-based action recognition.
Generalized Pseudo-Labeling in Consistency Regularization for Semi-Supervised Learning

Semi-Supervised Learning (SSL) reduces annotation cost by exploiting large amounts of unlabeled data. A popular idea in SSL image classification is Pseudo-Labeling (PL), where the predictions of a network are used in order to assign a label to an unlabeled image. However, this practice exposes learning to confirmation bias. In this paper we propose Generalized Pseudo-Labeling (GPL), a simple and generic way to exploit negative pseudo-labels in consistency regularization, entailing minimal additional computational overhead and hyperpameter fine-tuning. GPL makes learning more robust by using the information that an image does not belong to a certain class, which is more abundant and reliable. We showcase GPL in the context of FixMatch. In the benchmark using only 40 labels of the CIFAR-10 dataset, adding GPL on top of FixMatch improves the error rate from 7.93% to 6.58%, and on CIFAR-100 with 2500 labels, from 28.02% to 26.85%.
Context-Aware Pedestrian Trajectory Prediction with Multimodal Transformer

We propose a novel solution for predicting future trajectories of pedestrians. Our method uses a multimodal encoder-decoder transformer architecture, which takes as input both pedestrian locations and ego-vehicle speeds. Notably, our decoder predicts the entire future trajectory in a single-pass and does not perform one-step-ahead prediction, which makes the method effective for embedded edge deployment. We perform detailed experiments and evaluate our method on two popular datasets, PIE and JAAD. Quantitative results demonstrate the superiority of our proposed model over the current state-of-the-art, which consistently achieves the lowest error for 3 time horizons of 0.5, 1.0 and 1.5 seconds. Moreover, the proposed method is significantly faster than the state-of-the-art for the two datasets of PIE and JAAD. Lastly, ablation experiments demonstrate the impact of the key multimodal configuration of our method.
Error Concealment for Scalable Video Coding Based on Deformable Convolution Network

Scalable Video Coding (SVC) is a layered coding mechanism with advantages in network adaptation and error resiliency brought by the multiple layers. SVC encoder can encode a video into bitstreams, including a base layer that is usually given more protection and several enhancement layers. Although the enhancement layer is not as important as the base layer, frame loss in the enhancement layer can also result in quality degradation. Therefore, we propose a novel error concealment method for the enhancement layer. The proposed method consists of two modules, the RSR module and the QE module. The RSR module fully uses the relationship between the base and enhancement layers of the previous frame through deformable convolution to predict the intermediate image. The QE module post-processes the intermediate image using the difference between the enhancement and the upsampled base layer from the previous frame through a deep residual network to obtain the final predicted image for error concealment. We train our model on a public dataset and test it on a set of sequences. The experiment result demonstrates that the proposed method outperforms traditional error concealment methods and other deep-learning-based methods.
Robust Multispectral Pedestrian Detection Via Spectral Position-Free Feature Mapping

Recently, although multispectral pedestrian detection has achieved remarkable performances, there is still a problem to be handled, position shift problem. Due to the problem, a pedestrian looks like existing in different positions between each modal image. Then, a single bounding box usually fails to capture an entire pedestrian properly in both modal images at the same time, which means it would not contain some parts of a pedestrian and includes noisy backgrounds instead. In this paper, we propose a novel approach, that is, a pedestrian feature mapping from mis-captured pedestrian features to well-captured pedestrian features which encode an entire pedestrian properly in both modal images. To this end, we utilize a memory architecture which stores well-captured pedestrian features, and then, the well-captured features can enhance the quality of pedestrian representation by providing the distinctive information of a pedestrian. We validate the effectiveness of our approach with comprehensive experiments on two multispectral pedestrian detection datasets, achieving state-of-the-art performances.
Underwater Image Enhancement and Super-Resolution Using Implicit Neural Networks

Underwater images are often notably degraded by light scattering and absorption. To improve image quality and object details, we present a novel unsupervised underwater image enhancement and super-resolution method using implicit neural networks. Concretely, taking low-resolution coordinates as the inputs, we first leverage Fourier feature mapping to encode the coordinates. Then, three implicit neural networks are applied to estimate each component (i.e., the global background light, the transmission map, and the scene radiance) of the underwater formation model. Those components are further used to reconstruct the raw underwater image in a self-supervised fashion. In the inference stage, high-resolution coordinates are employed to predict a high-quality and high- resolution underwater image. Extensive experiments show that our method achieves a favorable performance in terms of both super-resolution and quality enhancement as compared with current approaches.
Latent-Shift: Gradient of Entropy Helps Neural Codecs

End-to-end image/video codecs are getting competitive compared to traditional compression techniques that have been developed through decades of manual engineering efforts. These trainable codecs have many advantages over traditional techniques such as easy adaptation on perceptual distortion metrics and high performance on specific domains thanks to their learning ability. However, state of the art neural codecs does not take advantage of the existence of gradient of entropy in decoding device. In this paper, we theoretically show that gradient of entropy (available at decoder side) is correlated with the gradient of the reconstruction error (which is not available at decoder side). We then demonstrate experimentally that this gradient can be used on various compression methods, leading to a 1−2% rate savings for the same quality. Our method is orthogonal to other improvements and brings independent rate savings.
Adversarial Example Detection Bayesian Game

Despite the increasing attack ability and transferability of adversarial examples (AE), their security, i.e., how unlikely they can be detected, has been ignored more or less. Without the ability to circumvent popular detectors, the chance that an AE successfully fools a deep neural network is slim. This paper gives a game theory analysis of the interplay between an AE attacker and an AE detection investigator. Taking the perspective of a third party, we introduce a game theory model to evaluate the ultimate performance when both the attacker and the investigator are aware of each other. Further, a Bayesian game is adopted to address the information asymmetry in practice. Solving the mixed-strategy Nash equilibrium of the game, both parties’ optimal strategies are obtained, and the security of AEs can be evaluated. We evaluate four popular attacks under a two-step test on ImageNet. The results may throw light on how a farsighted attacker or investigator will act in this adversarial environment. Our code is available at: https://github.com/zengh5/AED_BGame.
An Enhanced Neuron Attribution-Based Attack Via Pixel Dropping

Convolutional neural networks (CNNs) are vulnerable to adversarial examples (AEs). Existing feature-level attacks explore the neuron importance to distort the intrinsic object-aware features which are shareable among different CNNs, thus achieving great performance in transferability. In this work, we propose an enhanced neuron attribution-based attack via pixel dropping (ENAA) and try to increase the number of positive neurons to distort the object-aware features more fully than NAA. Specifically, when computing neuron attribution, we use a pixel dropping scheme to expand the regions where the source model pays attention to the image. Our ENAA can make the target model shift the attention regions of AEs far away from those of clean images. Experimental results validate that the proposed method outperforms the state-of-the-art feature-level attacks both in white-box and black-box settings.
Video Super-Resolution Via Event-Driven Temporal Alignment

Video super-resolution aims to restore low-resolution videos into their high-resolution counterparts. Existing methods typically rely on optical flow, which assumes linear motion and is sensitive to rapid lighting changes, to capture inter-frame information. Event cameras can asynchronously output high temporal resolution event streams, which can reflect nonlinear motion and are robust to lighting changes. Inspired by these characteristics, we propose an Event-driven Bidirectional Video Super-Resolution (EBVSR) framework. Firstly, we propose an event-assisted temporal alignment module that utilizes events to generate nonlinear motion to align adjacent frames, complementing flow-based methods. Secondly, we build an event-based frame synthesis module that enhances the network’s robustness to lighting changes through a bidirectional cross-modal fusion design. Experimental results on synthetic and real-world datasets demonstrate the superiority of our method. The code is available at https://github.com/DachunKai/EBVSR.
Single-Domain Generalization for Semantic Segmentation Via Dual-Level Domain Augmentation

The goal of single-domain generalization is to learn a domain- generalized model from only one single source domain. To avoid overfitting to the source domain, recent research focused on domain augmentation for learning domain generalized features. Therefore, domain diversity is indeed crucial to the generalization ability of the model. In this paper, we propose a novel dual-level domain augmentation framework to enrich the domain diversity for single-domain generalized semantic segmentation. We specifically devise an Image-Level and a Class-Level Augmentation Module (IAM and CAM) to enlarge the diversity of augmented images and per-class features, respectively. From the original and augmented data, we then design a Domain-Generalized Feature Learning to learn representative features regularized by a large-scale pretrained model. Experimental results on semantic segmentation benchmarks demonstrate the effectiveness and outperformance of the proposed method over previous work.
Hard Samples Based Margin Loss for Face Verification

Although softmax loss and its variants have achieved great success in face verification, the performance is still subject to the data imbalance and early saturation problems. In this paper, we define hard samples as minority class samples and early saturation samples, in order to address both issues, we propose a new loss function termed Hard-Samples based Margin (HSM) loss. Inspired by the class-variant margin normalized softmax loss, we add larger margin on minority classes, the proposed real-class margin overcomes the negative influence from the data imbalance via making the optimization more balanced, while by expanding the margin of early saturated samples, the proposed pseudo-class margin keeps the samples away from the saturation region. Comprehensive experiments show that our HSM loss consistently surpasses the state-of-the-art loss functions on four popular face verification benchmarks.
A Semi-Paired Approach for Label-to-Image Translation

Data efficiency, or the ability to generalize from a few labeled data, remains a major challenge in deep learning. Semi-supervised learning has thrived in traditional recognition tasks alleviating the need for large amounts of labeled data, yet it remains understudied in image-to-image translation (I2I) tasks. In this work, we introduce the first semi-supervised (semi-paired) framework for label-to-image translation, a challenging subtask of I2I which generates photorealistic images from semantic label maps. In the semi-paired setting, the model has access to a small set of paired data and a larger set of unpaired images and labels. Instead of using geometrical transformations as a pretext task like previous works, we leverage an input reconstruction task by exploiting the conditional discriminator on the paired data as a reverse generator. We propose a training algorithm for this shared network, and we present a rare classes sampling algorithm to focus on under-represented classes. Experiments on 3 standard benchmarks show that the proposed model outperforms state-of-the-art unsupervised and semi-supervised approaches, as well as some fully supervised approaches while using a much smaller number of paired samples.
FEDMBP: Multi-Branch Prototype Federated Learning on Heterogeneous Data

Federated learning (FL) enables collaborative model training across clients while preserving data privacy. However, FL faces the challenge of data heterogeneity, leading to biased local models that deviate from the global model during optimization. And existing FL algorithms like Federated Averaging (FedAvg) suffer from this issue. To address this problem, we propose a novel approach called multi-branch prototype federated learning (FedMBP). FedMBP creates auxiliary branches within each local model to integrate different levels of local and global prototypes, thus preventing local model drift by aligning local prototypes with global ones. We also introduce mixed cross-entropy on the auxiliary branches to effectively transfer global prototype knowledge to local models. We conduct experiments on three publicly available datasets, including natural and medical image domains. Our experiments demonstrate that FedMBP outperforms existing FL algorithms, achieving superior model performance.
Self-Reinforcing For Few-Shot Medical Image Segmentation

Few-shot learning serves as a viable solution for addressing data scarcity, thus exhibiting significant potential in the domain of medical image segmentation. In this work, we propose a simple and efficient framework for few-shot medical image segmentation, termed SRPNet, which leverages self-reinforcement between foreground and background. Notably, without the need for prior knowledge, the model autonomously adapts the segmentation effect of both foreground and background, thereby enhancing the segmentation of previously unseen classes. Experimental evaluations conducted on CT and MRI datasets demonstrate the superior performance of the proposed method compared to other state-of-the-art techniques. Code is available at https://github.com/q362096112/SRPNet.
Context-Aware Transformers for Weakly Supervised Baggage Threat Localization

Recent advances in deep learning have facilitated significant progress in the autonomous detection of concealed security threats from baggage X-ray scans, a plausible solution to overcome the pitfalls of manual screening. However, these data-hungry schemes rely on extensive instance-level annotations that involve strenuous skilled labor. Hence, this paper proposes a context-aware transformer for weakly supervised baggage threat localization, exploiting their inherent capacity to learn long-range semantic relations to capture the object-level context of the illegal items. Unlike the conventional single-class token transformers, the proposed dual-token architecture can generalize well to different threat categories by learning the threat-specific semantics from the token-wise attention to generate context maps. The framework has been evaluated on two public datasets, Compass-XP and SIXray, and surpassed other SOTA approaches.
DF-Net: Diversity-Focused Network for Video Object Detection

Video object detection is a challenging task due to deteriorated object appearances. To enhance per-frame features, one way is to aggregate features from several support frames. However, proposals generated by the region proposal network may not be precise and diverse due to the fixed anchors, limiting the detection performance. We propose a novel architecture called Diversity-Focused Network (DF-Net), which consists of three modules: 1) An affine transform module (ATM), which is proposed to model the deblurring process and fuse the feature maps of different receptive fields by a multi-level attention block; 2) A label assignment module (LAM), which is proposed to assign the labels to the proposals used in a fine-grained aggregation manner; 3) A regression-guided diffusion module (RGDM), which is proposed to obtain the features of diversity and higher quality. Experiments show that DF-Net achieves favorable results on the most representative large-scale ImageNet VID dataset. Remarkably, the DF-Net achieves 84.8% mAP with ResNet-101 without post-processing steps.
Lightweight CNN-Based in-Loop Filter for VVC Intra Coding

In versatile video coding (VVC), the in-loop filters suppress compression artifacts while reducing distortion. However, they have a limit of removing complicated compression artifacts due to the hand-crafted design. In this paper, we propose a convolutional neural network (CNN)-based in-loop filter for VVC intra coding. We introduce depthwise separable convolution and attention mechanism to make the proposed network lightweight and efficient. Moreover, we present two basic modules of residual attention block (RAB) and weakly connected attention block (WCAB) to extract and refine features. Besides, we design a multi-stage training strategy based on progressive learning to maximize the learning ability of the proposed network. Compared with VTM-11.0_NNVC-2.0 anchor, the proposed in-loop filter achieves average -7.08% (Y), -12.46% (U), -12.75% (V) BD-rate gains under All Intra (AI) configuration.