Reviewerly/exHarmony huggingface dataset:
Crowd-Powered Face Manipulation Detection: Fusing Human Examiner Decisions

We investigate the potential of fusing human examiner decisions for the task of digital face manipulation detection. To this end, various decision fusion methods are proposed incorporating the examiners’ decision confidence, experience level, and their time to take a decision. Conducted experiments are based on a psychophysical evaluation of digital face image manipulation detection capabilities of humans in which different manipulation techniques were applied, i.e. face morphing, face swapping and retouching. The decisions of 223 participants were fused to simulate crowds of up to seven human examiners. Experimental results reveal that (1) despite the moderate detection performance achieved by single human examiners, a high accuracy can be obtained through decision fusion and (2) a weighted fusion, which takes the examiners’ decision confidence into account, yields the most competitive detection performance.
A Multi-Source Image Matching Network for UAV Visual Location

Visual localization is an important but challenging task for unmanned aerial vehicles (UAV). Matching real-time UAV orthophotos to pre-existing georeferenced satellite images is the key problem for this task. However, UAV and satellite images are inconsistent in image styles, perspectives, and times. In this paper, a new fully convolutional siamese network is proposed to extract similar features for multi-source images. The Squeeze-and-Excitation structure is integrated into the densely connected network to adapt to multi-scale features and the texture differences of different regions. Besides, a loss function with a progressive sampling strategy is utilized to mine the similarity of matching multi-source images and improve the description compactness among dimensions. Extensive experimental results with in-depth analysis are provided, which indicate that the proposed framework can significantly improve the matching performance of the learned descriptor.
RPFNET: Complementary Feature Fusion for Hand Gesture Recognition

Hand gesture recognition (HGR) is one of the most challenging tasks because it is very sensitive to occlusion or background. Various modalities such as RGB, depth, and point cloud as well as their combinations have been proposed to improve the performance of HGR, but the fusion of RGB and point cloud with complementary characteristics has never been attempted. This paper analyzes the synergistic effect of the two complementary modalities, and then proposes a new multi-modal fusion network that quantifies and converges the mutual influence of two modalities. Also, to overcome the inherent limitation that the predicted mutual influence does not match the actual one, we propose the self-labeling-based adaptive guidance. Experimental results show that the proposed method achieved 2.46% higher performance than the SOTA method in the case of the NVGesture dataset.
Ganzzle: Reframing Jigsaw Puzzle Solving as a Retrieval Task using a Generative Mental Image

Puzzle solving is a combinatorial challenge due to the difficulty of matching adjacent pieces. Instead, we infer a mental image from all pieces, which a given piece can then be matched against avoiding the combinatorial explosion. Exploiting advancements in Generative Adversarial methods, we learn how to reconstruct the image given a set of unordered pieces, allowing the model to learn a joint embedding space to match an encoding of each piece to the cropped layer of the generator. Therefore we frame the problem as a R@1 retrieval task, and then solve the linear assignment using differentiable Hungarian attention, making the process end-to-end. In doing so our model is puzzle size agnostic, in contrast to prior deep learning methods which are single size. We evaluate on two new large-scale datasets, where our model is on par with deep learning methods, while generalizing to multiple puzzle sizes.
Bridging the Domain Gap in Real World Super-Resolution

In this paper, we propose a state-of-the-art super-resolution algorithm and a framework to effectively train this architecture to handle real-world images. Recent learning-based, super-resolution methods have achieved impressive performance on ideal datasets. However, their performance plummets when tested on real images, as their assumed degradation model deviates from reality. Instead of assuming simplistic degradations, we make use of generative methods to mimic real degradations in order to narrow the domain gap between training data and testing data. In particular, we employ a novel Normalizing Flow architecture to replicate realistic noise and camera pipeline degradations. This framework, in conjunction with a carefully designed neural network architecture, results in a super-resolution algorithm which compares favorably to other methods. As for the model architecture, the combination between its performance on real images and lower computational load makes it attractive for practical super-resolution applications. We compare our method with different state-of-art algorithms, both on synthetic and real images. The experimental results demonstrate the remarkable performance of our super-resolution algorithm in real-world applications.
NPCFORMER: Automatic Nasopharyngeal Carcinoma Segmentation Based on Boundary Attention and Global Position Context Attention

Nasopharyngeal carcinoma (NPC) is a malignant tumor whose accurate segmentation is a prerequisite for treatment. However, existing deep learning methods achieve unsatisfactory segmentation performance on NPC MR images, since NPC is infiltrative with small ambiguous boundary volume, making it indiscernible from tightly connected surrounding and complex backgrounds. To address the issues, a NPC segmentation network, termed NPCFormer, is proposed. The NPCFormer consists of two modules, Skip Residual Transformer (SRT) and Boundary Attention Unit (BAU), which are designed for NPC segmentation. The two modules are proposed via redesigning the multi-head self-attention to achieve accurate segmentation of NPC. The SRT exploits the global position context for the NPC locating. The BAU discriminates the tumor boundaries from its surrounding tissues by utilizing the global position context. Extensive experiments on our dataset demonstrate the proposed NPCFormer could distinguish and segment NPC from complex background tissues accurately.
Multi-Step Test-Time Adaptation with Entropy Minimization and Pseudo-Labeling

The accuracy of deep neural networks is easily degraded by image corruption. Therefore, there is a need to develop adaptation techniques to ensure durable models and predictions against changes in data distribution. We focus on the task to fit a trained model with a different distribution from training data under the condition that the training data are not available for test time. In this paper, we propose a novel adaptation method in test time for online learning named multi-step layer adaptation (MuSLA). The proposed method achieves high adaptive accuracy by sequentially applying loss functions to specific layers only, especially considering the roles and inter-actions of the layers and employing domain adaptation and semi-supervised learning techniques. The proposed method can be widely applied to already existing trained models with-out additional networks. We show that our approach outperforms conventional methods in image corruption benchmark data experiments.
Deep-Based Quality Assessment of Medical Images Through Domain Adaptation

Predicting the quality of multimedia content is often needed in different fields. In some applications, quality metrics are crucial with a high impact, and can affect decision making such as diagnosis from medical multimedia. In this paper, we focus on such applications by proposing an efficient and shallow model for predicting the quality of medical images without reference from a small amount of annotated data. Our model is based on convolution self-attention that aims to model complex representation from relevant local characteristics of images, which itself slide over the image to interpolate the global quality score. We also apply domain adaptation learning in unsupervised and semi-supervised manner. The proposed model is evaluated through a dataset composed of several images and their corresponding subjective scores. The obtained results showed the efficiency of the proposed method, but also, the relevance of the applying domain adaptation to generalize over different multimedia domains regarding the downstream task of perceptual quality prediction. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
A Comparison of Different Atmospheric Turbulence Simulation Methods for Image Restoration

Atmospheric turbulence deteriorates the quality of images captured by long-range imaging systems by introducing blur and geometric distortions to the captured scene. This leads to a drastic drop in performance when computer vision algorithms like object/face recognition and detection are performed on these images. In re-cent years, various deep learning-based atmospheric turbulence mitigation methods have been proposed in the literature. These methods are often trained using synthetically generated images and tested on real-world images. Hence, the performance of these restoration methods depends on the type of simulation used for training the network. In this paper, we systematically evaluate the effectiveness of various turbulence simulation methods on image restoration. In particular, we evaluate the performance of two state-or-the-art restoration networks using six simulations method on a real-world LRFID dataset consisting of face images degraded by turbulence. This paper will provide guidance to the researchers and practitioners working in this field to choose the suitable data generation models for training deep models for turbulence mitigation. The implementation codes for the simulation methods, source codes for the networks and the pre-trained models are available at https://github.com/Nithin-GK/Turbulence-Simulations
Using Deep Learning to Improve Detection and Decoding Of Barcodes

We propose an end-to-end pipeline for transforming raw images containing barcodes into sharp barcode images that can be accurately decoded. Our pipeline leverages recent deep learning approaches and consists of a rotation-decoupled detector (RDD) for oriented barcode detection and a deblurring model. The deblurring model uses a generative adversarial network (specifically, DeblurGAN-v2) trained on pairs of noisy and sharp images generated using ground truth numeric codes. Evaluation of the proposed pipeline using real barcode images enhanced by the DeblurGAN-v2 model shows a 14% improvement of the decoding rate as compared to the decoding rate obtained on the original images.
Self-Superflow: Self-Supervised Scene Flow Prediction in Stereo Sequences

In recent years, deep neural networks showed their exceeding capabilities in addressing many computer vision tasks including scene flow prediction. However, most of the advances are dependent on the availability of a vast amount of dense per pixel ground truth annotations, which are very difficult to obtain for real life scenarios. Therefore, synthetic data is often relied upon for supervision, resulting in a representation gap between the training and test data. Even though a great quantity of unlabeled real world data is available, there is a huge lack in self-supervised methods for scene flow prediction. Hence, we explore the extension of a self-supervised loss based on the Census transform and occlusion-aware bidirectional displacements for the problem of scene flow prediction. Regarding the KITTI scene flow benchmark, our method outperforms the corresponding supervised pre-training of the same network and shows improved generalization capabilities while achieving much faster convergence.
Optics Lens Design for Privacy-Preserving Scene Captioning

Image captioning is a challenging task that connects two major artificial intelligence fields: computer vision and natural language processing. Image captioning models use traditional images to generate a natural language description of the scene. However, the scene could contain private information that we want to hide but still generate the captions. Inspired by the trend of jointly designing optics and algorithms, this paper addresses the problem of privacy-preserving scene captioning. Our approach promotes privacy preservation, by hiding the faces in the images, during the acquisition process with a designed refractive camera lens while extracting useful features to perform image captioning. The refractive lens and an image captioning deep network architecture are optimized end-to-end to generate descriptions directly from the blurred images. Simulations show that our privacy-preserving approach degrades private visual attributes (e.g., face detection fails with our distorted images) while achieving comparable captioning performance with traditional non-private methods on the COCO dataset.
An Efficient Framework for Human Action Recognition Based on Graph Convolutional Networks

This paper presents a novel framework for skeleton-based Human Action Recognition (HAR) based on Graph Convolution Networks (GCNs). The proposed framework aims to increase human action recognition performance of GCN-based methods by incorporating a missing-joint-handling pre-processing step and a novel adjacency matrix construction method in a single human action recognition pipeline. The missing-joint-handling pre-processing step is utilized to infer missing data in the input sequence, which may occur due to imperfect skeleton extraction, based on imputation methods. The novel adjacency matrix construction method is executed offline to compute an improved weighted adjacency matrix specifically designed for HAR, which is utilized in every layer of the employed GCN. Moreover, both the pre-processing step and the adjacency construction method can be utilized along with any GCN architecture, allowing any GCN-based HAR method to be employed in the proposed framework. Experimental evaluation on two public datasets indicate favorable human action classification scores compared to the employed baseline and all competing methods both for 2D and 3D skeleton-based human action recognition, while using a GCN architecture with less learnable parameters.
Reference-Based Jpeg Image Artifacts Removal

Since information loss incurred in lossy image compression is irreversible, it is extremely challenging to achieve a satisfactory effect of artifacts removal by using the compressed image itself only. To solve the problem, a novel reference-based artifacts removal network (RARN) is proposed in this paper, which exploits a high-quality reference image to provide useful information for facilitating the removal of artifacts and the reconstruction of details. In our RARN, a feature extraction module is first established, which takes both the compressed and reference images as inputs and produces multi-scale feature pairs as outputs. Then, a feature transfer non-local block (FTNB) is developed to match the feature pairs and transfer relevant features from the reference image to the compressed one in the feature space. Finally, image information is recovered from the multi-scale outputs of FTNB by using a reconstruction module. Extensive experimental results clearly show that our proposed RARN can deliver superior performance over a number of state-of-the-art methods.
Efficient Self-Calibrated Convolution for Real-Time Image Super-Resolution

With the recent advancements in Convolutional Neural Net-work (CNN) architectures designs, many image processing tasks benefited from the design of such deep and complex networks including image super-resolution (SR). While deep and complex models achieve improved SR reconstruction, they lack the practicality in implementation for real-time applications, such as in mobile phones or online conferencing. This is due to the large number of parameters and excessive required multiply-accumulate operations (MACs). In this paper, an accurate real-time SR model structure is proposed. The proposed structure reduces the required number of MACs by performing all operations on low dimensional feature maps and reduces the model parameters by utilizing depthwise separable convolutional (DSC) layers. An efficient version of the recently introduced self-calibrated convolution with pixel attention (SC-PA) is introduced to further improve feature representation. Experimental results show that the proposed model improves performance in objective metrics, such as Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) index, over similar complexity real-time SR models.
Spline Human Motion Recovery

Simultaneous camera pose, 4D reconstruction of an object and deformation clustering from incomplete 2D point tracks in a video is a challenging problem. To solve it, in this work we introduce a union of piecewise subspaces to encode the 4D shape, where two modalities based on B-splines and Catmull-Rom curves are considered. We demonstrate that formulating the problem in terms of B-spline or Catmull-Rom functions, allows for a better physical interpretation of the resulting priors while C <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> and C <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> continuities are automatically imposed without needing any additional constraint. An optimization framework is proposed to sort out the problem in a unified, accurate, unsupervised and efficient manner. We extensively validate our claims on a wide range of human motions, including articulated and continuous deformations as well as those cases with noisy and missing measurements where our approach provides competing joint solutions.
Spatio-Temporal Attention Graph for Monocular 3d Human Pose Estimation

Single-view 3D human pose estimation (HPE) based on Graph Convolutional Networks (GCNs) currently suffers from problems such as insufficient spatial feature representation, difficult fusion of various information, and depth ambiguity in 2D to 3D pose mapping. This paper proposes a framework for monocular 3D human pose learning based on spatio-temporal attention graph. Firstly, we build a spatial graph feature acquisition scheme to obtain spatial semantic feature of 3D human pose with strong representativeness, by constructing a global to local attention graph through a coarse-to-fine way. And then we capture contextual information of temporal related images in the sequence as attention factors, evaluate their influence on the target image and achieve effective integration of spatio-temporal characteristics to mitigate the depth ambiguity problem. Extensive experimental results on two challenging benchmark datasets (Human3.6M and HumanEva-I) show that our method can effectively improve the accuracy of 3D HPE and outperform the state-of-the-arts.
Compression of User Generated Content Using Denoised References

Video shared over the internet is commonly referred to as user generated content (UGC). UGC video may have low quality due to various factors including previous compression. UGC video is uploaded by users, and then it is re-encoded to be made available at various levels of quality. In a traditional video coding pipeline the encoder parameters are optimized to minimize a rate-distortion criterion, but when the input signal has low quality, this results in sub-optimal coding parameters optimized to preserve undesirable artifacts. In this paper we formulate the UGC compression problem as that of compression of a noisy/corrupted source. The noisy source coding theorem reveals that an optimal UGC compression system is comprised of optimal denoising of the UGC signal, followed by compression of the denoised signal. Since optimal denoising is unattainable and users may be against modification of their content, we propose encoding the UGC signal, and using denoised references only to compute distortion, so the encoding process can be guided towards perceptually better solutions. We demonstrate the effectiveness of the proposed strategy for JPEG compression of UGC images and videos.
Entropy-Based Feature Extraction for Real-Time Semantic Segmentation

This paper introduces an efficient patch-based computational module, coined Entropy-based Patch Encoder (EPE) module, for resource-constrained semantic segmentation. The EPE module consists of three lightweight fully-convolutional encoders, each extracting features from image patches with a different amount of entropy. Patches with high entropy are being processed by the encoder with the largest number of parameters, patches with moderate entropy are processed by the encoder with a moderate number of parameters, and patches with low entropy are processed by the smallest encoder. The intuition behind the module is the following: as patches with high entropy contain more information, they need an encoder with more parameters, unlike low entropy patches, which can be processed using a small encoder. Consequently, processing part of the patches via the smaller encoder can significantly reduce the computational cost of the module. Experiments show that EPE can boost the performance of existing real-time semantic segmentation models with a slight increase in the computational cost. Specifically, EPE increases the mIOU performance of DFANet A by 0.9% with only 1.2% increase in the number of parameters and the mIOU performance of EDANet by 1% with 10% increase of the model parameters.
A Proper Orthogonal Decomposition Approach for Parameters Reduction of Single Shot Detector Networks

As a major breakthrough in artificial intelligence and deep learning, Convolutional Neural Networks have achieved an impressive success in solving many problems in several fields including computer vision and image processing. Real-time performance, robustness of algorithms and fast training processes remain open problems in these contexts. In addition object recognition and detection are challenging tasks for resource-constrained embedded systems, commonly used in the industrial sector. To overcome these issues, we propose a dimensionality reduction framework based on Proper Orthogonal Decomposition, a classical model order reduction technique, in order to gain a reduction in the number of hyperparameters of the net. We have applied such framework to SSD300 architecture using PASCAL VOC dataset, demonstrating a reduction of the network dimension and a remarkable speedup in the fine-tuning of the network in a transfer learning context.
A No-Reference Measure for Uneven Illumination Assessment on Laparoscopic Images

A frequent degradation in video-guided surgery and especially laparoscopic and endoscopic surgery is uneven illumination, due in large part to physical limitations of the sensors and uncontrolled lighting conditions in the internal structure of the digestive tract and particularly at the level of the intestines. Surgical as well as postoperative task accuracy can be seriously affected by the perceptual quality of the acquired images or videos. In this respect, a No-Reference Image Quality Assessment (NR-IQA) metric dedicated to uneven illumination is proposed in this paper. The key idea is to analyze the effect of contrast enhancement on the spatial distribution of the luminance component of the signal. The results obtained through extensive experiments, performed on a challenging dedicated database, have shown that the proposed metric significantly improves the state-of-the-art NR-IQA metrics when applied to this type of video content and distortion.