Reviewerly/exHarmony huggingface dataset:
Person Re-Identification in Panoramic Views Based on Bayesian Transformers

The panoramic view cameras offer more broad perspectives and continuous information for person re-identification (ReID). However, the panoramic-view videos suffer from objects distortion and bring more occlusion due to the fixed or moving capture points. This paper proposes a novel Bayesian Transformer Network (BTN) to adaptively capture the occlusion clues as Bayesian prior to guide the discriminative pedestrian-related feature extraction in the high-occlusion scenes. The Bayesian prior is built via a pre-trained CNN, which could recognize different occluded scenarios based on the severeness of noisy backgrounds. Moreover, to fully explore the occlusion prior, we propose to embed the semantic labels into a well-designed transformer network. By fostering the collaborative occlusion clues between the person and background, our method could achieve outstanding performance on both public benchmarks and panoramic view videos, which verifies the advantages of our BTN framework over existing methods.
Reverse Error Modeling for Improved Semantic Segmentation

We propose the concept of error-reversing autoencoders (ERA) for correcting pixel-wise errors made by an arbitrary semantic segmentation model. For this, we reframe the segmentation model as an error function applied to the ground truth labels. Then, we train an autoencoder to reverse this error function. During testing, the autoencoder reverses the approximated error function to correct the classification errors. We consider two sources of errors. First, we target the errors made by a model despite having being trained with clean, accurately labeled images. In this case, our proposed approach achieves an improvement of around 1% on the Cityscapes data set with the state-of-the-art DeepLabV3+ model. Second, we target errors introduced by compromised images. With JPEG-compressed images as input, our approach improves the segmentation performance by over 70% for high levels of compression. The proposed architecture is simple to implement, fast to train and can be applied to any semantic segmentation model as a post-processing step.
FewGAN: Generating from the Joint Distribution of a Few Images

We introduce FewGAN, a generative model for generating novel, high-quality and diverse images whose patch distribution lies in the joint patch distribution of a small number of N > 1 training samples. The method is, in essence, a hierarchical patch-GAN that applies quantization at the first coarse scale, in a similar fashion to VQ-GAN, followed by a pyramid of residual fully convolutional GANs at finer scales. Our key idea is to first use quantization to learn a fixed set of patch embeddings for training images. We then use a separate set of side images to model the structure of generated images using an autoregressive model trained on the learned patch embeddings of training images. Using quantization at the coarsest scale allows the model to generate both conditional and unconditional novel images. Subsequently, a patch-GAN renders the fine details, resulting in high-quality images. In an extensive set of experiments, it is shown that FewGAN outperforms baselines both quantitatively and qualitatively.
Conditional RGB-T Fusion for Effective Crowd Counting

A robust and efficient crowd counting framework forms a vital step towards the analysis of a crowded scene and finds its applications in social distance monitoring, traffic management, and video surveillance. We argue that visible RGB images fail to yield high-quality density maps owing to poor lighting conditions during the dark when an anomaly is more likely to occur. To tackle this scenario, we introduce a novel architecture Toggle-Fusion Network (TFNet) that effectively utilises a multimodal dataset, RGBT-CC, containing pairs of thermal and RGB images. Our approach eliminates the need for two branches for each of these modalities as proposed in the baseline by conditionally fusing the thermal and RGB images with the help of a toggle, saliency maps, and a rolling guidance filter. We conduct extensive experiments and delineate the importance of individual components of our method in the ablation study. TFNet, after evaluation on RGBT-CC, receives an RMSE value of 6.11 thereby establishing new state-of-the-art for the dataset. The code is made publicly available https://github.com/ShreyasSR/TFNet
Higher-Order Recurrent Network with Space-Time Attention for Video Early Action Recognition

Endowing visual agents with predictive capability is a key step towards video intelligence at scale. Early action recognition aims to predict the action labels before fully observing the complete video frames. Unlike action recognition, the model is asked to forecast the future or the effects by only observing the initial few frames. The strong reasoning ability over the temporal dimension is the key to success. To this end, in this paper, we propose a novel recurrent network with decomposed space-time attention and higher-order design to capture the temporal dependency associated with the specific actions. Our method achieves state-of-the-art performance on Something-Something and EPIC-Kitchens datasets under the early action recognition setting, showing evidence of predictive capability that we attribute to our higher-order recurrent design with space-time attention.
Infrared and Visible Image Registration for Airborne Camera Systems

Current airborne camera systems (ACS) used for surveillance and reconnaissance are generally equipped with cameras operating in different wavelengths. The registration and fusion of the images generated by these cameras enable a multimodal detection capability. In this paper, we propose a method to perform the geometric registration of infrared and visible imagery collected with an electro-optical and infrared (EO-IR) ACS. The registration process is a two-step approach. First, an initial transformation is computed using the expected orientation of the scene with respect to the camera and the camera’s intrinsic and extrinsic parameters. Then, the image alignment is refined using a technique based on mutual information which adapts the registration to the scene’s geometry and copes with video streams that are not perfectly synchronized. The image registration technique is implemented on an edge computing device and currently runs at 8 Hz. The intent is to use the resulting system to augment the capability of ACS that do no have built-in image registration and fusion capabilities. We present results obtained by applying the registration approach on real airborne imagery. We also evaluate the image registration error using physical markers installed on a building.
Dualfeat: Dual Feature Aggregation for Video Object Detection

Video object detection aims to detect and track each object in a given video. However, due to the problem of appearance deterioration in the video, it is still challenging to obtain good results when we apply traditional image object detection methods to videos. In this paper, we propose a new feature aggregation method, called Dual Feature Aggregation (DualFeat) for video object detection. By effectively combining the temporal and spatial attention mechanisms, we make full use of the temporal and spatial information in videos. Meanwhile, we leverage a real-time tracker to track detected objects in video frames, where features are aggregated again with previously obtained features. Such a way helps to obtain more comprehensive and richer features, greatly improving the accuracy of video object detection. We perform experiments on the ILSVRC2017 dataset, and the experimental results also verify the effectiveness of our method.
Fine-Tune Your Classifier: Finding Correlations with Temperature

Temperature is a widely used hyperparameter in various tasks involving neural networks, such as classification or metric learning, whose choice can have a direct impact on the model performance. Most of existing works select its value using hyperparameter optimization methods requiring several runs to find the optimal value. We propose to analyze the impact of temperature on classification tasks by describing a dataset as a set of statistics computed on representations on which we can build a heuristic giving us a default value of temperature. We study the correlation between these extracted statistics and the observed optimal temperatures. This preliminary study on more than a hundred combinations of different datasets and features extractors highlights promising results towards the construction of a general heuristic for temperature.
Mask-Guided Attention and Episode Adaptive Weights for Few-Shot Segmentation

Few-shot segmentation aims to segment objects with novel classes in a query image, given a support set which consists of few annotated support images. A key factor in few-shot segmentation is to effectively exploit information for the target classes from the support set. In addition, we argue that the overall quality of information available in each training episode varies depending on the given support samples. In this paper, we propose Mask-Guided Attention module to extract more beneficial features for few-shot segmentation from the support images. Taking advantage of the support masks, the area correlated to the foreground object is highlighted and enables the support encoder to extract comprehensive support features with contextual information. Furthermore, we propose Episode Adaptive Weight to balance the training between different episodes. It adaptively adjusts loss weight according to the difficulty of each episode determined by self-supervised segmentation loss of support images and encourages the model to pay more attention to more difficult episodes. Extensive experimental results including comparisons with the state-of-the-art methods and ablation studies demonstrate the effectiveness of the proposed method.
Improving Robustness to out-of-Distribution Data by Frequency-Based Augmentation

Although Convolutional Neural Networks (CNNs) have high accuracy in image recognition, they are vulnerable to adversarial examples and out-of-distribution data, and the difference from human recognition has been pointed out. In order to improve the robustness against out-of-distribution data, we present a frequency-based data augmentation technique that replaces the frequency components with other images of the same class. When the training data are CIFAR10 and the out-of-distribution data are SVHN, the Area Under Receiver Operating Characteristic (AUROC) curve of the model trained with the proposed method increases from 89.22% to 98.15%, and further increased to 98.59% when combined with another data augmentation method. Furthermore, we experimentally demonstrate that the robust model for out-of-distribution data uses a lot of high-frequency components of the image.
End-to-End Spatial-Angular Light Field Super-Resolution Using Parallax Structure Preservation Strategy

Limited by the sensor resolution of plenoptic cameras, it is challenging to obtain spatial and angular high resolution light field (LF). In this paper, we propose a method for spatial and angular super-resolution (SR) simultaneously while preserving parallax structure, which generates a high resolution (HR) dense LF from a low resolution (LR) sparse one with detailed textures. The proposed framework consists of four parts, including a depth estimation module to get scene geometry, an image warping process to generate intermediate warped views, an epipolar plane image (EPI) generation module to recover structural features, and a spatial-angular SR module to fully explore the informative features in 4D LF. Finally, the high quality HR scenes can be effectively reconstructed. Experimental results over several LF datasets demonstrate that our approach not only outperforms the existing spatial and angular SR methods separately, but also achieves better quantitative and qualitative results than the one-stage and two-stage spatial-angular approaches.
Differential Pseudo-Image for Skeleton-Based Dynamic Gesture Recognition

The approach described in this paper uses the potential of deep convolutional neural networks (DCNN), pre-trained to recognize images, to classify dynamic gestures. For this purpose, the sequences of articulated human poses, obtained with RGB-D imaging devices, and called skeletons, are replaced by space-time images. Since such images, observed by humans, do not have a clear physical interpretation, they are called pseudo-images. This paper reviews the existing mapping techniques and proposes a new one called Differential Pseudo-Image (DPI). The developed method is compared with approaches using other types of pseudo-images and with other methods of recognizing human actions available in the literature. It turns out, that for the Polish Sign Language (PSL) dataset and the publicly available UTD Multimodal Human Action Dataset (UTD-MHAD), DPI outperforms other forms of mapping the skeleton sequence. The method based on pseudo-images and convolutional neural networks has the potential that justifies further research in this direction.
JPEG Pleno Light Field Encoder with Breakpoint Dependent Affine Wavelet Transform for Disparity Maps

The JPEG Pleno light field encoder can perform disparity compensated view prediction for coding HDCA views. For operating in this prediction mode, disparity maps need to be communicated along with the 2D array of views. In this work we explore a new adaptive wavelet transform for coding disparity maps, known as tri-breakpoint (TriBRK) dependent DWT, that is currently being considered as part of JPEG 2000 Part-17 extensions. We show that the scalable TriBRK dependent DWT, defined on a hierarchical triangular grid, provides rate-distortion gains for coding disparity maps and improves the compression of HDCA views that rely upon the disparity information. Visual quality improvements are also observed, specifically at object boundaries of decoded views.
3D Head Pose Estimation Based on Graph Convolutional Network from A Single RGB Image

Research of head pose estimation in computer vision has been at the center of much attention. This work presents a framework based on adaptive graph convolution network (AGCN) to process both 2D and 3D facial landmarks extracted from the input RGB image. The network has a two-streams (teacher/3D-student/2D streams) architecture, trained with a 3D to 2D knowledge distillation training process, to transfer features of the 3D stream to the 2D stream for performance promotion. Several processing modules, such as depth-denoising for detected 3D landmarks, multi-stream fusion in inference, were also proposed for further increase of the prediction performance and robustness of our proposed method. In experiments, we follow standard protocols (in terms of datasets and metrices) to evaluate our performance. Three datasets 300W-LP, AFLW2000 and BIWI were used. The performance is measured in mean absolute error (MAE). We can achieve better performance compared to most of the state-of-the-art methods.
Latency Compensation Through Image Warping For Remote Rendering-Based Volumetric Video Streaming

Rendering multiple high-quality volumetric videos is still a challenge for today’s mobile devices. Remote rendering offloads complex rendering operations to a powerful server and provides the final result to the end device as a 2D video stream. A drawback of remote rendering is the significant increase of interaction latency that can degrade the user experience. We present a planar homography-based approach that compensates minor changes of the user’s head pose due to the interaction latency by warping the transmitted image on the client-side, just before it is sent to the display. In detail, we use the homography between the initial head pose when the image is rendered at the server and the latest available head pose of the user at the client. We perform controlled experiments using artificial camera traces to evaluate our approach. The results show that the proposed approach reduces the rendering errors significantly in terms of the mean-squared error between the rendered and reference images, especially combined with initial head motion prediction.
Class-Wise FM-NMS for Knowledge Distillation of Object Detection

The trade-off between accuracy and speed for an object detection model is important. When we implement an object detection model in embedded devices, a lightweight model can accelerate the detection speed. Meanwhile, the detection accuracy will be decreased. In this paper, we propose a knowledge distillation method for a lightweight object detection model. The proposed method introduces an improved feature map novel non-maximum suppression (FM-NMS) method. The improved FM-NMS uses different focus size with respect to each object class, which can suppress false positives and improve detection accuracy. In our experiments, we use onestage object detection methods, YOLOv4 as a teacher model and YOLOv4-tiny as a student model, and we apply the proposed method to them. The experimental results demonstrate that the proposed method improves the detection accuracy of the student model while maintaining the lightweight model size.
A Unified Framework for Masked and Mask-Free Face Recognition Via Feature Rectification

Face recognition under ideal conditions is now considered a well-solved problem with advances in deep learning. Recognizing faces under occlusion, however, still remains a challenge. Existing techniques often fail to recognize faces with both the mouth and nose covered by a mask, which is now very common under the COVID-19 pandemic. Common approaches to tackle this problem include 1) discarding information from the masked regions during recognition and 2) restoring the masked regions before recognition. Very few works considered the consistency between features extracted from masked faces and from their mask-free counterparts. This resulted in models trained for recognizing masked faces of-ten showing degraded performance on mask-free faces. In this paper, we propose a unified framework, named Face Feature Rectification Network (FFR-Net), for recognizing both masked and mask-free faces alike. We introduce rectification blocks to rectify features extracted by a state-of-the-art recognition model, in both spatial and channel dimensions, to minimize the distance between a masked face and its mask-free counterpart in the rectified feature space. Experiments show that our unified framework can learn a rectified feature space for recognizing both masked and mask-free faces effectively, achieving state-of-the-art results. Project code: https://github.com/haoosz/FFR-Net
Learning an Evolved Mixture Model for Task-Free Continual Learning

Recently, continual learning (CL) has gained significant interest because it enables deep learning models to acquire new knowledge without forgetting previously learnt information. However, most existing works require knowing the task identities and boundaries, which is not realistic in a real context. In this paper, we address a more challenging and realistic setting in CL, namely the Task-Free Continual Learning (TFCL) in which a model is trained on non-stationary data streams with no explicit task information. To address TFCL, we introduce an evolved mixture model whose network architecture is dynamically expanded to adapt to the data distribution shift. We implement this expansion mechanism by evaluating the probability distance between the knowledge stored in each mixture model component and the current memory buffer using the Hilbert Schmidt Independence Criterion (HSIC). We further introduce two simple dropout mechanisms to selectively remove stored examples in order to avoid memory overload while preserving memory diversity. Empirical results demonstrate that the proposed approach achieves excellent performance.
Microscale Image Enhancement Via PCA and Well-Exposedness Maps

The restrictions of accessing high-end microscopes, microscale cameras and high-tech imaging lenses result in a high demand on low-cost microscopes. However, low-cost microscopes are facing with many image capture and quality limitations due to incompatible equipped instrumentation. This study aims at overcoming illumination and contrast problems, color aberration issues, and blur and noise corruption in low-cost microscopes at high image magnification rates. The three color channels of the input image are enhanced via principal component analysis and well-exposedness feature maps by means of cross-channel histogram matching, Laplacian and non-local means filtering. The proposed approach produces sharper, and better color and illumination fixed outputs when compared to existing methods in literature.
A Novel System for Deep Contour Classifiers Certification Under Filtering Attacks

The lack of interpretability, explainability and transparency makes deep learning models untrusted to perform reliably for making critical decisions. Despite their evaluation against disturbances including geometric transformations, occlusion and convolutional noises in the case of DNN-based image classifiers, the evaluation of contour classifiers has only been studied against rigid displacements (rotation and translation). In this paper, we introduce ContourCertif: a new system to certify deep contour classifiers against convolutional attacks. We use the abstract interpretation theory in order to formulate the Lower and Upper Bounds with abstract intervals to support other classes of advanced attacks including filtering.
Robust Grid Detection in Historical Map Images

This paper presents a novel method for grid detection in historical maps. The approach is based on Hough transform accompanied with a sophisticated post-processing. They are applied to detect the grid that consists of graticule lines. It works without any training and does not require any annotated data. The proposed approach is very efficient in detecting the rectangular grid and the intersection points as shown in the international "MapSeg" segmentation competition, where it won the Task 3 with a significant margin. The robustness of the proposed method has been demonstrated by evaluating on another dataset composed of significantly different cadastral map images with excellent results.
On The Limits of Perceptual Quality Measures for Enhanced Underwater Images

The appearance of objects in underwater images is degraded by the selective attenuation of light, which reduces contrast and causes a colour cast. This degradation depends on the water environment, and increases with depth and with the distance of the object from the camera. Despite an increasing volume of works in underwater image enhancement and restoration, the lack of a commonly accepted evaluation measure is hindering the progress as it is difficult to compare methods. In this paper, we review commonly used colour accuracy measures, such as colour reproduction error and CIEDE2000, and no-reference image quality measures, such as UIQM, UCIQE and CCF, which have not yet been systematically validated. We show that none of the no-reference quality measures satisfactorily rates the quality of enhanced underwater images and discuss their main shortcomings. Images and results are available at https://puiqe.eecs.qmul.ac.uk.
Funque: Fusion of Unified Quality Evaluators

Fusion-based quality assessment has emerged as a powerful method for developing high-performance quality models from quality models that individually achieve lower performances. A prominent example of such an algorithm is VMAF, which has been widely adopted as an industry standard for video quality prediction along with SSIM. In addition to advancing the state-of-the-art, it is imperative to alleviate the computational burden presented by the use of a heterogeneous set of quality models. In this paper, we unify "atom" quality models by computing them on a common transform domain that accounts for the Human Visual System, and we propose FUNQUE, a quality model that fuses unified quality evaluators. We demonstrate that in comparison to the state-of-the-art, FUNQUE offers significant improvements in both correlation against subjective scores and efficiency, due to computation sharing.