Reviewerly/exHarmony huggingface dataset:
Recognition of Freely Selected Keypoints on Human Limbs

Nearly all Human Pose Estimation (HPE) datasets consist of a fixed set of keypoints. Standard HPE models trained on such datasets can only detect these keypoints. If more points are desired, they have to be manually annotated and the model needs to be retrained. Our approach leverages the Vision Transformer architecture to extend the capability of the model to detect arbitrary keypoints on the limbs of persons. We propose two different approaches to encode the desired keypoints. (1) Each keypoint is defined by its position along the line between the two enclosing key-points from the fixed set and its relative distance between this line and the edge of the limb. (2) Keypoints are defined as coordinates on a norm pose. Both approaches are based on the TokenPose [12] architecture, while the key-point tokens that correspond to the fixed keypoints are replaced with our novel module. Experiments show that our approaches achieve similar results to TokenPose on the fixed keypoints and are capable of detecting arbitrary keypoints on the limbs.
A New Non-central Model for Fisheye Calibration

A new non-central model suitable for calibrating fisheye cameras is proposed. It is a direct extension of the popular central model developed by Scaramuzza et al., used by Matlab Computer Vision Toolbox fisheye calibration tool. It allows adapting existing applications that are using this central model to a non-central projection that is more accurate, especially when objects captured in the images are close to the camera, and it makes it possible to switch easily between the more accurate non-central characterization of the fisheye camera and the more convenient central approximation, as needed. It is shown that the algorithms proposed by Scaramuzza et al. for their central model can be modified to accommodate the angle dependent axial viewpoint shift. This means, besides other, that a similar process can be used for calibration involving the viewpoint shift characterization and a user-friendly calibration tool can be produced with this new non-central model that does not require the user to provide detailed lens design specifications or an educated guess for the initial parameter values. Several other improvements to the Scaramuzza's central model are also introduced, helping to improve the performance of both the central model, and its non-central extension.
On the Effect of Atmospheric Turbulence in the Feature Space of Deep Face Recognition

When captured over long distances, image quality is degraded by inconsistent refractive indexes in the atmosphere. This effect, known as Atmospheric Turbulence (AT), leads to lower performance for vision-based biometric systems such as face recognition. To account for AT, the literature has proposed methods to restore face-images from atmospheric turbulence, but has limited success. There is still a need to understand how atmospheric turbulence breaks recognition performance. We offer a first-look in this direction by providing a study on the effect of atmospheric turbulence in the feature space of deep-learning-based face recognition. We present results on recognition performance and feature space transformation caused by a wide range of AT levels. In deep feature space, we find interesting phenomena such as increasing feature magnitudes, which contradicts the expected result from the literature. From our results, we are able to identify an effect that makes face recognition under atmospheric turbulence uniquely difficult, which we call feature defection. In total, our findings suggest several areas of available improvement which can be used as a guideline for further progress in building models that are robust to AT.
Physics Based Image Deshadowing Using Local Linear Model

Image deshadowing algorithms remove shadows from images. This requires both detecting where the shadow is and, once detected, removing it from the image. This work focuses on the shadow removal part. We follow a common physical shadow formation model and learn its parameters using a deep neural network. Our model consists of an existing network for shadow detection, and a novel network for shadow removal. The shadow removal network gets the predicted mask of the shadow region and the shadow image and predicts six parameters per pixel. Remarkably, a straightforward network architecture, that is considerably smaller compared to alternative methods, produces better results on standard datasets <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
Image Quality Assessment with Gradient Siamese Network

In this work, we introduce Gradient Siamese Network (GSN) for image quality assessment. The proposed method is skilled in capturing the gradient features between distorted images and reference images in full-reference image quality assessment (IQA) task. We utilize Central Differential Convolution to obtain both semantic features and detail difference hidden in image pair. Furthermore, spatial attention guides the network to concentrate on regions related to image detail. For the low-level, mid-level, and high-level features extracted by the network, we innovatively design a multi-level fusion method to improve the efficiency of feature utilization. In addition to the common mean square error supervision, we further consider the relative distance among batch samples and successfully apply KL divergence loss to the image quality assessment task. We experimented the proposed algorithm GSN on several publicly available datasets and proved its superior performance. Our network won the second place in NTIRE 2022 Perceptual Image Quality Assessment Challenge track 1 Full-Reference [1].
Domain Adaptable Normalization for Semi-Supervised Action Recognition in the Dark

Action recognition in the dark is gaining more and more attention with the rapid development of intelligent recognition applications in real-world applications, e.g. self-driving at night and night surveillance. However, limited by the expensive labeling cost, it is impractical to produce a large-scale labeled dataset only for dark environments. Therefore, a practical solution adopted is to transfer models trained from clear environments to dark environments through semi-supervised learning. However, prior works rely heavily on additional efforts such as extra annotations, or extra sensors. To this end, we proposed a novel and simple Domain Adaptable Normalization (DANorm) method to align different domains directly, which consists of feature normalization, angle constraint and the Pseudo-Label. Specifically, the proposed DANorm method enables the model automatically learning the associated features between labeled source domain and unlabeled target domain by constraining the feature subspace vectors. Experimental results show that our model achieves superiority performance on Semi-supervised ARID dataset. Code is available at: https://github.com/NikkiElwin/DANorm.
Few-Shot Image Classification Benchmarks are Too Far From Reality: Build Back Better with Semantic Task Sampling

Every day, a new method is published to tackle Few-Shot Image Classification, showing better and better performances on academic benchmarks. Nevertheless, we observe that these current benchmarks do not accurately represent the real industrial use cases that we encountered. In this work, through both qualitative and quantitative studies, we expose that the widely used benchmark tieredImageNet is strongly biased towards tasks composed of very semantically dissimilar classes e.g. bathtub, cabbage, pizza, schipperke, and cardoon. This makes tieredImageNet (and similar benchmarks) irrelevant to evaluate the ability of a model to solve real-life use cases usually involving more fine-grained classification. We mitigate this bias using semantic information about the classes of tieredImageNet and generate an improved, balanced benchmark. Going further, we also introduce a new benchmark for Few-Shot Image Classification using the Danish Fungi 2020 dataset. This benchmark proposes a wide variety of evaluation tasks with various fine-graininess. Moreover, this benchmark includes many-way tasks (e.g. composed of 100 classes), which is a challenging setting yet very common in industrial applications. Our experiments bring out the correlation between the difficulty of a task and the semantic similarity between its classes, as well as a heavy performance drop of state-of-the-art methods on many-way few-shot classification, raising questions about the scaling abilities of these methods. We hope that our work will encourage the community to further question the quality of standard evaluation processes and their relevance to real-life applications.
Cross Transferring Activity Recognition to Word Level Sign Language Detection

The lack of large scale labelled datasets in word-level sign language recognition (WSLR) poses a challenge to detecting sign language from videos. Most WSLR approaches operate on datasets that do not model real-world settings very well, as they do not have a high degree of variability in terms of signers, background, lighting and inter signer variation. We chose the MS-ASL dataset to overcome these limitations as they model open-world settings very well. This paper benchmarks successful action recognition architectures on the MS-ASL dataset using transfer learning. We have achieved new state-of-the-art accuracy (92.35%) with an improvement of 7.03% over the previous state-of-the-art introduced by the MS-ASL paper. We have analyzed how action-recognition architectures fair in the task of WSLR, and we propose SlowFast 8×8 ResNet 101 as a robust and suitable architecture for the task of WSLR.
Dual-Domain Image Synthesis using Segmentation-Guided GAN

We introduce a segmentation-guided approach to synthesise images that integrate features from two distinct domains. Images synthesised by our dual-domain model belong to one domain within the semantic-mask, and to another in the rest of the image - smoothly integrated. We build on the successes of few-shot StyleGAN and single-shot semantic segmentation to minimise the amount of training required in utilising two domains.The method combines few-shot cross-domain StyleGAN with a latent optimiser to achieve images containing features of two distinct domains. We use a segmentation-guided perceptual loss, which compares both pixel-level and activations between domain-specific and dual-domain synthetic images. Results demonstrate qualitatively and quantitatively that our model is capable of synthesising dual-domain images on a variety of objects (faces, horses, cats, cars), domains (natural, caricature, sketches) and part-based masks (eyes, nose, mouth, hair, car bonnet). The code is publicly available <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
Auxiliary Learning for Self-Supervised Video Representation via Similarity-based Knowledge Distillation

Despite the outstanding success of self-supervised pretraining methods for video representation learning, they generalise poorly when the unlabeled dataset for pretraining is small or the domain difference between unlabelled data in source task (pretraining) and labeled data in target task (finetuning) is significant. To mitigate these issues, we propose a novel approach to complement self-supervised pretraining via an auxiliary pretraining phase, based on knowledge similarity distillation, auxSKD, for better generalisation with a significantly smaller amount of video data, e.g. Kinetics-100 rather than Kinetics-400. Our method deploys a teacher network that iteratively distils its knowledge to the student model by capturing the similarity information between segments of unlabelled video data. The student model meanwhile solves a pretext task by exploiting this prior knowledge. We also introduce a novel pretext task, Video Segment Pace Prediction or VSPP, which requires our model to predict the playback speed of a randomly selected segment of the input video to provide more reliable self-supervised representations. Our experimental results show superior results to the state of the art on both UCF101 and HMDB51 datasets when pretraining on K100 in apple-to-apple comparisons. Additionally, we show that our auxiliary pretraining, auxSKD, when added as an extra pretraining phase to recent state of the art self-supervised methods (i.e. VCOP, VideoPace, and RSPNet), improves their results on UCF101 and HMDB51. Our code is available at https://github.com/Plrbear/auxSKD.
Doppelgänger Saliency: Towards More Ethical Person Re-Identification

Modern surveillance systems have become increasingly dependent on artificial intelligence to provide actionable information for real-time decision making. A critical question relates to how these systems handle difficult ethical dilemmas, such as the re-identification of similar looking individuals. Potential misidentification of individuals can have severe negative consequences, as evidenced by recent headlines of individuals who were wrongly targeted for crimes they did not commit based on false matches. A computer vision-based saliency algorithm is proposed to help identify pixel-level differences in pairs of images containing visually similar individuals, which we term "doppelgängers." The computed saliency maps can alert human users of the presence of doppelgängers and provide important visual evidence to reduce the potential of false matches in these high-stakes situations. We show both qualitative and quantitative saliency results on doppelgängers found in a video-based person re-identification dataset (MARS) using three different state-of-the-art models. Our results suggest that this novel use of visual saliency can improve overall outcomes by helping human users in the person re-identification setting, while assuring the ethical and trusted operation of surveillance systems.
Performance Prediction for Semantic Segmentation by a Self-Supervised Image Reconstruction Decoder

In supervised learning, a deep neural network’s performance is measured using ground truth data. In semantic segmentation, ground truth data is sparse, requires an expensive annotation process, and, most importantly, it is not available during online operation. To tackle this problem, recent works propose various forms of performance prediction. However, they either rely on inference data histograms, additional sensors, or additional training data. In this paper, we propose a novel per-image performance prediction for semantic segmentation, with (i) no need for additional sensors (sensor efficiency), (ii) no need for additional training data (data efficiency), and (iii) no need for a dedicated retraining of the semantic segmentation (training efficiency). Specifically, we extend an already trained semantic segmentation network having fixed parameters with an image reconstruction decoder. After training and a subsequent regression, the image reconstruction quality is evaluated to predict the semantic segmentation performance. We demonstrate our method’s effectiveness with a new state-of-the-art benchmark both on KITTI and Cityscapes for image-only input methods, on Cityscapes even excelling a LiDAR-supported benchmark.
Adaptive Differential Filters for Fast and Communication-Efficient Federated Learning

Federated learning (FL) scenarios inherently generate a large communication overhead by frequently transmitting neural network updates between clients and server. To minimize the communication cost, introducing sparsity in conjunction with differential updates is a commonly used technique. However, sparse model updates can slow down convergence speed or unintentionally skip certain update aspects, e.g., learned features, if error accumulation is not properly addressed. In this work, we propose a new scaling method operating at the granularity of convolutional filters which 1) compensates for highly sparse updates in FL processes, 2) adapts the local models to new data domains by enhancing some features in the filter space while diminishing others and 3) motivates extra sparsity in updates and thus achieves higher compression ratios, i.e., savings in the overall data transfer. Compared to unscaled updates and previous work, experimental results on different computer vision tasks (Pascal VOC, CIFAR10, Chest X-Ray) and neural networks (ResNets, MobileNets, VGGs) in uni-, bidirectional and partial update FL settings show that the proposed method improves the performance of the central server model while converging faster and reducing the total amount of transmitted data by up to 377×.
Class-wise Thresholding for Robust Out-of-Distribution Detection

We consider the problem of detecting Out-of-Distribution (OoD) input data when using deep neural networks, and we propose a simple yet effective way to improve the robustness of several popular OoD detection methods against label shift. Our work is motivated by the observation that most existing OoD detection algorithms consider all training/test data as a whole, regardless of which class entry each input activates (inter-class differences). Through extensive experimentation, we have found that such practice leads to a detector whose performance is sensitive and vulnerable to label shift. To address this issue, we propose a class-wise thresholding scheme that can apply to most existing OoD detection algorithms and can maintain similar OoD detection performance even in the presence of label shift in the test distribution.
CPS: Full-Song and Style-Conditioned Music Generation with Linear Transformer

Many deep music generation algorithms have recently been able to produce good-sounding music, but there have been few studies on controlled generation. In this process, the human sense of participation is usually very weak, and it is difficult to integrate one’s own musical motivation into the creation. In this study, we will introduce CPS (Compound word with style), a model that can specify a target style and generate a complete musical composition from scratch. We first added the genre meta-information to the music representation and distinguished it from other low-level music representations, thus strengthening the influence of the control signal. We modeled with the linear transformer, while used an adaptive strategy with different settings for different types of music tokens to reduce the probability of disharmonic music. The experiments show that, when compared to the baseline model, our model performs better in terms of basic music metrics as well as metrics for evaluating controlled ability.
An Optical Tool to Optimize the Output of a Photonic Integrated Chip Architecture

Photonic Integrated Chips (PICs) are an up-andcoming solution for base building blocks of quantum computers.The ready accessibility of photonics integrated systems and the fact that they allow for direct monolithic integration with electronics in a compact architecture have pushed the academic community to inspect their capabilities further.Still, at this stage, their characterization, control, and fine-tuning is an open issue for which this work aims to provide a viable solution.In this paper we present a method for an easy setup and an efficient architecture to simultaneously optimize multiple outputs of a PIC.The proposed solution can be embedded in an automation tool for tuning and verifying output on-and offchip detectors.We fabricated a Silicon Oxynitride (SiON)-based PIC featuring a network of Mach-Zehnder interferometers (MZI) arranged in Clements architecture driven by Titanium-Titanium Nitride (TiTiN) thermistors.We interfaced the chip electronically through commercial driver modules and optically through optical guides and controlled it through a Python optimization algorithm.Finally, we retrieved the feedback of the control both by sensing it through integrated silicon photodetectors and by an image processing algorithm capable of inspecting the light intensity scattered by the waveguides in a custom region.
Multi-view Multi-label Canonical Correlation Analysis for Cross-modal Matching and Retrieval

In this paper, we address the problem of cross-modal retrieval in presence of multi-view and multi-label data. For this, we present Multi-view Multi-label Canonical Correlation Analysis (or MVMLCCA), which is a generalization of CCA for multi-view data that also makes use of high-level semantic information available in the form of multi-label annotations in each view. While CCA relies on explicit pairings/associations of samples between two views (or modalities), MVMLCCA uses the available multi-label annotations to establish correspondence across multiple (two or more) views without the need of explicit pairing of multi-view samples. Extensive experiments on two multi-modal datasets demonstrate that the proposed approach offers much more flexibility than the related approaches without compromising on scalability and cross-modal retrieval performance. Our code and precomputed features are available at https://github.com/Rushil231100/MVMLCCA.
S2F2: Single-Stage Flow Forecasting for Future Multiple Trajectories Prediction

In this work, we present a single-stage framework, named S2F2, for forecasting multiple human trajectories from raw video images by predicting future optical flows. S2F2 differs from the previous two-stage approaches in that it performs detection, Re-ID, and forecasting of multiple pedestrians at the same time. Unlike the prior approaches, the computational burden of S2F2 remains consistent even as the number of pedestrians grows. The experimental results demonstrate that S2F2 is able to outperform two conventional forecasting algorithms and a recent learning-based two-stage model [1], while maintaining its tracking performance on par with the contemporary MOT models.
Exposure Correction Model to Enhance Image Quality

Exposure errors in an image cause a degradation in the contrast and low visibility in the content. In this paper, we address this problem and propose an end-to-end expo-sure correction model in order to handle both under- and overexposure errors with a single model. Our model contains an image encoder, consecutive residual blocks, and image decoder to synthesize the corrected image. We utilize perceptual loss, feature matching loss, and multi-scale discriminator to increase the quality of the generated image as well as to make the training more stable. The experimental results indicate the effectiveness of proposed model. We achieve the state-of-the-art result on a large-scale exposure dataset. Besides, we investigate the effect of exposure set-ting of the image on the portrait matting task. We find that under- and overexposed images cause severe degradation in the performance of the portrait matting models. We show that after applying exposure correction with the proposed model, the portrait matting quality increases significantly. https://github.com/yamand16/ExposureCorrection.