Reviewerly/exHarmony huggingface dataset:
Subjective and Objective Quality Assessment of High-Motion Sports Videos at Low-Bitrates

Videos often have to be transmitted and stored at low bitrates due to poor network connectivity during adaptive bitrate streaming. Designing optimal bitrate ladders that would select the perceptually-optimized resolution, frame-rate, and compression level for low-bitrate videos for adaptive streaming across the internet is therefore a task of great interest. Towards that end, we conducted the first large-scale study of medium and low-bitrate videos from live sports for two codecs (Elemental AVC and HEVC) and created the Amazon Prime Video Low-Bitrate Sports (APV LBS) dataset. The study involved 94 participants and 742 videos, with more than 23,000 human opinion scores collected in total. We analyzed the data obtained and we also conducted an extensive evaluation of objective Video Quality Assessment (VQA) algorithms and benchmarked their performance, and make recommendations on bitrate ladder design. We're making the metadata and VQA features available at https://github.com/JoshuaEbenezer/lbmfr-public.
Adaptive Radial Projection on Fourier Magnitude Spectrum for Document Image Skew Estimation

Skew estimation is one of the vital tasks in document processing systems, especially for scanned document images, because its performance impacts subsequent steps directly. Over the years, an enormous number of researches focus on this challenging problem in the rise of digitization age. In this research, we first propose a novel skew estimation method that extracts the dominant skew angle of the given document image by applying an Adaptive Radial Projection on the 2D Discrete Fourier Magnitude spectrum. Second, we introduce a high quality skew estimation dataset DISE-2021 to assess the performance of different estimators. Finally, we provide comprehensive analyses that focus on multiple improvement aspects of Fourier-based methods. Our results show that the proposed method is robust, reliable, and outperforms all compared methods. The source code is available at github.com/phamquiluan/jdeskew.
Ventriloquist-Net: Leveraging Speech Cues for Emotive Talking Head Generation

In this paper, we propose Ventriloquist-Net: A Talking Head Generation model that uses only a speech segment and a single source face image. It places emphasis on emotive expressions. Cues for generating these expressions are implicitly inferred from the speech clip only. We formulate our framework to comprise of independently trained modules to expedite convergence. This not only allows extension to datasets in a semi-supervised manner but also facilitates handling in-the-wild source images. Quantitative and qualitative evaluations on generated videos demonstrate state-of-the-art performance even on unseen input data. Implementation and supplementary videos are available at https://github.com/dipnds/VentriloquistNet.
Trellis-Coded Quantization for End-to-End Learned Image Compression

The performance of variational auto-encoders (VAE) for image compression has steadily grown in recent years, thus becoming competitive with advanced visual data compression technologies. These neural networks transform the source image into a latent space with a channel-wise representation. In most works, the latents are scalar quantized before being entropy coded. On the other hand, vector quantizers generally achieve denser packings of high-dimensional data regardless of the source distribution. Hence, low-complexity variants of these quantizers are implemented in the compression standards JPEG 2000 and Versatile Video Coding. In this paper we demonstrate coding gains by using trellis-coded quantization (TCQ) over scalar quantization. For the optimization of the networks with regard to TCQ, we employ a specific noisy representation of the features during the training stage. For variable-rate VAEs, we obtained 7.7% average BD-rate savings on the Kodak images by using TCQ over scalar quantization. When different networks per target bitrate are optimized, we report a relative coding gain of 2.4% due to TCQ.
Style Transfer Using Optimal Transport Via Wasserstein Distance

Universal style transfer has been proven to be effective through CNN models and VGG networks. However, how well to apply the algorithm’s style is a separate issue. This problem is especially evident in high-resolution images in which case the division and color at the boundary lines are more complex than low-resolution images. For the WCT model, if the image segment is smaller than the filter size, it will be blurred. High-resolution images have much larger number of small segments and the existing WCT model cannot render them clearly. This paper proposes two methods. It uses the Wasserstein distance-based optimal transport so that the resulting style distribution is the same up to the secondary statistics when a style image is applied to the content image, and proposes collaborative distillation, a method to overcome the encoder and decoder dependency of the WCT module. We propose a module that combines these two methods to apply subtle style transfer even to high-resolution images.
An Unsupervised Parameter-Free Nuclei Segmentation Method for Histology Images

An unsupervised nuclei segmentation method for histology images is proposed in this work. It consists of three modules applied to each of non-overlapping blocks: 1) data-driven color transform for dimension reduction, 2) fully-automated adaptive binarization, and 3) incorporation of geometric priors with morphological processing. The method is called CBM, which comes from the first letter of the three modules – "Color transform", "Binarization" and "Morphological processing". Experiments on the MoNuSeg dataset validate the effectiveness of the proposed CBM method. It outperforms all other unsupervised methods and offers a competitive standing among supervised models based on the Aggregated Jaccard Index (AJI) met-ric.
RW-HAZE: A Real-World Benchmark Dataset to Evaluate Quantitatively Dehazing Algorithms

Most existing image dehazing approaches are able to achieve desirable results whose differences are too subtle for people to qualitatively judge. Therefore, it is important to adopt quantitative assessment on real-world hazy images. However, many dehazing works have not been quantitatively evaluated on real-world hazy images due to the lack of appropriate real-world datasets. In this work, we attempt to address the issue and present a well-aligned real-world benchmark dataset, namely RW-Haze, for image dehazing evaluation, which had been lacking for a long period of time. It contains 210 pairs of well-aligned haze-free images and hazy images with distinct haze densities, which were captured from six cities in China by fixed cameras. To the best of our knowledge, RW-Haze is the first real-world dataset that is made up of well-aligned image pairs of haze-free and hazy images with diverse haze levels. We select 13 state-of-the-art single image dehazing works for making comprehensive evaluations among them on RW-Haze dataset. Experimental results show that there still exist rich rooms for image dehazing research to improve its robustness on natural hazy images, especially dense haze scenes.
Image Deblurring Using Deep Multi-Scale Distortion Prior

Deep neural networks have recently advanced state-of-the-art in motion deblurring. However, non-uniform non-blind image deblurring has not been studied in depth. State-of-the-art methods shows improvement over conventional algorithms, but they are still not feasible for mobile deployment. Having informative prior information could improve performance of non-uniform deblurring. In this work, we propose a new deep framework that allows extracting spatially variant latent feature to Distortion Prior map from a pair of calibration sharp-blur images, without having to capture or model training dataset. We propose to use multi-scale Distortion Prior map that can fully utilize spatially variant information in the further restoration via multi-scale attention mechanism. Unlike prior art, we use image pyramid at decoder side, by fusing its fine level with coarse level of feature map via level attention and by injecting Distortion Prior at various resolution levels. Experiments show that proposed network outperforms state-of-the-art deblur networks both in terms of image quality and inference time. We demonstrate that proposed framework can successfully deblur non-uniform, non-blind applications, such as defocus blur removal. Being computationally efficient, it is feasible for mobile deployment.
ICIP 2022 Challenge: PEDCMI, TOOD Enhanced by Slicing-Aided Fine-Tuning and Inference

This paper describes the approach for the Parasitic Egg Detection and Classification in Microscopic Images challenge. Our solution relies on a robust deep learning pipeline implementing a five-fold training schema to pursue the challenge goal. The final methodology utilizes the TOOD model, further enhanced by slicing-aided fine-tuning and inference. The slicing helps to overcome the image size invariability of the dataset and allows the model to access all images in high resolution, and consequently helps it learn detailed features needed to distinguish different classes and find a precise object position. Our results demonstrate the importance of proper data analysis and consequent pre and post-processing to improve prediction performance.
Transformer Visual Tracker Based on Template Features Corresponding to Foreground Region

In visual tracking, the size of template patch on image is usually several times the size of object bounding box. So the background information around object would be encoded into some template features. However, these features would also be matched with search features, which interferes with the tracker’s ability to accurately separate the object from the background. In this work, we present a novel feature fusion network based on Transformer for visual tracking. Specifically, to reduce the interference of background information in template patch, we extract the template features corresponding to foreground region on image, called TFFR, and fuse them with search features by attention mechanism. On that basis, we design a concise Transformer visual tracker based on TFFR, called TVT-TFFR. Extensive experiments show that our TVT-TFFR achieves state-of-the-art performance on several prevalent tracking benchmarks, and runs at 38 FPS, meeting the real-time requirement.
Translation of Illustration Artist Style Using Sailormoonredraw Data

The decision to draw requires answers to two questions: what to draw and how to draw. The latter refers to artist style and is an important factor in creating any illustration. In this paper, we propose a novel task, artist style translation, which translates one artist’s illustration into that of another artist style using deep learning. To solve this task in a supervised manner, we created a novel illustration dataset, SailormoonDataset, which consists of more than 2,000 artist’s stylistic illustrations of the same content. In addition, we propose a method based on the Swapping Autoencoder by introducing a new loss function for supervised learning and using multiple images to represent an artist style. We translate the face illustration of Sailor Moon into styles of different artists. By comparing the current results to those of the Swapping Autoencoder, we find that the proposed method successfully achieves the artist style translation.
MLS-GAN: Multi-Level Semantic Guided Image Colorization

Image colorization predicts plausible color versions of given grayscale images. Recently, several methods incorporate image semantics to assist image colorization and have shown impressive performance. To further exploit and take full advantage of more semantic information, in this paper, we propose a Multi-Level Semantic guided Generative Adversarial Network (MLS-GAN) for image colorization. Specifically, we utilize three different levels of semantics to guide the colorization process: image level, segmentation level and contextual level. Image-level classification semantics is used to learn category and high-level semantics, ensuring the reasonability of color results. At the segmentation level, multi-scale saliency map semantics is extracted to provide figure-background separation information, which can efficiently alleviate semantic confusion, especially for images with complex backgrounds. Furthermore, we novelly use non-local blocks to capture long-range semantic dependencies at the contextual level. Experiments show that our method enhances color consistency and can produce more vivid color in visually important regions, outperforming state-of-the-art methods qualitatively and quantitatively.
Revisiting the Efficiency of UGC Video Quality Assessment

UGC video quality assessment (UGC-VQA) is a challenging research topic due to the high video diversity and limited public UGC quality datasets. State-of-the-art (SOTA) UGC quality models tend to use high complexity models, and rarely discuss the trade-off among complexity, accuracy, and generalizability. We propose a new perspective on UGC-VQA, and show that model complexity may not be critical to the performance, whereas a more diverse dataset is essential to train a better model. We illustrate this by using a light weight model, UVQ-lite, which has higher efficiency and better generalizability (less overfitting) than baseline SOTA models. We also propose a new way to analyze the sufficiency of the training set, by leveraging UVQ’s comprehensive features. Our results motivate a new perspective about the future of UGC-VQA research, which we believe is headed toward more efficient models and more diverse datasets.
Deep Incremental Optical Flow Coding For Learned Video Compression

This work addresses motion coding in end-to-end learned video compression. The efficiency of motion coding is critical at low bit rates, at which a large portion of the bitstream signals motion information. Most end-to-end learned video codecs adopt an intra-coding approach to coding motion information as individual optical flow maps. Some recent studies introduce predictive motion coding to encode optical flow map residuals. Still, motion coding remains an active research area for learned video compression. We present an incremental optical flow coding scheme. It first leverages an extrapolated flow together with the reference frame in estimating an incremental flow between the reference and the target frames for efficient motion coding. It then derives the final flow map for motion compensation by integrating the incremental and the extrapolated flows in a double-warping scheme. Experimental results on commonly used datasets show the superiority of our method over predictive motion coding and other advanced schemes.
Strong-Weak Integrated Semi-Supervision for Unsupervised Domain Adaptation

Unsupervised domain adaptation (UDA) focuses on transferring knowledge learned in the labeled source domain to the unlabeled target domain. Semi-supervised learning is a proven strategy for improving UDA performance. In this paper, we propose a novel strong-weak integrated semi-supervision (SWISS) learning strategy for unsupervised domain adaptation. Under the proposed SWISSUDA framework, a strong representative set with high confidence but low diversity target domain samples and a weak representative set with low confidence but high diversity target domain samples are updated constantly during the training process. Both sets are fused randomly to generate an augmented strong-weak training batch with pseudo-labels to train the network during every iteration. Moreover, a novel adversarial logit loss is proposed to reduce the intra-class divergence between source and target domains, which is back-propagated adversarially with a gradient reverse layer between the classifier and the rest of the network. Experimental results based on two popular benchmarks, Office-Home, and DomainNet, show the effectiveness of the proposed SWISS framework with our method achieving the best performance in both Office-Home and DomainNet.
Practical Bulk Denoising Of Large Binary Images

This paper explores the problem of removing real, non-simulated noise from a large body of large binary images. Three denoising methods are evaluated for their efficacy and speed: the well known DUDE, a novel variant of it which we call the Quorum Denoiser, and an adaptation of the Non-Local Means (NLM) method for binary images, B-NLM which, to our knowledge, is faster than other known variants. The methods are compared and tested both on simulated noise (as a benchmark) and on the real life images. All three methods produce good results on real noise. However, despite being optimized, the B-NLM is significantly slower than the other two, whose speeds are comparable to a plain median filter. Overall, the Quorum denoiser appears to be the best option, both in quality (real and simulated) and speed.
Dose-Blind Denoising With Deep Learning in Cardiac Spect

Deep learning denoising methods have been found to be effective for noise suppression in reduced-dose studies in medical imaging applications. In this work, we investigate the feasibility of improving the generalizability of a denoising network by using a dose-blind training approach, in which the network is trained with a loss function defined to accommodate the varying data statistics associated with multiple reduced-dose levels. In the experiments, we demonstrated this approach on quarter- and eighth-dose data from a set of 895 clinical cardiac SPECT perfusion imaging acquisitions. The quantitative results show that a dose-blind denoising network could generalize well over both dose levels, and outperformed dose-specific training in detection of perfusion defects at both quarter- and eighth-dose data (p-values < 10 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">-4</sup> ; paired t-test).
MR. TOMP : Inversion of the Modulo Radon Transform (MRT) via Orthogonal Matching Pursuit (OMP)

In the recent years, practitioners in the area of tomography have proposed high dynamic range (HDR) solutions that are inspired by the multi-exposure fusion strategy in computational photography. To this end, multiple Radon Transform projections are acquired at different exposures that are algorithmically fused to facilitate HDR re-construction. A single-shot alternative to multi-exposure fusion approach has been proposed in our recent line of work which is based on the Modulo Radon Transform (MRT). In this case, Radon Transform projections are folded via modulo non-linearity. This folding allows HDR values to be mapped into the dynamic range of the sensor and, thus, avoids saturation or clipping. The folded measurements are then mapped back to their ambient range using algorithms. The main goal of this paper is to introduce a novel, Fourier domain recovery method, namely, the OMP-FBP method, which is based on the Orthogonal Matching Pursuit (OMP) algorithm and Filtered Back Projection (FBP) formula. The proposed OMP-FBP method offers several advantages; it is agnostic to the modulo threshold or the number of folds, can handle much lower sampling rates than previous approaches and is empirically stable to noise and outliers. Computer simulations as well as hardware experiments in the paper validate the effectivity of the OMP-FBP recovery method.
Scene Representation Learning from Videos Using Self-Supervised and Weakly-Supervised Techniques

Holistic understanding of videos requires the recognition of the overall scene beyond detecting foreground activity and objects. It provides valuable information for various video understanding tasks such as video summarization, scene change detection and content filtering. While significant effort has been put into developing models for scene classification in images (e.g. Places365), video-level scene recognition is relatively nascent. The scope of this paper is to address this problem of going from image representations to video for scene classification. In particular, we compare self-supervised deep learning methods on video scene recognition task using the HVU dataset. Starting from strong image level scene representations, with triplets based contrastive loss, we train a video-level scene classifier. We propose triplet sampling strategies that aid the self-supervision. We compare the self-supervised techniques against the image level scene representations, as well as a weakly supervised classifier trained on image labels. We observe that the models learned using self-supervised method outperform both baselines (with statistical significance), showing that we are able to retain the representative power of the video-level scene representations compared to a competitive image-level scene recognition model trained on Places365, while showing benefits over weakly supervised techniques.
Implicit Shape Biased Few-Shot Learning for 3D Object Generalization

The current state-of-the-art (SOTA) methods validate the role of shape in object categorization, however, except few, most of them neglect object shape information. Motivated by low-shot learning and increasing synthetic data in vision tasks, we investigated how image-based embedding generalization can be improved by the data itself. We propose a new data augmentation approach for low-shot object generalization regime based on image-only. The proposed method learns a discriminative embedding space using SIFT shape points for 3D objects, such that it’s easier to map images and point clouds into one. Numerous experiments show that the proposed approach is superior to the existing low-shot SOTA methods.
Image Restoration Using Probability-Inducing Nuclear Norm Minimization

To reproduce the latent high-quality image from its observed image, traditional image restoration approaches based on low-rank prior usually solve the low-rank matrix approximation problem by minimizing the nuclear norm. However, most of approaches work on the desired singular values individually, and ignore the underlying statistical property of singular values. In this work, we propose a probability-inducing nuclear norm minimization (PINNM) algorithm, where a probability-inducing singular value estimator is presented to estimate the desired singular values. For further filling-in the image details lost in handling the singular values, a simple yet efficient residual cascade scheme is designed to refine the image quality by using the intermediate recovered image. Then, the proposed PINNM algorithm is applied on two classic image restoration tasks: image super-resolution and denoising. Experimental results demonstrate that, the proposed PINNM algorithm outperforms many state-of-the-art approaches both quantitatively and qualitatively for the aforementioned tasks. The source code is available at https://github.com/cvzh/PINNM.
Aggregated Context Network For Semantic Segmentation Of Aerial Images

With the considerable advancement of remote sensing technology and computer vision, automatic scene understanding for very high-resolution aerial (VHR) imagery became a necessary research topic. Semantic segmentation of VHR imagery is an important task where context information plays a crucial role. Adequate feature delineation is difficult due to high-class imbalance in remotely sensed data. In this work, we proposed a variant of encoder-decoder-based architecture where residual attentive skip connections are incorporated. We added a multi-context block in each of the encoder units to capture multi-scale and multi-context features and used dense connections for effective feature extraction. A comprehensive set of experiments reveal that the proposed scheme outperformed recently published work by 3% in overall accuracy and F1 score for ISPRS Vaihingen and ISPRS Potsdam benchmark datasets.