Reviewerly/exHarmony huggingface dataset:
GCN-Based Multi-Modal Multi-Label Attribute Classification in Anime Illustration Using Domain-Specific Semantic Features

This paper presents a multi-modal multi-label attribute classification model in anime illustration based on Graph Convolutional Networks (GCN) using domain-specific semantic features. In animation production, since creators often intentionally highlight the subtle characteristics of the characters and objects when creating anime illustrations, we focus on the task of multi-label attribute classification. To capture the relationship between attributes, we construct a multi-modal GCN model that can adopt semantic features specific to anime illustration. To generate the domain-specific semantic features that represent the semantic contents of anime illustrations, we construct a new captioning framework for anime illustration by combining real images and their style transformation. The contributions of the proposed method are two-folds. 1) More comprehensive relationships between attributes are captured by introducing GCN with semantic features into the multi-label attribute classification task of anime illustrations. 2) More accurate image captioning of anime illustrations can be generated by a trainable model by using only real-world images. To our best knowledge, this is the first work dealing with multi-label attribute classification in anime illustration. The experimental results show the effectiveness of the proposed method by comparing it with some existing methods including the state-of-the-art methods.
Clustering-Based Psychometric No-Reference Quality Model for Point Cloud Video

Point cloud video streaming is a fundamental application of immersive multimedia. In it, objects represented as sets of points are streamed and displayed to remote users. Given the high bandwidth requirements of this content, small changes in the network and/or encoding can affect the users' perceived quality in unexpected manners. To tackle the degradation of the service as fast as possible, real-time Quality of Experience (QoE) assessment is needed. As subjective evaluations are not feasible in real time due to their inherent costs and duration, low-complexity objective quality assessment is a must. Traditional No-Reference (NR) objective metrics at client side are best suited to fulfill the task. However, they lack on accuracy to human perception. In this paper, we present a cluster-based objective NR QoE assessment model for point cloud video. By means of Machine Learning (ML)-based clustering and prediction techniques combined with NR pixel-based features (e.g., blur and noise), the model shows high correlations (up to a 0.977 Pearson Linear Correlation Coefficient (PLCC)) and low Root Mean Squared Error (RMSE) (down to 0.077 on a zero-to-one scale) towards objective benchmarks after evaluation on an adaptive streaming point cloud dataset consisting of sixteen source videos and 453 sequences in total.
Lightweight Dual-Domain Network for Real-Time Medical Image Segmentation

With the development of deep learning, deep convolution neural networks for medical image segmentation tasks have become more and more complex in pursuit of higher accuracy. In most scenarios, medical image segmentation pursues accuracy rather than speed, However, real-time performance is crucial in some scenarios, such as surgical navigation and diagnosis of acute stroke. So design of high-precision, lightweight and real-time medical image segmentation network has become an urgent need. To this end, a novel lightweight dual-domain network (LDD-Net) has been proposed in this paper. LDD-Net is comprised of two branches, learning respectively from the frequency domain and the spatial domain. In the frequency domain branch, the image spatial resolution is compressed via discrete cosine transform to have a large receptive field, so that better semantic context features can be learned. In the spatial domain branch, high-resolution feature representations with more details are learned. Finally, the learned features of these two branches are fused to yield high accuracy with low computational cost. The proposed method has been validated on two medical image segmentation datasets to yield the state-of-the-art performances with greatly reduced inference time and parameters of the learned models.
Unsupervised Anomaly Detection with Self-Training and Knowledge Distillation

Anomaly Detection (AD) aims to find defective patterns or abnormal samples among data, and has been a hot research topic due to various real-world applications. While various AD methods have been proposed, most of them assume the availability of a clean (anomaly-free) training set, which however may be hard to guarantee in many real-world industry applications. This motivates us to investigate Unsupervised Anomaly Detection (UAD) in which the training set includes both normal and abnormal samples. In this paper, we address the UAD problem by proposing a Self-Training and Knowledge Distillation (STKD) model. STKD combats anomalies in the training set by iteratively alternating between excluding samples of high anomaly probabilities and training the model with the purified training set. Despite that the model is trained with a cleaner training set, the inevitably existing anomalies may still cause negative impact. STKD alleviates this by regularizing the model to respond similarly to a teacher model which has not been trained with noisy data. Experiments show that STKD consistently produces more robust performance with different levels of anomalies.
Blind Video Quality Assessment via Space-Time Slice Statistics

User-generated contents (UGC) have gained increased attention in the video quality community recently. Perceptual video quality assessment (VQA) of UGC videos is of great significance for content providers to monitor, process, and deliver massive numbers of UGC videos. Blind video quality prediction of UGC videos is challenging since complex mixtures of spatial and temporal distortions contribute to the overall perceptual quality. In this paper, we develop a simple, effective, and efficient blind VQA framework (STS-QA) based on the statistical analysis of space-time slices (STS) of videos. Specifically, we extract spatio-temporal statistical features along different orientations of video STS, that capture directional global motion, then train a shallow quality predictor. The proposed framework can be used to easily extend any existing video/image quality model to account for temporal or motion regularities. Our experimental results on three publicly available UGC databases demonstrate that our proposed STS-QA model can significantly boost prediction performance compared to baselines. The code will be released at: https://github.com/uniqzheng/STS_BVQA.
Flexible-Rate Learned Hierarchical Bi-Directional Video Compression with Motion Refinement and Frame-Level Bit Allocation

This paper presents improvements and novel additions to our recent work on end-to-end optimized hierarchical bidirectional video compression [1] to further advance the state-of-the-art in learned video compression. As an improvement, we combine motion estimation and prediction modules and compress refined residual motion vectors for improved rate-distortion performance. As novel addition, we adapted the gain unit proposed for image compression to flexible-rate video compression in two ways: first, the gain unit enables a single encoder model to operate at multiple rate-distortion operating points; second, we exploit the gain unit to control bit allocation among intra-coded vs. bi-directionally coded frames by fine tuning corresponding models for truly flexible-rate learned video coding. Experimental results demonstrate that we obtain state-of-the-art rate-distortion performance exceeding those of all prior art in learned video coding.
Exploring Active Learning for Semiconductor Defect Segmentation

The development of X-Ray microscopy (XRM) technology has enabled non-destructive inspection of semiconductor structures for defect identification. Deep learning is widely used as the state-of-the-art approach to perform visual analysis tasks. However, deep learning based models require large amount of annotated data to train. This can be time-consuming and expensive to obtain especially for dense prediction tasks like semantic segmentation. In this work, we explore active learning (AL) as a potential solution to alleviate the annotation burden. We identify two unique challenges when applying AL on semiconductor XRM scans: large domain shift and severe class-imbalance. To address these challenges, we propose to perform contrastive pretraining on the unlabelled data to obtain the initialization weights for each AL cycle, and a rareness-aware acquisition function that favors the selection of samples containing rare classes. We evaluate our method on a semiconductor dataset that is compiled from XRM scans of high bandwidth memory structures composed of logic and memory dies, and demonstrate that our method achieves state-of-the-art performance.
Boosting the Performance of Weakly-Supervised 3d Human Pose Estimators With Pose Prior Regularizers

This work aims to boost the performance of 3D human pose estimators trained in a weakly-supervised setting where there are much fewer annotated 3D poses than unlabeled video data. We formulate two self-supervised pose prior regularizers (PPR) - bone proportion and joint mobility constraints that are pose translation, scale, and rotation invariant. These regularizers, combined with bone symmetry loss, reduce overfitting to the 2D reprojection loss commonly used in weakly-supervised settings by optimizing the bone lengths and joint rotations of estimated 3D poses. Consequently, improving the accuracy of 3D pose estimators. The regularizers are network independent and can be applied to any network architecture without modifications. We apply our proposed PPR to VideoPose3D network [1] and show that it decreases the MPJPE by 24% when using ≤5% of annotated H36M [2] 3D data, improving state-of-the-art accuracy by 7.9 mm.
Mask-Vit: an Object Mask Embedding in Vision Transformer for Fine-Grained Visual Classification

Fine-grained visual classification (FGVC) targets to accurately identify the subordinate categories from a target class. Convolutional neural network (CNN) based methods prove that the attention mechanism can enhance the representation of local regions and improve the recognition accuracy. Recently, vision transformer (ViT) has shown great application potential in image classification tasks by taking advantage of its inherent self-attention mechanism and early global information acquisition capability. However, this global information acquisition approach involves an irrelevant environment in the interaction process, which makes it difficult for fine-grained tasks that rely on local differences to quickly learn discriminant features. To this end, we propose a hybrid network termed Mask-ViT, which can effectively avoid environmental interference and express more robust features by focusing on the instance itself. Specifically, Contour Knowledge Embedding (CKE) is employed to transferred prior location information to ViT and guided the subsequent recognition. The experiments on three benchmarks demonstrate the effectiveness of the proposed method.
Temporal Flow Mask Attention for Open-Set Long-Tailed Recognition of Wild Animals in Camera-Trap Images

Camera traps, unmanned observation devices, and deep learning-based image recognition systems have greatly reduced human effort in collecting and analyzing wildlife images. However, data collected via above apparatus exhibits 1) long-tailed and 2) open-ended distribution problems. To tackle the open-set long-tailed recognition problem, we propose the Temporal Flow Mask Attention Network that comprises three key building blocks: 1) an optical flow module, 2) an attention residual module, and 3) a meta-embedding classifier. We extract temporal features of sequential frames using the optical flow module and learn informative representation using attention residual blocks. Moreover, we show that applying the meta-embedding technique boosts the performance of the method in open-set long-tailed recognition. We apply this method on a Korean De-militarized Zone (DMZ) dataset. We conduct extensive experiments, and quantitative and qualitative analyses to prove that our method effectively tackles the open-set long-tailed recognition problem while being robust to unknown classes.
A Robust Ensemble Model For Parasitic Egg Detection And Classification

Intestinal parasitic infections, as a leading causes of morbidity worldwide, still lacks time-saving, high-sensitivity and user-friendly examination method. The development of deep learning technique reveals its broad application potential in biological image. In this paper, we apply several object detectors such as YOLOv5 and variant cascadeRCNNs to automatically discriminate parasitic eggs in microscope images. Through specially-designed optimization including raw data augmentation, model ensemble, transfer learning and test time augmentation, our model achieves excellent performance on challenge dataset. In addition, our model trained with added noise gains a high robustness against polluted input, which further broaden its applicability in practice.
Semantic Unfolding of Stylegan Latent Space

Generative adversarial networks (GANs) have proven to be surprisingly efficient for image editing by inverting and manipulating the latent code corresponding to an input real image. This editing property emerges from the disentangled nature of the latent space. In this paper, we identify that the facial attribute disentanglement is not optimal, thus facial editing relying on linear attribute separation is flawed. We thus propose to improve semantic disentanglement with supervision. Our method consists in learning a proxy latent representation using normalizing flows, and we show that this leads to a more efficient space for face image editing.
A Framework for Contrast Enhancement Algorithms Optimization

We present a general-purpose framework for the optimization of parametric contrast enhancement algorithms. We first define a regression module for image acceptability, which is based on deep neural features and which is trained on a large dataset of user-expressed preferences. This regression module is then used as the objective function of a Bayesian optimization process, guiding the search for the optimal parameters of a given contrast enhancement algorithm. In our experiments we optimize three different contrast enhancement algorithms of varying levels of complexity. The effectiveness of our optimization framework is experimentally confirmed by evaluating the output of the optimized contrast enhancement algorithms with respect to reference enhanced images.
CCL: Class-Wise Curriculum Learning for Class Imbalance Problems

Computer vision datasets usually present long-tailed training distributions where the classes are not represented with the same number of training samples. This so-called class imbalance problem hinders the proper learning of inference models, biasing them towards over-represented classes and decreasing their generalization. Adopted solutions to tackle the effect of class imbalance are based on weighting the training loss according to the number of class samples, leading to regimes where low-represented classes guide the learning just accounting for their cardinal number. To also incorporate class complexity in the process, we propose a novel training scheme called CCL: Class-wise Curriculum Learning. Classes are first sorted based on a difficulty criterion which not only accounts for the number of training samples but also for their training outcomes. The curriculum is then used to guide the training: easy classes are fed first and—incrementally, the more difficult ones are added. The proposed approach is validated for image classification using long-tailed datasets. Results show that when the proposed Class-wise Curriculum Learning scheme is used, trained models outperform specific state-of-the-art methods devoted to handle the class imbalance problem. The code, data and reported models described along this paper are publicly available at https://github.com/vpulab/CCL
Eldnet: Establishment and Refinement of Edge Likelihood Distributions for Camouflaged Object Detection

Camouflaged object detection (COD) focuses on detecting objects assimilated into surroundings. Distractions from the background make it extremely challenging to find the correct semantics and border details. Some of the current methods aim to explore edge details directly which has the risk of leading to boundary errors under strong background interference. While other conservative methods find rough semantic locations, then implicitly infer the edge against the background which cannot fully pay attention to the spatial details manifesting as partial semantic loss and blurring. Different from the existence, we propose a framework that generates progressively refined edge likelihood maps to guide the feature fusion of camouflaged objects. To get robust and refined edge likelihood maps, a module is first trained to fit the reasonable edge likelihood distribution based on semantic position clues and then a progressive architecture is designed to refine the possible edge area which gradually approaches the real edge. Our method can avoid catastrophic errors during exploring the fine edge, but also improves border details, solving the border blurring. Experiments on four datasets demonstrate the significant improvements and generalization ability of ours compared with state-of-the-art methods.
Grad-Cam Aware Supervised Attention for Visual Question Answering for Post-Disaster Damage Assessment

In this paper, we present a Grad-Cam aware supervised attention framework for visual question answering (VQA) tasks for post-disaster damage assessment purposes. Visual-attention in visual question-answering tasks aims to focus on relevant image regions according to questions to predict answers. However, the conventional attention mechanisms in VQA work in an unsupervised manner, learning to give importance to visual contents by minimizing only task-specific loss. This approach fails to provide appropriate visual attention where the visual contents are very complex. The content and nature of UAV images in FlooNet-VQA dataset are very complex as they depict the hazardous scenario after Hurricane Harvey from a high altitude. To tackle this, we propose a supervised attention mechanism that uses explainable features from Grad-Cam to supervise visual attention in the VQA pipeline. The mechanism we propose operates in two stages. In the first stage of learning, we derived the visual explanations through Grad-Cam by training a baseline attention-based VQA model. In the second stage, we supervise our visual content for each question by incorporating the Grad-Cam explanations from the previous phase of the training process. We have improved the model performance over the state-of-the-art VQA models by a considerable margin on FloodNet dataset.
Iterative Kernel Reconstruction for Deep Learning-Based Blind Image Super-Resolution

Deep learning based methods have received a great deal of interest in recent years to solve the single image super-resolution (SISR) problem and their performance is proven to be superior when compared to classical SR techniques. Yet, most of these methods fail to generalize well on real life image datasets because they are trained on synthetic datasets with a small range of blur kernels. This makes data-driven approaches inherently weak when it comes to real images. Therefore, applying image super-resolution independently of the blur kernel is still a challenging task. In this paper we propose IKR-Net, Iterative Kernel Reconstruction network, for blind SISR. In the proposed approach, kernel estimation and high resolution image reconstruction are carried out iteratively using deep models. The iterative refinement provides significant improvement in both the reconstructed image and the estimated blur kernel. IKR-Net achieves state-of-the-art results in blind SISR, especially for images with motion blur.
Camera Self-Calibration: Deep Learning from Driving Scenes

Prior to driving, cameras embedded in an autonomous driving system need to be calibrated intrinsically. Calibration is crucial to ensure that safety-related perception functions can reliably perceive the environment. Vehicle cameras are also exposed to mechanical perturbations requiring periodic re-calibration with regular uses. The current widely-accepted calibration approaches are based on robust but potentially demanding target-based methods. Such methods require a car to be taken offline and rely on static infrastructure and operators. Targetless online calibration approaches exist but remain largely unadopted due to the accuracy gaps compared to the classical methods. We propose a deep-learning-based self-calibration strategy for the vehicular camera that learns from driving scenes—they make an inherently large-scale dataset—and is validated back-to-back against checkerboard reprojection error. Our approach results in a 2.5% decrease in subpixel reprojection error compared to the existing deep-learning-based approaches. We also demonstrate its practical application in the automotive domain.
Authentication Of Copy Detection Patterns Under Machine Learning Attacks: A Supervised Approach

Copy detection patterns (CDP) are an attractive technology that allows manufacturers to defend their products against counterfeiting. The main assumption behind the protection mechanism of CDP is that these codes printed with the smallest symbol size (1x1) on an industrial printer cannot be copied or cloned with sufficient accuracy due to data processing inequality. However, previous works have shown that Machine Learning (ML) based attacks can produce high-quality fakes, resulting in decreased accuracy of authentication based on traditional feature-based authentication systems. While Deep Learning (DL) can be used as a part of the authentication system, to the best of our knowledge, none of the previous works has studied the performance of a DL-based authentication system against ML-based attacks on CDP with 1x1 symbol size. In this work, we study such a performance assuming a supervised learning (SL) setting.
Back to Old Constraints to Jointly Supervise Learning Depth, Camera Motion and Optical Flow in a Monocular Video

In structure from motion or similarly in monocular SLAM problems, spatio-temporal image variations, motion and scene geometry are intimately related and in the absence of such a constraint, unsupervised deep learning methods often tend to state the problem under multiple constraints. We readdress the problem of 3D interpretation estimation as an unsupervised deep learning process where depth and camera motion are learned to satisfy the 3D brightness constraint for rigid objects. We introduce for the first time a new learning paradigm where the spatio-temporal variations of image sequences are coupled to 3D interpretation to minimize the loss without need to add more ad-hoc constraints that are not related to the 3D interpretation. Experimental results show that our method competes and sometimes outperforms the state-of-the-art methods.