Reviewerly/exHarmony huggingface dataset:
Light Field Image Quality Assessment with Dense Atrous Convolutions

Unlike regular images that represent only light intensities, Light Field (LF) contents carry information about the intensity of light in a scene, including the direction in which light rays are traveling in space. This allows for a richer representation of our world, but requires large amounts of data that need to be processed and compressed before being transmitted to the viewer. Since these techniques may introduce distortions, the design of Light Field Image Quality Assessment (LF-IQA) methods is important. Currently, most LF-IQA methods use traditional 2D image quality assessment techniques or rely on low-level spatial features. In this paper, we propose a novel no-reference LF-IQA method that takes into account both LF angular and spatial information. The proposed method is made up of two processing streams with identical blocks of Convolutional Neural Network (CNN), Atrous Convolution layers (ACL), and a regression block for quality prediction. The results show that the method is robust and outperforms current state-of-the-art methods.
Illumination-Aware Style Transfer for Image Harmonization

Image harmonization aims to make composite images visually consistent by adjusting the appearance of the foreground to make it harmonious with the background. Previous methods mainly focus on transferring the global style of composite images. However, these methods don’t perform well when the foreground and background lighting conditions are quite different. In this paper, we propose an illumination-aware style transfer method, of which the goal is to enhance the realism of composite images with inconsistent illumination between the foreground and background. Firstly, we use the style encoder network that has been specially designed to produce style features. And the local light estimates are calculated from the patches of the image. Then the local light estimates are re-fined by the contribution-weighted aggregation to obtain the global estimates. Finally, the global light estimates are adaptively fused with the style features according to the distinction between foreground and background light estimates, and the fused features are decoded to generate the harmonized image. We also construct a dataset for evaluating image lighting harmonization. Extensive experiments on the iHarmony4 dataset and our dataset demonstrate that our method achieves state-of-the-art performance.
Deep Learning Classification of Large-Scale Point Clouds: A Case Study on Cuneiform Tablets

This paper introduces a novel network architecture for the classification of large-scale point clouds. The network is used to classify metadata from cuneiform tablets. As more than half a million tablets remain unprocessed, this can help create an overview of the tablets. The network is tested on a comparison dataset and obtains state-of-the-art performance. We also introduce new metadata classification tasks on which the network shows promising results. Finally, we introduce the novel Maximum Attention visualization, demonstrating that the trained network focuses on the intended features. Code available at https://github.com/fhagelskjaer/dlc-cuneiform
PPT: Anomaly Detection Dataset of Printed Products with Templates

Visual anomaly detection has been an active topic in industrial applications. In particular, it aims to classify anomalies and precisely locate defective areas in the printed products. To the best of our knowledge, there is no anomaly detection dataset for industrial printings. In this paper, we are the first to introduce a Printed Products with Templates (PPT) dataset, which contains large templates and sliced images collected from industry scene images. PPT is a challenging dataset with more variable surface defects and more disturbing background than existing related benchmarks. Furthermore, we propose a template matching method for anomaly detection of printed products, which consists of a fast template matching block with a convolutional operation using the test sliced image as its kernel, and a prediction network for generating an anomaly map of the test sliced image. Experimental results show that our method achieves state-of-the-art performance compared to the related anomaly detection approaches.
Guided Sampling Based Feature Aggregation for Video Object Detection

Video object detection is a challenging task due to the presence of appearance deterioration in video frames. Recently, feature aggregation based methods which aggregate context information from object proposals in different frames to improve the performance, have dominated the task. However, much invalid information may be introduced during feature aggregation since frames and proposals are usually selected at random. In this paper, we propose a guided sampling based feature aggregation network (GSFA) to perform more effective feature aggregation. Specifically, we introduce a frame-level sampling module and a proposal-level sampling module to sample informative frames and proposals from a video sequence adaptively. As a result, the proposed GSFA can effectively aggregate context information from the semantically rich frames and proposals to boost the performance. Experimental results on the ImageNet VID dataset show the proposed GSFA achieves the state-of-the-art performance of 84.8% mAP with ResNet-101 and 85.8% mAP with ResNeXt-101.
A Joint Secret Image Sharing and Jpeg Compression Scheme

Along with the development of cloud computing, ensuring fast and secure image exchanges became necessary. In this context, the scientific community has taken a growing interest in JPEG image security. Crypto-compression is generally used to make secure the content of a secret image, while allowing to reduce its size. The drawback of crypto-compression is that it depends on a secret key. Indeed, if this key is lost, so is the content of the secret image. In this paper, we propose a joint secret image sharing and JPEG compression method. Shamir’s secret sharing scheme is used during the JPEG Huffman coding step, ensuring the visual security of the secret image in the compressed domain, while solving the problem of secret key loss. Our approach is fully format compliant and size preserving when compared to a standard JPEG compression of the original secret image.
Prior Semantic Harmonization Network for Few-Shot Semantic Segmentation

Few-shot semantic segmentation(FSS) is intended to segment a foreground object from a query image with a novel object using only a few annotated support images. Although attracting the attention of many researchers, this challenging problem remains to be not well solved due to two critical issues: (1)The information mismatching between support and query features leads to model distraction. (2)The key feature of query images is not activated well. In this paper, we introduce the Prior Semantic Harmonization Network(PSHNet) to tackle these limitations. PSHNet is composed of three effective modules. The Semantic Harmonization Module(SHM) corrects the information matching between support and query images, while the Feature Activation Module(FAM) activates the key feature of query images. Furthermore, we introduce a Hierarchical Aggregation Module(HAM) to refine each output of the multi-scale module. Experiments show that our model achieves an excellent performance on both PASCAL-5 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sup> and COCO-20 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sup> datasets.
Mask Guided Spatial-Temporal Fusion Network for Multiple Object Tracking

Multi-object trackers make the association almost perfectly when no occlusion occurred between two or more targets. However, it is hard to extract reliable features on account of partial occlusion caused by a nearby object, which often leads to tracking failure. In this paper, we utilize mask to guide attention of the neural network in order to focus on the visible part of the target and design a tracklet-level feature extraction method. Then, a tracking framework is proposed based on a mask guided fusion network and multi-hypothesis tracking algorithm. Comprehensive evaluation on the MOT17 dataset shows that our approach achieves competitive results.
Encoder Enabled Gan-based Video Generators

This research study proposes a compatible encoder-enabled video generating method. The encoder-enabled method adds an inference mechanism for enhancing the ability of Generative Adversarial Networks (GAN) based video generators. The proposed video generating method is called Encoding GAN3 (EncGAN3) and decomposes the video into two streams representing content and movement, respectively. The proposed model consists of three processing modules, representing Encoder, Generator and Discriminator, each trained separately, by considering its own loss function. Enc-GAN3 is shown to generate videos of high quality, according to both visual and numerical results.
Predicting Human Perception of Scene Complexity

It is apparent that humans are intrinsically capable of determining the degree of complexity present in an image; but it is unclear which regions in that image lead humans towards evaluating an image as complex or simple. Here, we develop a novel deep learning model for predicting human perception of the complexity of natural scene images in order to address these problems. For a given image, our approach, ComplexityNet, can generate both single-score complexity ratings and two-dimensional per-pixel complexity maps. These complexity maps indicate the regions of scenes that humans find to be complex, or simple. Drawing on work in the cognitive sciences we integrate metrics for scene clutter and scene symmetry, and conclude that the proposed metrics do indeed boost neural network performance when predicting complexity.
Shuffle Attention Multiple Instances Learning for Breast Cancer Whole Slide Image Classification

Multiple instance learning (MIL) has recently become a powerful tool to solve the weakly supervised classification problem on whole slide image (WSI) pathological diagnosis. However, current MIL methods lie in two drawbacks: 1) seldom depicting feature dependencies in multiple dimensions, 2) limitation on capturing dependencies of selected instances for predicting bag-level results. To address these issues, this work presents a novel two-stage shuffle attention MIL (SAMIL) model for breast cancer WSI classification. SAMIL first introduces shuffle attention to extract important features from both spatial and channel dimensions, which well includes pixel-level pairwise relationships and channel dependencies, thus helping select more discriminant breast cancer instances for bag-level prediction. Additionally, it stacks multi-head attention with long short-term memory (LSTM) to construct an aggregator, and this adaptively highlights the most distinctive instance features while exploring the correlation between selected breast cancer instances more effectively. Experiment results on the Camelyon-16 dataset demonstrate its superior performance compared with the state-of-the-art MIL methods. The code is available at https://github.com/CunqiaoHou/SAMIL.
Uncertainty Aware Multitask Pyramid Vision Transformer for UAV-Based Object Re-Identification

Object Re-IDentification (ReID), one of the most significant problems in biometrics and surveillance systems, has been extensively studied by image processing and computer vision communities in the past decades. Learning a robust and discriminative feature representation is a crucial challenge for object ReID. The problem is even more challenging in ReID based on Unmanned Aerial Vehicle (UAV) as the images are characterized by continuously varying camera parameters (e.g., view angle, altitude, etc.) of a flying drone. To address this challenge, multiscale feature representation has been considered to characterize images captured from UAV flying at different altitudes. In this work, we propose a multitask learning approach, which employs a new multiscale architecture without convolution, Pyramid Vision Transformer (PVT), as the backbone for UAV-based object ReID. By uncertainty modeling of intraclass variations, our proposed model can be jointly optimized using both uncertainty-aware object ID and camera ID information. Experimental results are reported on PRAI and VRAI, two ReID data sets from aerial surveillance, to verify the effectiveness of our proposed approach.
Explore Spatial and Channel Attention in Image Quality Assessment

As a subjective concept, perceived image quality is heavily affected by visual mechanisms, e.g., selective attention and contrast sensitivity. This work proposes a lightweight attention module in image quality assessment (IQA) to simulate spatial attention and contrast sensitivity mechanisms. The attention module can extract essential information from a CNN backbone for image quality perception using two sequential attention blocks: a spatial block for mimicking selective attention in spatial domain and a channel block for contrast sensitivity. Experimental results on two large-scale publicly available IQA datasets have demonstrated promising performance of the proposed approach. The source code can be found at https://github.com/junyongyou/sca_iqa.
Banding vs. Quality: perceptual impact and objective assessment

Staircase-like contours introduced to a video by quantization in flat areas, commonly known as banding, have been a longstanding problem in both video processing and quality assessment communities. The fact that even a relatively small change of the original pixel values can result in a strong impact on perceived quality makes banding especially difficult to be detected by objective quality metrics. In this paper, we study how banding annoyance compares to more commonly studied scaling and compression artifacts with respect to the overall perceptual quality. We further propose a simple combination of VMAF and the recently developed banding index, CAMBI, into a banding-aware video quality metric showing improved correlation with overall perceived quality.
An Efficient End-To-End Image Compression Transformer

Image and video compression received significant research attention and expanded their applications. Existing entropy estimation-based methods combine with hyperprior and local context, limiting their efficacy. This paper introduces an efficient end-to-end transformer-based image compression model, which generates a global receptive field to tackle the long-range correlation issues. A hyper encoder-decoder-based transformer block employs a multi-head spatial reduction self-attention (MHSRSA) layer to minimize the computational cost of the self-attention layer and enable rapid learning of multi-scale and high-resolution features. A Casual Global Anticipation Module (CGAM) is designed to construct highly informative adjacent contexts utilizing channel-wise linkages and identify global reference points in the latent space for end-to-end rate-distortion optimization (RDO). Experimental results demonstrate the effectiveness and competitive performance of the KODAK dataset.
D-CBRS: Accounting for Intra-Class Diversity in Continual Learning

Continual learning – accumulating knowledge from a sequence of learning experiences – is an important yet challenging problem. In this paradigm, the model’s performance for previously encountered instances may substantially drop as additional data are seen. When dealing with class-imbalanced data, forgetting is further exacerbated. Prior work has proposed replay-based approaches which aim at reducing forgetting by intelligently storing instances for future replay. Although Class-Balancing Reservoir Sampling (CBRS) has been successful in dealing with imbalanced data, the intraclass diversity has not been accounted for, implicitly assuming that each instance of a class is equally informative. We present Diverse-CBRS (D-CBRS), an algorithm that allows us to consider within class diversity when storing instances in the memory. Our results show that D-CBRS outperforms state-of-the-art memory management continual learning algorithms on data sets with considerable intra-class diversity.
LB-NERF: Light Bending Neural Radiance Fields for Transparent Medium

Neural radiance fields (NeRFs) have been proposed as methods of novel view synthesis and have been used to address various problems because of its versatility. NeRF can represent colors and densities in 3D space using neural rendering assuming a straight light path. However, a medium with a different refractive index in the scene, such as a transparent medium, causes light refraction and breaks the assumption of the straight path of light. Therefore, the NeRFs cannot be learned consistently across multi-view images. To solve this problem, this study proposes a method to learn consistent radiance fields across multiple viewpoints by introducing the light refraction effect as an offset from the straight line originating from the camera center. The experimental results quantitatively and qualitatively verified that our method can interpolate viewpoints better than the conventional NeRF method when considering the refraction of transparent objects.
AV-GAZE: A Study on the Effectiveness of Audio Guided Visual Attention Estimation for Non-profilic Faces

In challenging real-life conditions such as extreme head-pose, occlusions, and low-resolution images where the visual information fails to estimate visual attention/gaze direction, audio signals could provide important and complementary information. In this paper, we explore if audio-guided coarse head-pose can further enhance visual attention estimation performance for non-prolific faces. Since it is difficult to annotate audio signals for estimating the head-pose of the speaker, we use off-the-shelf state-of-the-art models to facilitate cross-modal weak-supervision. During the training phase, the framework learns complementary information from synchronized audio-visual modality. Our model can utilize any of the available modalities i.e. audio, visual or audio-visual for task-specific inference. It is interesting to note that, when AV-Gaze is tested on benchmark datasets with these specific modalities, it achieves competitive results on multiple datasets, while being highly adaptive toward challenging scenarios.
Boundary-Area Enhanced Module for Instance Segmentation

Segmenting instances is a challenging task, especially for precise boundary segmentation. Modern methods always predict instance masks with imprecise boundaries due to the fact they do not pay enough attention to boundary information. To address this problem, we propose a conceptually simple yet effective attention-based module for better boundary segmentation, termed boundary-area enhanced module. In this model, we extend the thin boundary lines to boundary areas by broadening one line using its inner pixels within a distance. Boundary areas can be captured more explicitly and accurately, and then low-level features and high-level features are fused to predict instance masks better. The proposed boundary-area enhanced model yields significant improvements over the Mask R-CNN framework baseline on COCO and Cityscapes datasets. Extensive experiments show that our boundary-area enhanced module outperforms previous state-of-the-art models and can help other two-stage segmentation models to achieve better performance.
3D Human Motion Generation from the Text Via Gesture Action Classification and the Autoregressive Model

In this paper, a deep learning-based model for 3D human motion generation from the text is proposed via gesture action classification and an autoregressive model. The model focuses on generating special gestures that express human thinking, such as waving and nodding. To achieve the goal, the proposed method predicts expression from the sentences using a text classification model based on a pretrained language model and generates gestures using the gate recurrent unit-based autoregressive model. Especially, we proposed the loss for the embedding space for restoring raw motions and generating intermediate motions well. Moreover, the novel data augmentation method and stop token are proposed to generate variable length motions. To evaluate the text classification model and 3D human motion generation model, a gesture action classification dataset and action-based gesture dataset are collected. With several experiments, the proposed method successfully generates perceptually natural and realistic 3D human motion from the text. Moreover, we verified the effectiveness of the proposed method using a public-available action recognition dataset to evaluate cross-dataset generalization performance.
On Adversarial Robustness of Deep Image Deblurring

Recent approaches employ deep learning-based solutions for the recovery of a sharp image from its blurry observation. This paper introduces adversarial attacks against deep learning-based image deblurring methods and evaluates the robustness of these neural networks to untargeted and targeted attacks. We demonstrate that imperceptible distortion can significantly degrade the performance of state-of-the-art deblurring networks, even producing drastically different content in the output, indicating the strong need to include adversarially robust training not only in classification but also for image recovery.
Bi-Polar Mask for Joint Cell and Nuclei Instance Segmentation

Cell and nucleus instance segmentation is crucial for computer-assisted interpretation of Cervical Cytology. However, due to the overlapping of cells and miscellaneous objects, it is not trivial to segment them jointly and bind the pair of instances accurately. In this work, we propose Bi-Polar Mask that utilizes two polars to represent the contours of cell and nucleus. The polar representation in nature overcomes the challenges of object overlapping and cell folding. We also design a shared center at the nucleus, which allows for straightforward binding of the pair of contours. A large dataset with thousands of cell and nucleus annotations was constructed for evaluation. Experimental results demonstrate the superior performance of the proposed method.
Document Shadow Removal with Foreground Detection Learning From Fully Synthetic Images

Shadow removal for document images is a major task for digitized document applications. Recent shadow removal models have been trained on pairs of shadow images and shadow-free images. However, obtaining a large-scale and diverse dataset is laborious and remains a great challenge. Thus, only small real datasets are available. To create relatively large datasets, a graphic renderer has been used to synthesize shadows, nonetheless, it is still necessary to capture real documents. Thus, the number of unique documents is limited, which negatively affects a network’s performance. In this paper, we present a large-scale and diverse dataset called fully synthetic document shadow removal dataset (FSDSRD) that does not require capturing documents. The experiments showed that the networks (pre-)trained on FSDSRD provided better results than networks trained only on real datasets. Additionally, because foreground maps are available in our dataset, we leveraged them during training for multitask learning, which provided noticeable improvements. The code is available at: https://github.com/IsHYuhi/DSRFGD.