Reviewerly/exHarmony huggingface dataset:
Coupling Attention and Convolution for Heuristic Network in Visual Dialog

Visual Dialog is a typical AI-agent task on images, in which the agent interprets information from heterogeneous modalities and provides the correct answer. In this area, most approaches are based on the attention mechanism. When the agent enjoys the large-capacity advantage of attention, the lack of the right inductive bias compared with convolution hinders its success. Therefore, in order to utilize their advantages and compensate for their respective shortcomings, inspired by the paraventricular thalamus (PVT) in the brain, we couple convolution and attention, termed as Attention Convolution Enhanced (ACE) method to enhance the agent’s activation of key features and strengthening the semantic understanding of visual and textual data. Meanwhile, we propose Heuristic Adjustment (HA) module to globally strengthen the agent’s semantic understanding and reduce language bias that is easy to occur after using the enhanced features. Finally, we concatenate the ACE and the HA in our Coupling Attention and Convolution for Heuristic Network (CACH-Net) to train the agent for better semantic comprehension and generalization ability. Extensive experiments on the VisDial v1.0 benchmark show that our CACH-Net has a better performance.
PASTS: Toward Effective Distilling Transformer for Panoramic Semantic Segmentation

Recently, panoramic imaging system has been attracting a lot of attention in various real-world applications due to its all-around sensing abilities. Despite the success of semantic segmentation, the performance of panoramic segmentation is still poor because the number of annotated panoramic datasets is insufficient and existing methods cannot handle the structural distortions in panoramic images caused by wide FoV. In this paper, we present a novel PAnoramic Segmentation Transformers (PASTs) trained by a knowledge distillation strategy with teacher-student branches. We first train the teacher using labeled pinhole images. The knowledge learned from the teacher is transferred to the student via feature distillation. To this end, we exploit the distorted pinhole images to force the attention and the prediction from the teacher consistent with those from the student. In addition, we adopt the entropy loss to train the student with unlabeled panoramic images. Experimental results demonstrate the effectiveness of our method, both qualitatively and quantitatively.
SYGNet: A SVD-YOLO based GhostNet for Real-time Driving Scene Parsing

In this paper, we propose SYGNet to strengthen the scene parsing ability of autonomous driving under complicated road conditions. The SYGNet includes feature extraction component and SVD-YOLO GhostNet component. The SVD-YOLO GhostNet component combines Singular Value Decomposition (SVD), You Only Look Once (YOLO) and GhostNet. In the feature extraction component, we propose an algorithm based on VoxelNet to extract point cloud features and image features. In SVD-YOLO GhostNet component, the image data is decomposed by SVD, and we obtain data with stronger spatial and environmental characteristics. YOLOv3 is used to obtain the future map, then convert to GhostNet, which is used to realize the real-time scene parsing. We use KITTI data set to perform our experiments and the results show that the SYGNet is more robust and can further enhance the accuracy of real-time driving scene parsing. The model code, data set, and results of the experiments in this paper are available at: https://github.com/WangHewei16/SYGNet-for-Real-time-Driving-Scene-Parsing.
Predicting Radiologist Attention During Mammogram Reading with Deep and Shallow High-Resolution Encoding

Radiologists’ eye-movement during diagnostic image reading reflects their personal training and experience, which means that their diagnostic decisions are related to their perceptual processes. For training, monitoring, and performance evaluation of radiologists, it would be beneficial to be able to automatically predict the spatial distribution of the radiologist’s visual attention on the diagnostic images. The measurement of visual saliency is a well-studied area that allows for prediction of a person’s gaze attention. However, compared with the extensively studied natural image visual saliency (in free viewing tasks), the saliency for diagnostic images is less studied; there could be fundamental differences in eye-movement behaviours between these two domains. Most current saliency prediction models have been optimally developed for natural images, which could lead them to be less adept at predicting the visual attention of radiologists during the diagnosis. In this paper, we propose a method specifically for automatically capturing the visual attention of radiologists during mammogram reading. By adopting high-resolution image representations from both deep and shallow encoders, the proposed method avoids potential detail losses and achieves superior results on multiple evaluation metrics in a large mammogram eye-movement dataset.
ProX: A Reversed Once-for-All Network Training Paradigm for Efficient Edge Models Training in Medical Imaging

The usage of edge models in medical field has a huge impact on promoting the accessibility of real-time medical services in the under-developed regions. However, the handling of latency-accuracy trade-off to produce such an edge model is very challenging. Although the recent Once-For-All (OFA) network is able to directly produce a set of sub-network designs with Progressive Shrinking (PS) algorithm, it still suffers from training resource and time inefficiency downfall. In this paper, we propose a new OFA training algorithm, namely the Progressive Expansion (ProX). Empirically, we showed that the proposed paradigm can reduce training time up to 68%; while still able to produce sub-networks that have either similar or better accuracy compared to those trained with OFA-PS in ROCT (classification), BRATS and Hippocampus (3D-segmentation) public medical datasets.
Single Image Reflection Removal Based on Bi-Channels Prior

Single image reflection removal is a crucial technique which can improve the performance of object detection, semantic segmentation, and various computer vision applications. In this paper, we present a novel reflection removal algorithm using bi-channel priors (i.e., the dark channel prior and the bright channel prior). We observe that the values of dark channel pixels are not near 0, and those of bright channel pixels are not closer to 1 under the reflection scenario. We first demonstrate these phenomena statistically and mathematically. Then, we apply these properties as the constraints in optimizing the proposed reflection removal process. Extensive experiments on several well-known benchmarks demonstrate that our approach achieves desirable reflection suppression results compared with other methods.
Revisiting Natural Scene Statistical Modeling Using Deep Features for Opinion-Unaware Image Quality Assessment

Opinion-unaware no-reference (OU-NR) methods for image quality assessment (IQA) are of great interest since they can predict visual quality independent of a reference image and knowledge of human quality opinions. Models of image naturalness trained on a corpus of pristine images have shown potential for developing OU-NR methods. However, the extracted features may not match the preferences of the human visual system (HVS). This paper aims to utilize the features of convolutional neural networks to achieve a richer representation of the naturalness space. In addition, the IQA processing steps from training to quality measurement are revisited and the naturalness model is improved by incorporating HVSinspired criteria. Experimental results show the higher performance and generalizability of the naturalness model – constructed using HVS-aligned deep features – under different distortion types and image contents. The source code of the quality index is available at https://gitlab.com/saeedmp/dni.
Differential Contrast Based Adaptive Quantization for Perceptual Quality Optimization in Image Coding

We consider the perceptual quality optimization in image coding through adaptive quantization. A differential contrast model is proposed to measure the visual sensitivity to the quantization distortions, and thereby deriving the spatially adaptive quantization strategy. A complementary quantitative approach is provided as a means to efficiently calculate the proposed differential contrast model. The resulting visual quality improvement is experimentally demonstrated.
Deep Neural Network-Based Noisy Pixel Estimation for Breast Ultrasound Segmentation

The success of modern deep learning algorithms for image segmentation heavily relies on the availability of high-quality labels for training. However, obtaining accurate labels is time-consuming and tedious, and requires expertise. If directly trained with dataset with noisy annotations, networks can easily overfit to noisy labels and result in poor performance, which might lead to serious misinterpretation. To this end, we propose a noisy pixel estimation approach based on deep neural network, which helps correct the noisy annotations resulting in better prediction performance. First, a deep neural network is trained to detect noisy pixels from image annotations. Then, the estimated noisy pixels are used to correct the noisy annotations. Finally, the corrected annotations are used to train the deep learning model. Our proposed framework is validated on the breast tumor segmentation task. The obtained experimental results show that our proposed method can improve the robustness of deep learning model under noisy annotations while achieving favorable performance against existing noisy label correction methods.
Investigating Inconsistencies in PRNU-Based Camera Identification

PRNU (Photo-response non-uniformity) is widely considered a unique and reliable fingerprint for identifying the source of an image. The PRNU patterns of two different sensors, even if belonging to the same camera model, are strongly uncorrelated. Therefore, such a fingerprint is used as evidence by various law enforcement agencies for source identification, manipulation detection, etc. However, in recent smartphones, images are subjected to significant in-camera processing associated with computational photography. This heavy processing introduces non-unique artifacts (NUA) in such images and masks the uniqueness of the PRNU fingerprint. In this work, we investigate the robustness of PRNU in modern smartphones. We propose a model that explains the unexpected behavior of PRNU in such smartphones. Finally, we present two methods to identify images suffering from NUA. Our methods achieve high accuracy in identifying such images.
Video-Analytics Task-Aware Quad-Tree Partitioning and Quantization for HEVC

Video analytics systems designed for computer vision tasks use deep learning models that rely on high-quality input data to maximize performance. However, in a real-world system, these inputs are often compressed using video codecs such as HEVC. Video compression degrades the quality of the inputs, thereby degrading the performance of these models. Region-of-interest (ROI) coding enables bits to be allocated to improve performance; however, the method to select regions should be computationally simple since it must occur during or before the video is compressed and transmitted for further processing. In this paper, we propose a task-aware quad-tree (TA-QT) partitioning and quantization method to achieve ROI coding for HEVC and other video coding standards. TA-QT uses a lightweight edge-based model to guide task-aware video encoding to improve end-stage video analytics (ESVA) performance while reducing both bit-rate and encoding time. We demonstrate the effectiveness of our approach in terms of (a) the performance of the ESVA on compressed inputs, (b) transmission bit-rates, and (c) encoding time.
GraphEx: Facial Action Unit Graph for Micro-Expression Classification

Facial micro-expressions are crucial cues for expressing human emotions. Existing works have shown substantial progress in detecting micro-expressions for various applications in the computer vision field. However, it is still onerous for existing methods to handle and interpret micro-expressions efficiently. This paper proposes a deep learning-based approach leveraging spatio-temporal and graph representation learning for micro-expression classification. We design a novel Spatial-Temporal Info Extraction Network (STIENet) for learning facial appearance and muscle motion from high dimensional video clip frames and summarizes them into more meaningful feature maps. We construct an action unit (AU) relation graph to further represent the AU co-occurrence in the same micro-expression video clip. A graph neural network (GNN) is used to learn AU-related graph embedding for the downstream classification task. Performance evaluation on two mainstream micro-expression datasets, i.e., CASME II and SAMM, show that the proposed framework outperforms other state-of-the-art methods for micro-expression classification.
Self-Supervised Learning of Optical Flow, Depth, Camera Pose and Rigidity Segmentation with Occlusion Handling

In this work, we propose a self-supervised scene flow framework for joint learning of optical flow, stereo depth, camera pose, and rigidity map and handle the occlusion during training. Specifically, we propose a feature masking method to alleviate the occlusion impact on the correlation result and reduce the outliers in both optical flow and depth map. We use the improved optical flow and depth to estimate the camera motion directly using the Perspective-n-Point method, which improves it accordingly. Furthermore, we recursively update the optical flow in both occluded and non-occluded regions with self-supervised cues learned from the rigid and optical flows. Reducing the error in the occluded regions enhances the rigidity map and improves the final optical flow accordingly. Our model achieved the state-of-the-art performance on KITTI 2015 benchmark for optical flow and produced competitive results for the depth, pose, and segmentation tasks.
Distilling Facial Knowledge with Teacher-Tasks: Semantic-Segmentation-Features For Pose-Invariant Face-Recognition

This paper demonstrates a novel approach to improve face-recognition pose-invariance using semantic-segmentation features. The proposed Seg-Distilled-ID network jointly learns identification and semantic-segmentation tasks, where the segmentation task is then "distilled" (MobileNet en-coder). Performance is benchmarked against three state-of-the-art encoders on a publicly available data-set emphasizing head-pose variations. Experimental evaluations show the Seg-Distilled-ID network shows notable robustness benefits, achieving 99.9% test-accuracy in comparison to 81.6% on ResNet-101, 96.1% on VGG-19 and 96.3% on InceptionV3. This is achieved using approximately one-tenth of the top encoder’s inference parameters. These results demonstrate distilling semantic-segmentation features can efficiently ad-dress face-recognition pose-invariance.
End-To-End Depth Map Compression Framework Via Rgb-To-Depth Structure Priors Learning

In this paper, we propose a novel framework to exploit and utilize the shared information inner RGB-D data for efficient depth map compression. Two main codecs, designed based on the existing end-to-end image compression network, are adopted for RGB image compression and enhanced depth image compression with RGB-to-Depth structure prior, respectively. In particular, we propose a Structure Prior Fusion (SPF) module to extract the structure information from both RGB and depth codecs at multi-scale feature levels and fuse the cross-modal feature to generate more efficient structure priors for depth compression. Extensive experiments show that the proposed framework can achieve competitive rate-distortion performance as well as RGB-D task-specific performance at depth map compression compared with the direct compression scheme.
Learned Image Compression with Multi-Scale Spatial and Contextual Information Fusion

Although learned image compression based on convolution neural network and hyperprior makes significant progress, the distinction between original and reconstructed images is still obvious. In order to reconstruct compressed image with higher quality, a novel model based on fusing multi-scale spatial and context information is proposed in this work. Since spatial information might be dropped during the forward propagation when neural networks go deeper, a multi-scale information fusion module is designed to help the encoder to retain the necessary spatial information while removing the redundancy in latent representation. Meanwhile, a multi-scale 3D context module with varying-sized masked 3D convolution kernels is devised to obtain multi-scale correlation in latent representation. The experiments demonstrate the superiority of the proposed approach over a number of state-of-the-art image compression methods and the versatile video coding.
Positive Unlabeled Learning by Semi-Supervised Learning

Positive and Unlabeled learning (PU learning) trains a binary classifier based on only positive (P) and unlabeled (U) data, where the unlabeled data contains positive or negative samples. Previous importance reweighting approaches treat all unlabeled samples as weighted negative samples, achieving state-of-the-art performance. However, in this paper, we surprisingly find that the classifier could misclassify negative samples in U data as positive ones at the late training stage by weight adjustment. Motivated by this discovery, we leverage Semi-Supervised Learning (SSL) to address this performance degradation problem. To this end, we propose a novel SSL-based framework to tackle PU learning. Firstly, we introduce the dynamic increasing sampling strategy to progressively select both negative and positive samples from U data. Secondly, we adopt MixMatch to take full advantage of the unchosen samples in U data. Finally, we propose the Co-learning strategy that iteratively trains two independent networks with the selected samples to avoid the confirmation bias. Experimental results on four benchmark datasets demonstrate the effectiveness and superiority of our approach when compared with other state-of-the-art methods.
OMNET: Real-Time Stereo Matching with Unsupervised Occlusion Mask

Although CNN has powerful learning capability, it is still difficult for CNN to judge corresponding points in the occlusion region. The ghosting effect in the occlusion region during feature warping is the bottleneck of performance improvement for many stereo matching networks. In this paper, we propose an Occlusion-Aware Refinement Module (OARM), which can learn a rough occlusion map from multi-scale aggregated cost volumes without occlusion supervision to mask and filter pernicious occluded regions in a warped image. Cooperating with well-designed simple yet efficient 2D-based Intra/Cross-Level Aggregation Modules to effectively and efficiently aggregate information of different scales before the disparity refinement stage, OARM helps our proposed OM-Net achieve an error rate of 1.82% (D1-all) on KITTI 2015 dataset, which is even better than most 3D-based networks. Meanwhile, OMNet keeps real-time characteristic and could process a 1248×384 resolution image pair at 26 fps.
IID-NORD: A Comprehensive Intrinsic Image Decomposition Dataset

The goal of intrinsic image decomposition is to recover low level features of images. Most of the studies tend to consider only reflectance and shading, even though it is known that increasing the number of intrinsics is beneficial for many applications. Existing intrinsic image datasets are quite limited. In this study, a dataset is introduced to provide a comprehensive benchmark to the field of intrinsic image decomposition. IID-NORD contains a large number of scenes and for each scene ground truth reflectance, shading, surface normal vectors, light vectors and depth map is provided to allow detailed decomposition. Moreover, diverse illuminants, viewing angles, and dynamic shadows are used to prevent any bias. To the best of available knowledge, IID-NORD is the most comprehensive dataset in the field of intrinsic image decomposition. IID-NORD will be available on the first author’s official webpage.
Continual Learning in Vision Transformer

Continual learning aims to continuously learn new tasks from new data while retaining the knowledge of tasks learned in the past. Recently, the Vision Transformer, which utilizes the Transformer initially proposed in natural language processing for computer vision, has shown higher accuracy than Convolutional Neural Networks (CNN) in image recognition tasks. However, there are few methods that have achieved continual learning with Vision Transformer. In this paper, we compare and improve continual learning methods that can be applied to both CNN and Vision Transformers. In our experiments, we compare several continual learning methods and their combinations to show the differences in accuracy and the number of parameters.
CNN-Based Local Tone Mapping in the Perceptual Quantization Domain

A typical way of local tone mapping (TM) is based on multi-layer decomposition of the source image. For this, the source image is decomposed into a base layer and a detail layer to compress from high dynamic range (HDR) to low dynamic range (LDR). Perceptual quantization (PQ) is a standardized non-linear transfer function for HDR content. It mimics the non-linearity of human vision by compressing more strongly in bright regions and less in dark areas, when converting luminance values to electrical signals. We propose a CNN-based pipeline for local TM, which operates on the low-frequency base layer of the luminance signal, while keeping the detail layer unchanged. The proposed method works entirely in the PQ domain and is adaptable to display peak luminance. The tone-mapped LDR images obtained with our learning-based approach show significant improvements in PSNR, while the network size is reduced compared to previous work. Our experiments on the HDR datasets from Fairchild and Funt show PSNR improvements of 8 dB compared to the state-of-the-art approaches.
Improving Model Adaptation for Semantic Segmentation by Learning Model-Invariant Features with Multiple Source-Domain Models

In this paper, we focus on a problem remaining to be studied: multi-source model adaptation, which is derived from multi-source unsupervised domain adaptation and replaces the source-domain data with source-domain pre-trained models. Pre-trained models are always easier to share than training data so that multiple source-domain models are available in many practical scenarios. Therefore, the problem setting of multi-source domain adaptation is practical in real-world applications. In this setting, we challenge the task of semantic segmentation which is difficult also in the traditional unsupervised domain adaptation due to the pixel-level knowledge transfer. Our method takes full advantage of the multiple source-domain models by learning model-invariant features, which aims to obtain target-domain features with similar distributions from the models pre-trained in different source domains. The adaptation models trained with the model-invariant feature learning benefit from the diversity of the source-domain models and can thus produce more generalizable features to the target domain. Experimental results in several adaptation settings validate the effectiveness and superiority of our method.