Reviewerly/exHarmony huggingface dataset:
A Temporally Coherent Background Model for DIBR View Synthesis

Aligned color and depth images allow exploring a scene by projecting the virtual camera to arbitrary positions. Depth-image-based rendering (DIBR) methods are often adopted in practical solutions, requiring fewer data than other approaches that explore stereo or multi-view setups. As the virtual camera moves, artifacts are expected to appear, which must be tackled adequately. For video sequences, not only spatial but also temporal inconsistencies affect the user experience. This paper proposes a background model that can be coupled to reduce the flickering of state-of-the-art still-image DIBR methods applied to video sequences captured by static cameras. Qualitative and quantitative results show the potential of the use of our method in practical applications.
Scanpath Prediction Via Semantic Representation of The Scene

Aiming at the problem that the current scanpath prediction methods have insufficient representation of object association, we propose a scanpath prediction model based on semantic representation of the scene. Our model uses a panoramic segmentation network to separate object instances and backgrounds in scenes, and uses the attention mechanism to learn the semantic correlation between objects, which effectively extracts the deep image information related to the current task. We also propose a dual-branch structure predicting the fixation position and duration simultaneously, to fully simulate the temporal and spatial distribution of the human eye’s attention in visual search. Experimental results show that our model has obvious advantages over the existing scanpath prediction methods in search efficiency and scanpath similarity, and can accurately predict the fixation duration.
Structured Dropconnect for Uncertainty Inference in Image Classification

Uncertainty inference has become an important task to prove the reliability for deep neural networks. For image classification tasks, we propose a structured DropConnect (SDC) framework to model the output of a deep neural network into a distribution. We introduce a DropConnect strategy in a structured manner in the fully connected layers, split the network into several sub-networks during testing, and choose the Dirichlet distribution to model the outputs of these sub-networks. The entropy of the parameterized Dirichlet distribution is finally utilized for uncertainty inference. In this paper, this framework is implemented on VGG16, and ResNet18 models for misclassification detection and open-set out-of-domain detection on CIFAR-10 and CIFAR-100 datasets. Experimental results show that the performance of the proposed SDC can be comparable to other uncertainty inference methods.
Meta-Learned Initialization For 3D Human Recovery

We propose a novel framework for 3D human recovery from a single image that integrates meta-learning with test-time optimization. Compared to previous optimization-based or learning-based methods that showed limited generalization ability, the test-time optimization framework enables us to estimate an optimal network parameter, which yields better performance and generalization ability, but it is highly sensitive to an initial network parameter. To alleviate this, we present a meta-learning framework to learn better initial parameters for test-time optimization. Experimental results on standard benchmarks show that the proposed framework boost the test-time optimization performance compared to state-of-the-arts.
Computing Curvature, Mean Curvature and Weighted Mean Curvature

Traditional computing methods for curvatures require the image to be second-order differentiable. Such requirement is not always satisfied, especially at sharp edges. In this paper, we propose a novel method that can compute curvatures without requiring the image second-order differentiable. We first establish the link between various curvatures and the standard Laplace operator. Then we propose to compute curvatures by a half kernel Laplace method. Our method has a smaller support region and thus is more accurate than traditional methods. It can be further adopted to compute curvature, mean curvature, and weighted mean curvature. Our method is compared with the classical schemes on both synthetic and real images, showing its effectiveness and efficiency.
Identifying Document Images with Glare Using Global and Localized Feature Fusion

This paper presents a framework to identify whether a document image is perturbed with glare. Glare identification for document images is particularly challenging because of predominantly white background and dearth of training dataset. We addresses the dataset bottleneck by introducing a glare synthesis framework to generate a large training dataset. The proposed training model consists of a global deep neural network supplemented by extracted localized feature. To our best knowledge, this is one of the first works towards classifying document image for presence of glare. Experiments on real glare dataset showcase benefits of combined global and local features and also outperform recent glare segmentation model adapted for the classification task.
An Algebraic Optimization Approach To Image Registration

High-speed is an essential requirement for many applications of image registration. However, existing methods are usually time-consuming due to the difficulty of the task. In this paper, different from usual feature-based ideas, we convert the matching problem into an algebraic optimization task. By solving a series of quadratic optimization equations, the underlying deformation (rotation, scaling and shift) between image pairs can be retrieved. This process is extremely fast and can be performed in real-time. Experiments show that our method can achieve good performance at a much lower computation cost. When used to initialize our earlier parametric Local All-Pass (LAP) registration algorithm, the results obtained improve significantly over the state of the art.
COFENet: Co-Feature Neural Network Model for Fine-Grained Image Classification

It is challenging to classify patterns with small inter-class variations but large intra-class variations especially for textured objects with relatively small sizes and blurry boundaries. We propose the Co-Feature Network (COFENet), a novel deep learning network for fine-grained texture-based image classification. State-of-the-art (SoTA) methods on this mostly rely on feature concatenation by merging convolutional features into fully connected layers. Some existing work explored the variation between pair-wise features during learning, they only considered the relations in the feature channels, and did not explore the spatial or structural relations among the image regions where the features are extracted from. We propose to leverage such information among the features and their relative spatial layouts to capture richer pairwise, orientationwise, and distancewise relations among feature channels for end-to-end learning of intra-class and inter-class variations.
Iterative Seeded Region Growing for Brain Tissue Segmentation

Brain tissue segmentation from magnetic resonance imaging (MRI) is of significant importance for clinical application and cognitive research. The promising deep learning based methods heavily depend on the quality and quantity of training datasets, and also ignore the domain knowledge. To overcome this issue, this paper proposes a novel Iterative Seeded Region Growing (ISRG) approach for brain tissue segmentation with only one reference image. After super-voxel generation and matching, we first select the high confidence seeded regions based on the high similarity between individual brain images. Then, we obtain initial the voxel-wise tissue probabilities with a proposed fully convolutional network (named TPUNet). Thirdly, the seeded regions are updated according to the voxel-wise tissue probabilities. The second and the third steps are iteratively performed until the segmentation labels of the entire image are obtained. The proposed approach is evaluated on IBSR18 dataset and achieves better results compared with other methods.
CBPT: A New Backbone for Enhancing Information Transmission of Vision Transformers

This paper presents an efficient multi-scale vision transformer, called CBPT, that capably serves as a general-purpose backbone for computer vision. A challenging issue in transformer design is that window self-attention(WSA) often limits the information transmission of each token, whereas enlarging WSA’s receptive field is very expensive to compute. To address this issue, we develop the Locally-Enhanced Window Self-attention mechanism to double the receptive field and have a similar computational complexity to the typical WSA. In addition, we also propose Information-Enhanced Patch Merging, which solves the loss of information in sampling the attention map. Incorporated with these designs and the Cross Block Partial connection, CBPT not only significantly surpasses Swin by +1 box AP and mask AP on COCO object detection and instance segmentation, but also has 30% fewer parameters and 35% fewer FLOPs than Swin.
Adversarial Pairwise Reverse Attention for Camera Performance Imbalance in Person Re-Identification: New Dataset And Metrics

Existing evaluation metrics for Person Re-Identification (Person ReID) models focus on system-wide performance. However, our studies reveal weaknesses due to the uneven data distributions among cameras and different camera properties that expose the ReID system to exploitation. In this work, we raise the long-ignored ReID problem of camera performance imbalance and collect a real-world privacy-aware dataset from 38 cameras to assist the study of the imbalance issue. We propose new metrics to quantify camera performance imbalance and further propose the Adversarial Pairwise Reverse Attention (APRA) Module to guide the model towards learning camera invariant features with a novel pairwise attention inversion mechanism.
Width-Wise Parameter Sharing for Multi-Domain Gan Learning

In this work, we propose a new parameter efficient sharing method for the training of GAN generators. While there has been recent progress in transfer learning for generative models with limited data, they are either limited to domains close to the original one, or adapt a large part of the parameters. This is somewhat redundant, as the goal of transfer learning should be to reuse old features. In this way, we propose width wise parameter sharing, which can learn a new domain with ten times fewer trainable parameters without a significant drop in quality. Previous approaches are less flexible than our method and also fail to preserve image quality for challenging transfers. Finally, as our goal is ultimately parameter reuse, we show that our method performs well in the multi-domain setting, wherein several domains are learned simultaneously with higher visual quality than the state of the art StarGAN-V2.
Viewport-Oriented Panoramic Image Inpainting

Panoramic images are usually viewed through Head Mounted Displays (HMDs), which renders only a narrow field of view from the raw panoramic image. This distinctive viewing feature has largely been ignored when inpainting panoramic images. To address this issue, we propose a viewport-oriented generative adversarial panoramic image inpainting network in this paper. For capturing the distorted features accurately in the generating process of equirectangular projection (ERP) panoramic image, a latitude-adaptive feature fusion module is devised to aggregate the latitude-level features in ERP image and less-distorted patch-level viewport-domain features. Furthermore, a novel cross-domain discriminator is proposed to force the inpainting network to generate more plausible results in viewports. Extensive experiments show that our model achieves better performance compared to the baseline methods, especially in the viewport images.
Autolv: Automatic Lecture Video Generator

We propose an end-to-end lecture video generation system that can generate realistic and complete lecture videos directly from annotated slides, instructor’s reference voice and instructor’s reference portrait video. Our system is primarily composed of a speech synthesis module with few-shot speaker adaptation and an adversarial learning-based talking-head generation module. It is capable of not only reducing instructors’ workload but also changing the language and accent which can help the students follow the lecture more easily and enable a wider dissemination of lecture contents. Our experimental results show that the proposed model outperforms other current approaches in terms of authenticity, naturalness and accuracy. Here is a video demonstration of how our system works, and the outcomes of the evaluation and comparison: https://youtu.be/cY6TYkI0cog.
CyEDA: Cycle-Object Edge Consistency Domain Adaptation

A difficulty of global-level translation is to preserve instance-level details in an image. Although some instance level translation methods can retain the details, most of them require either pre-trained object detection/segmentation network or annotation labels. In this work, we propose a novel method namely CyEDA to perform global level domain adaptation that can preserve image contents without any pre-trained networks integration or annotation labels. Specifically, we introduce blending masks and cycle-object edge consistency loss which exploit the preservation of image objects. We show that our approach can outperform other SOTAs in terms of image quality and FID score in both BDD100K and GTA datasets. The code and pre-trained models are publicly available at https://github.com/bjc1999/CyEDA.
A Person Re-Identification Baseline Based on Attention Block Neural Architecture Search

Person re-identification (re-ID) aims to match images of the same person in different camera views. The general convolutional neural network has the problem of insufficient ability to discriminate specific targets, resulting in limited feature representation when directly applied in person re-ID task. In this paper, a person re-ID baseline model based on attention block neural architecture search is proposed. Since the attention mechanism is helpful to improve the feature expression ability of the network, attention blocks are automatically searched and added to ResNet-50 model, which aims to find the most suitable model structure for person re-ID task. Besides, in order to integrate information between the samples of same identity, an intra-class self-distillation loss is introduced according to the idea of knowledge integration. Experiments on two popular datasets confirm the effectiveness of our baseline. The code has been released in https://github.com/Nicholasxin/Attention-NASReID.
Truncated Lottery Ticket for Deep Pruning

New deep learning models are massively over-parametrized, e.g., GPT-3 and Turing-NLG exceed 100Bs of parameters. Naturally, model reduction techniques such as pruning, quantization, and distillation have been proposed and deployed. In the pruning literature, the Lottery Ticket Hypothesis (LTH) is amongst the most cited. LTH provides a recipe for reducing the free parameters of a deep network by eliminating a large fraction of its edges by 1) remembering and reusing only the edges on the highest-traffic input-output paths and 2) resetting remaining parameters to their initial values after each round of elimination. Here, we extend this idea two steps further. We add a random buildup phase, starting from a small percentage of parameters to reach a predefined accuracy, often needing fewer than 50% of the parameters of a full network. Then we eliminate parameters using an LTH-like procedure combined with a binary search. We demonstrate the efficiency of the proposed technique using three well-known image data sets by sparsifying three standard deep nets. We exceed the elimination power of LTH (over 98% pruned) and effectively match the performance of full networks. We conclude that for both reducing computation and maintaining a sparse network for inference, the proposed scheme is effective and practical.
Real-World Video Anomaly Detection by Extracting Salient Features

We propose a lightweight and highly accurate method for detecting anomalies in videos. Existing methods use multiple-instance learning (MIL) to determine the normal/abnormal status of each segment of the video. Recent successful researches argue that it is important to learn the temporal relationships among segments to achieve high accuracy, instead of focusing on only a single segment. We analyzed the existing methods that have been successful in recent years, and found that while it is indeed important to learn all segments together, the temporal relationships among them are irrelevant to achieving high accuracy. Based on this finding, we do not use the MIL framework, but instead introduce a self-attention mechanism to automatically extract features that are important for determining normal/abnormal from all input segments. As a result, the neural network with 1.3% of the number of parameters of the existing method can achieve the comparable or better accuracy than the existing method.
Entropy-Reduced Attention for Image Compression

Image compression is a challenging work. However, few works consider using the property of image entropy to compress images. In this paper, we observe that there is a certain law between the entropy of feature maps in the field of image compression based on deep learning. That is, the subtraction of different level feature maps can obtain smaller information entropy. Smaller information entropy can bring lower uncertainty, thereby removing redundancy in the compression process, which can achieve better compression efficiency. Based on the observation above, we propose a framework based on information entropy reduction attention (IERA) for image compression to obtain better compression efficiency. Meanwhile, we design a IERA module to reduce the information entropy of feature maps during encoding and decoding. Extensive experiments demonstrate that our model outperforms many existing classical and learned image compression frameworks.
How Accurate Is Passive Stereo For 3d Face Reconstruction?

The highest quality 3D face reconstructions are produced using multi-view stereo methods, reporting errors below 0.5mm. Unfortunately, these methods typically employ dozens of high-resolution cameras in a large laboratory capture gantry. In contrast, monocular 3D face reconstruction using sophisticated deep learning models are suited for casual mobile phone imaging outside the lab and report a mean error of 1-2mm.This paper investigates whether classic stereo methods can be used in scenarios with only a few low-resolution images available. We expect to find that it cannot since multi-view stereo performs well only when many high-resolution images are provided. When only two low-resolution images are available, stereo produces very noisy results which are not directly usable. Surprisingly, however, our analysis shows that this visually noisy data has lower error than comparison state-of-the-art methods. We find that the visual artifacts from stereo can be removed using a morphable face model to constrain face shape.
A Lightweight Network with Multi-Stage Feature Fusion Module for Single-View 3d Face Reconstruction

3D face reconstruction has attracted great attentions of researchers from both academic and industry for its potential application in many scenarios such as face alignment and recognition across large poses. 3D Morphable Model which reconstructs a 3D face through basis coefficients prediction, is usually adopted as the typical parametric framework for 3D face and is suitable to combine with deep learning. Existing cascade regression method predicts coefficients by multiple iterations, which is time-consuming. In this paper, we propose an efficient and end-to-end method for single-view 3D face reconstruction. We build a lightweight network based on mobile blocks with faster speed for parameter extraction and smaller model size. Especially, a multi-stage feature fusion module is designed for enhancing the end-to-end learning. To match the setting of input image size, we updated the pose label of images under various sizes in training dataset before training. Extensive experiments on challenging datasets validate the efficiency of our method for both 3D face reconstruction and face alignment.
Dispense Mode for Inference to Accelerate Branchynet

With the increase of depth and width, Deep Neural Network has got the best results in the computer vision, but its massive calculation has brought a heavy burden to IOT devices. To speed up the inference of DNN models, Branchynet creatively puts forward the early exit, which means that samples exit from shallow layers to reduce the calculation amount of the model. But Branchynet has some unnecessary intermediate calculations in the inference process. We propose a dispense mode to solve this problem, which can optimize the accuracy and latency of BranchyNet at the same time. The dispense mode directly determines the exit position of the sample in the multi-branch network according to the difficulty of the sample without intermediate trial errors. Under the same accuracy requirements, the inference speed is improved by 30%-50% compared with the cascade mode of Branchynet. Moreover, while further reducing redundant calculation, it provides a method for dynamic adjustment of accuracy. Thus, our framework can easily adjust the accuracy of the model to meet higher throughputs.
Stable Clustering Ensemble Based on Evidence Theory

As an unsupervised ensemble learning strategy, clustering ensemble combines multiple base clusterings into a high-quality one and has achieved successful applications in image analysis and data mining. However, extant clustering ensemble methods are ineffective to handle the data uncertainty in clustering consensus process, which may mislead to poor clustering ensemble results. To tackle the problem, we propose a stable clustering ensemble (SCE) method based on evidence theory (Dempster–Shafer theory) in this paper. Specifically, we construct a belief function of cluster membership to measure the uncertainty and stability of data instances in clustering ensemble and thereby implement the stable clustering ensemble algorithm. We test the proposed stable clustering ensemble method in the tasks of structural data clustering and image segmentation. The experimental results validate the proposed method is effective to process the uncertain data and produce high-quality data clusterings.
History Dependent Significance Coding for Incremental Neural Network Compression

This paper presents an improved probability estimation scheme for the entropy coder of Incremental Neural Network Coding (INNC), which is currently under standardization in ISO/IEC MPEG. More specifically, the paper first analyzes the compression performance of INNC and how the bitstream size relates to the neural network (NN) layers. For the layers requiring the most bits, it analyzes the coded NN weight updates and their temporal dependencies. Major finding is that the probability of a significant (i.e., non-zero) update for a weight can depend considerably on whether the weight has been updated before. Based on this finding, the paper proposes a new probability estimation scheme: Depending on whether a significant update has been received before (i.e., based on the weight’s history), the entropy coder models the probability for a current significant update differently. This scheme achieves a bitstream size reduction of about 2% and 1% in a transfer and a federated learning scenario, respectively, without any accuracy loss or significant complexity increase. Therefore, MPEG adopted our history dependent significance probability (HDSP) scheme to its emerging standard for INNC.