GenAI4ELab/papercli-papers huggingface dataset:
High-quality Single-model Deep Video Compression with Frame-Conv3D and Multi-frame Differential Modulation
Authors: Wenyu Sun; Chen Tang; Weigui Li; Zhuqing Yuan; Huazhong Yang; Yongpan Liu
Venue: ECCV 2020 (main)

Abstract: Deep learning (DL) methods have revolutionized the paradigm of computer vision tasks and DL-based video compression is becoming a hot topic. This paper proposes a deep video compression method to simultaneously encode multiple frames with Frame-Conv3D and differential modulation. We first adopt Frame-Conv3D instead of traditional Channel-Conv3D for efficient multi-frame fusion. When generating the binary representation, the multi-frame differential modulation is utilized to alleviate the effect of quantization noise. By analyzing the forward and backward computing flow of the modulator, we identify that this technique can make full use of past frames' information to remove the redundancy between multiple frames, and thus achieves better performance. A dropout scheme combined with the differential modulator is proposed to enable bit rate optimization within a single model. Experimental results show that the proposed approach outperforms the H.264 and H.265 codecs in the region of low bit rate. Compared with recent DL-based methods, our model also achieves competitive performance. "
Instance-Aware Embedding for Point Cloud Instance Segmentation
Authors: Tong He; Yifan Liu; Chunhua Shen; Xinlong Wang; Changming Sun
Venue: ECCV 2020 (main)

Abstract: Although recent works have made significant progress in encoding meaningful context information for instance segmentation in 2D images, the works for 3D point cloud counterpart lag far behind. Conventional methods use radius search or other similar methods for aggregating local information. However, these methods are unaware of the instance context and fail to realize the boundary and geometric information of an instance, which are critical to separate adjacent objects. In this work, we study the influence of instance-aware knowledge by proposing an Instance-Aware Module (IAM). The proposed IAM learns discriminative instance embedding features in two-fold: (1) Instance contextual regions, covering the spatial extension of an instance, are implicitly learned and propagated in the decoding process. (2) Instance-dependent geometric knowledge is included in the embedding space, which is informative and critical to discriminate adjacent instances. Moreover, the proposed IAM is free from complex and time-consuming operations, showing superiority in both accuracy and efficiency over the previous methods. To validate the effectiveness of our proposed method, comprehensive experiments have been conducted on three popular benchmarks for instance segmentation: ScannetV2, S3DIS, and PartNet. The flexibility of our method allows it to handle both indoor scenes and CAD objects. We achieve state-of-the-art performance, outperforming previous methods substantially."
Self-Paced Deep Regression Forests with Consideration on Underrepresented Examples
Authors: Lili Pan; Shijie Ai; Yazhou Ren; Zenglin Xu
Venue: ECCV 2020 (main)

Abstract: Deep discriminative models (e.g.deep regression forests, deep neural decision forests) have achieved remarkable success recently to solve problems such as facial age estimation and head pose estimation. Most existing methods pursue robust and unbiased solutions either through learning discriminative features, or reweighting samples. We argue what is more desirable is learning gradually to discriminate like our human beings, and hence we resort to self-paced learning (SPL). Then, a natural question arises: can self-paced regime lead deep discriminative models to achieve more robust and less biased solutions? To this end, this paper proposes a new deep discriminative model—self-paceddeep regression forests with consideration on underrepresented examples (SPUDRFs). It tackles the fundamental ranking and selecting problem in SPL from a new perspective: fairness. This paradigm is fundamental and could be easily combined with a variety of deep discriminative models (DDMs). Extensive experiments on two computer vision tasks, i.e., facial age estimation and head pose estimation, demonstrate the efficacy of SPUDRFs, where state-of-the-art performances are achieved."
Manifold Projection for Adversarial Defense on Face Recognition
Authors: Jianli Zhou; Chao Liang; Jun Chen
Venue: ECCV 2020 (main)

Abstract: Although deep convolutional neural network based face recognition system has achieved remarkable success, it is susceptible to adversarial images: carefully constructed imperceptible perturbations can easily mislead deep neural networks. A recent study has shown that in addition to regular off-manifold adversarial images, there are also adversarial images on the manifold. In this paper, we propose adversarial variational autoencoder (A-VAE), a novel framework to tackle both types of attacks. We hypothesize that both off-manifold and on-manifold attacks move the image away from the high probability region of image manifold. We utilize variational autoencoder (VAE) to estimate the lower bound of the log-likelihood of image and explore to project the input images back into the high probability regions of image manifold again. At inference time, our model synthesizes multiple similar realizations of a given image by random sampling, then the nearest neighbor of the given image is selected as the final input of the face recognition model. We also use adversarial training to enhance the robustness of our model against adversarial perturbations. As a preprocessing operation, our method is attack-agnostic and can adapt to a wide range of resolutions. The experimental results on LFW demonstrate that our method achieves state-of-the-art defense success rate against conventional off-manifold attacks such as FGSM, PGD, and C\&W under both grey-box and white-box settings, and even on-manifold attack."
Weakly Supervised Learning with Side Information for Noisy Labeled Images
Authors: Lele Cheng; Xiangzeng Zhou; Liming Zhao; Dangwei Li; Hong Shang; Yun Zheng; Pan Pan; Yinghui Xu
Venue: ECCV 2020 (main)

Abstract: In many real-world datasets, like WebVision, the performance of DNN based classier is often limited by the noisy labeled data. To tackle this problem, some image related side information, such as captions and tags, often reveal underlying relationships across images. In this paper, we present an efficient weakly-supervised learning by using a Side Information Network (SINet), which aims to effectively carry out a large scale classication with severely noisy labels. The proposed SINet consists of a visual prototype module and a noise weighting module. The visual prototype module is designed to generate a compact representation for each category by introducing the side information. The noise weighting module aims to estimate the correctness of each noisy image and produce a condence score for image ranking during the training procedure. The propsed SINet can largely alleviate the negative impact of noisy image labels, and is benecial to train a high performance CNN based classier. Besides, we release a ne-grained product dataset called AliProducts, which contains more than 2.5 million noisy web images crawled from the internet by using queries generated from 50,000 fine-grained semantic classes. Extensive experiments on several popular benchmarks (i.e. Webvision, ImageNet and Clothing-1M) and our proposed AliProducts achieve state-of-the-art performance. The SINet has won the rst place in the 5000 category classication task on WebVision Challenge 2019, and outperforms other competitors by a large margin."
Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision
Authors: Peng Wu; Jing Liu; Yujia Shi; Yujia Sun; Fangtao Shao; Zhaoyang Wu; Zhiwei Yang
Venue: ECCV 2020 (main)

Abstract: but also Listen: Learning Multimodal Violence Detection under Weak Supervision","Violence detection has been studied in computer vision for years. However, previous work are either superficial, e.g., classification of short-clips, and the single scenario, or undersupplied, e.g., the single modality, and hand-crafted features based multimodality. To address this problem, in this work we first release a large-scale and multi-scene dataset named XD-Violence with a total duration of 217 hours, containing 4754 untrimmed videos with audio signals and weak labels. Then we propose a neural network containing three parallel branches to capture different relations among video snippets and integrate features, where holistic branch captures long-range dependencies using similarity prior, localized branch captures local positional relation using proximity prior, and score branch dynamically captures the closeness of predicted score. Besides, our method also includes an approximator to meet the needs of online detection. Our method outperforms other state-of-the-art methods on our released dataset and other existing benchmark. Moreover, extensive experimental results also show the positive effect of multimodal (audio-visual) input and modeling relationships. The code and dataset will be released in https://roc-ng.github.io/XD-Violence/."
SNE-RoadSeg: Incorporating Surface Normal Information into Semantic Segmentation for Accurate Freespace Detection
Authors: Rui Fan; Hengli Wang; Peide Cai; Ming Liu
Venue: ECCV 2020 (main)

Abstract: Freespace detection is an essential component of visual perception for self-driving cars. The recent efforts made in data-fusion convolutional neural networks (CNNs) have significantly improved semantic driving scene segmentation. Freespace can be hypothesized as a ground plane, on which the points have similar surface normals. Hence, in this paper, we first introduce a novel module, named surface normal estimator (SNE), which can infer surface normal information from dense depth/disparity images with high accuracy and efficiency. Furthermore, we propose a data-fusion CNN architecture, referred to as RoadSeg, which can extract and fuse features from both RGB images and the inferred surface normal information for accurate freespace detection. For research purposes, we publish a large-scale synthetic freespace detection dataset, named Ready-to-Drive (R2D) road dataset, collected under different illumination and weather conditions. The experimental results demonstrate that our proposed SNE module can benefit all the state-of-the-art CNNs for freespace detection, and our SNE-RoadSeg achieves the best overall performance among different datasets."
Modeling the Space of Point Landmark Constrained Diffeomorphisms
Authors: Chengfeng Wen; Yang Guo; Xianfeng Gu
Venue: ECCV 2020 (main)

Abstract: Surface registration plays a fundamental role in shape analysis and geometric processing. Generally, there are three criteria in evaluating a surface mapping result: diffeomorphism, small distortion and feature alignment. In order to fulfill these requirements, this work proposes a novel model of the space of point landmark constrained diffeomorphisms. Based on Teichm\""uller theory, this mapping space is generated by the Beltrami coefficients, which are infinitesimally Teichm\""uller equivalent to $0$. These Beltrami coefficients are the solutions to a linear equation group. By using this theoretic model, optimal registrations can be achieved by iterative optimization with linear constraints in the diffeomorphism space, such as harmonic maps and Teichm\""uller maps, which minimize different type of distortions. The theoretic model is rigorous and has practical value. Our experiment results demonstrate the efficiency and efficacy of the proposed method."
PieNet: Personalized Image Enhancement Network
Authors: Han-Ul Kim; Young Jun Koh; Chang-Su Kim
Venue: ECCV 2020 (main)

Abstract: Image enhancement is an inherently subjective process since people have diverse preferences for image aesthetics. However, most enhancement techniques pay less attention to the personalization issue despite its importance. In this paper, we propose the first deep learning approach to personalized image enhancement, which can enhance new images for a new user, by asking him or her to select about 10$\sim$20 preferred images from a random set of images. First, we represent various users' preferences for enhancement as feature vectors in an embedding space, called preference vectors. We construct the embedding space based on metric learning. Then, we develop the personalized image enhancement network (PieNet) to enhance images adaptively using each user's preference vector. Experimental results demonstrate that the proposed algorithm is capable of achieving personalization successfully, as well as outperforming conventional general image enhancement algorithms significantly. The source codes and trained models are available at https://github.com/hukim1124/PieNet."
Rotational Outlier Identification in Pose Graphs Using Dual Decomposition
Authors: Arman Karimian; Ziqi Yang; Roberto Tron
Venue: ECCV 2020 (main)

Abstract: In the last few years, there has been an increasing trend to consider Structure from Motion (SfM, in computer vision) and Simultaneous Localization and Mapping (SLAM, in robotics) problems from the point of view of pose averaging (also known as global SfM, in computer vision) or Pose Graph Optimization (PGO, in robotics), where the motion of the camera is reconstructed by considering only relative rigid body transformations instead of including also 3-D points (as done in a full Bundle Adjustment). At a high level, the advantage of this approach is that modern solvers can effectively avoid most of the problems of local minima, and that it is easier to reason about outlier poses (caused by feature mismatches and repetitive structures in the images). In this paper, we contribute to the state of the art of the latter, by proposing a method to detect incorrect orientation measurements prior to pose graph optimization by checking the geometric consistency of rotation measurements. The novel aspects of our method are the use of Expectation-Maximization to fine-tune the covariance of the noise in inlier measurements, and a new approximate graph inference procedure, of independent interest, that is specifically designed to take advantage of evidence on cycles with better performance than standard approaches (Belief Propagation). The paper includes simulation and experimental results that evaluate the performance of our outlier detection and cycle-based inference algorithms on synthetic and real-world data."
Speech-driven Facial Animation using Cascaded GANs for Learning of Motion and Texture
Authors: Dipanjan Das; Sandika Biswas; Sanjana Sinha; Brojeshwar Bhowmick
Venue: ECCV 2020 (main)

Abstract: Speech-driven facial animation methods should produce accurate and realistic lip motions with natural expressions and realistic texture portraying target-specific facial characteristics. Moreover, the methods should also be adaptable to any unknown faces and speech quickly during inference. Current state-of-the-art methods fail to generate realistic animation from any speech on unknown faces due to their poor gen-eralization over different facial characteristics, languages, and accents. Some of these failures can be attributed to the end-to-end learning of the complex relationship between the multiple modalities of speech and the video. In this paper, we propose a novel strategy where we partition the problem and learn the motion and texture separately. Firstly, we train a GAN network to learn the lip motion in a canonical landmark using DeepSpeech features and induce eye-blinks before transferring the motion to the person-specific face. Next, we use another GAN based texture generator network to generate high fidelity face corresponding to the motion on person-specific landmark. We use meta-learning to make the texture generator GAN more flexible to adapt to the unknown subject’s traits of the face during inference. Our method gives significantly improved facial animation than the state-of-the-art methods and generalizes well across the different datasets, different languages, and accents,and also works reliably well in presence of noises in the speech."
Solving Phase Retrieval with a Learned Reference
Authors: Rakib Hyder; Zikui Cai; M. Salman Asif
Venue: ECCV 2020 (main)

Abstract: Fourier phase retrieval is a classical problem that deals with the recovery of an image from the amplitude measurements of its Fourier coefficients. Conventional methods solve this problem via iterative (alternating) minimization by leveraging some prior knowledge about the structure of the unknown image. The inherent ambiguities about shift and flip in the Fourier measurements make this problem especially difficult; and most of the existing methods use several random restarts with different permutations. In this paper, we assume that a known (learned) reference is added to the signal before capturing the Fourier amplitude measurements. Our method is inspired by the principle of adding a reference signal in holography. To recover the signal, we implement an iterative phase retrieval method as an unrolled network. Then we use back propagation to learn the reference that provides us the best reconstruction for a fixed number of phase retrieval iterations. We performed a number of simulations on a variety of datasets under different conditions and found that our proposed method for phase retrieval via unrolled network and learned reference provides near-perfect recovery at fixed (small) computational cost. We compared our method with standard Fourier phase retrieval methods and observed significant performance enhancement using the learned reference."
Dual Grid Net: Hand Mesh Vertex Regression from Single Depth Maps
Authors: Chengde Wan; Thomas Probst; Luc Van Gool; Angela Yao
Venue: ECCV 2020 (main)

Abstract: We aim to recover the dense 3D surface of the hand from depth maps and propose a network that can predict mesh vertices, transformation matrices for every joint and joint coordinates in a single forward pass. Use fully convolutional architectures, we first map depth image features to the mesh grid and then regress the mesh coordinates into real world 3D coordinates. The final mesh is found by sampling from the mesh grid refit in closed-form based on an articulated template mesh.When trained with supervision from sparse key-points, our accuracy is comparable with state-of-the-art on the NYU dataset for key point localization, all while recovering mesh vertices and dense correspondences. Under multi-view settings for training, our framework can also learn through self-supervision by minimizing a set of data-fitting terms and kinematic priors. Our approach is competitive with strongly supervised methods and showcases the potential for self-supervision in dense mesh estimation."
Learning Depth from Focus in the Wild
Authors: Changyeon Won; Hae-Gon Jeon
Venue: ECCV 2022 (main)

Abstract: "For better photography, most recent commercial cameras including smartphones have either adopted large-aperture lens to collect more light or used a burst mode to take multiple images within short times. These interesting features lead us to examine depth from focus/defocus. In this work, we present a convolutional neural network-based depth estimation from single focal stacks. Our method differs from relevant state-of-the-art works with three unique features. First, our method allows depth maps to be inferred in an end-to-end manner even with image alignment. Second, we propose a sharp region detection module to reduce blur ambiguities in subtle focus changes and weakly texture-less regions. Third, we design an effective downsampling module to ease flows of focal information in feature extractions. In addition, for the generalization of the proposed network, we develop a simulator to realistically reproduce the features of commercial cameras, such as changes in field of view, focal length and principal points. By effectively incorporating these three unique features, our network achieves the top rank in the DDFF 12-Scene benchmark on most metrics. We also demonstrate the effectiveness of the proposed method on various quantitative evaluations and real-world images taken from various off-the-shelf cameras compared with state-of-the-art methods. Our source code is publicly available at https://github.com/wcy199705/DfFintheWild."
Learning-Based Point Cloud Registration for 6D Object Pose Estimation in the Real World
Authors: Zheng Dang; Lizhou Wang; Yu Guo; Mathieu Salzmann
Venue: ECCV 2022 (main)

Abstract: "In this work, we tackle the task of estimating the 6D pose of an object from point cloud data. While recent learning-based approaches to addressing this task have shown great success on synthetic datasets, we have observed them to fail in the presence of real-world data. We thus analyze the causes of these failures, which we trace back to the difference between the feature distributions of the source and target point clouds, and the sensitivity of the widely-used SVD-based loss function to the range of rotation between the two point clouds. We address the first challenge by introducing a new normalization strategy, Match Normalization, and the second via the use of a loss function based on the negative log likelihood of point correspondences. Our two contributions are general and can be applied to many existing learning-based 3D object registration frameworks, which we illustrate by implementing them in two of them, DCP and IDAM. Our experiments on the real-scene TUD-L, LINEMOD and Occluded-LINEMOD datasets evidence the benefits of our strategies. They allow for the first time learning-based 3D object registration methods to achieve meaningful results on real-world data. We therefore expect them to be key to the future development of point cloud registration methods."
An End-to-End Transformer Model for Crowd Localization
Authors: Dingkang Liang; Wei Xu; Xiang Bai
Venue: ECCV 2022 (main)

Abstract: "Crowd localization, predicting head positions, is a more practical and high-level task than simply counting. Existing methods employ pseudo-bounding boxes or pre-designed localization maps, relying on complex post-processing to obtain the head positions. In this paper, we propose an elegant, end-to-end Crowd Localization TRansformer named CLTR that solves the task in the regression-based paradigm. The proposed method views the crowd localization as a direct set prediction problem, taking extracted features and trainable embeddings as input of the transformer-decoder. To reduce the ambiguous points and generate more reasonable matching results, we introduce a KMO-based Hungarian matcher, which adopts the nearby context as the external matching cost. Extensive experiments conducted on five datasets in various data settings show the effectiveness of our method. In particular, the proposed method achieves the best localization performance on the NWPU-Crowd, UCF-QNRF, and ShanghaiTech Part A datasets."