mtybilly/Instruct2DS huggingface dataset:
RadarNet: Exploiting Radar for Robust Perception of Dynamic Objects
Authors: Runsheng Guo, Ming Liang, Sergio Casas, Raquel Urtasun Bin Yang
Published in: European Conference on Computer Vision (ECCV), 2020 (Main Conference Track), 2020

Abstract: We tackle the problem of exploiting Radar for perception in the context of self-driving as Radar provides complementary information to other sensors such as LiDAR or cameras in the form of Doppler velocity. The main challenges of using Radar are the noise and measurement ambiguities which have been a struggle for existing simple input or output fusion methods. To better address this, we propose a new solution that exploits both LiDAR and Radar sensors for perception. Our approach, dubbed RadarNet, features a voxel-based early fusion and an attention-based late fusion, which learn from data to exploit both geometric and dynamic information of Radar data. RadarNet achieves state-of-the-art results on two large-scale real-world datasets in the tasks of object detection and velocity estimation. We further show that exploiting Radar improves the perception capabilities of detecting faraway objects and understanding the motion of dynamic objects."
Seeing the Un-Scene: Learning Amodal Semantic Maps for Room Navigation
Authors: Erik Wijmans, Xinlei Chen, Trevor Darrell, Dhruv Batra, Devi Parikh, Amanpreet Singh Medhini Narasimhan
Published in: European Conference on Computer Vision (ECCV), 2020 (Main Conference Track), 2020

Abstract: We introduce a learning-based approach for room navigation using semantic maps. Our proposed architecture learns to predict top-down belief maps of regions that lie beyond the agent’s field of view while modeling architectural and stylistic regularities in houses. First, we train a model to generate amodal semantic top-down maps indicating beliefs of location, size, and shape of rooms by learning the underlying architectural patterns in houses. Next, we use these maps to predict a point that lies in the target room and train a policy to navigate to the point. We empirically demonstrate that by predicting semantic maps, the model learns common correlations found in houses and generalizes to novel environments. We also demonstrate that reducing the task of room navigation to point navigation improves the performance further. We will make our code publicly available and hope our work paves the way for further research in this space. "
Learning to Separate: Detecting Heavily-Occluded Objects in Urban Scenes
Authors: Vitaly Ablavsky, Kaihong Wang, Qi Feng, Margrit Betke Chenhongyi Yang
Published in: European Conference on Computer Vision (ECCV), 2020 (Main Conference Track), 2020

Abstract: While visual object detection with deep learning has received much attention in the past decade, cases when heavy intra-class occlusions occur have not been studied thoroughly. In this work, we propose a novel Non-Maximum-Suppression (NMS) algorithm that dramatically improves the detection recall while maintaining high precision in scenes with heavy occlusions. Our NMS algorithm is derived from a novel embedding mechanism, in which the semantic and geometric features of the detected boxes are jointly exploited. The embedding makes it possible to determine whether two heavily-overlapping boxes belong to the same object in the physical world. Our approach is particularly useful for car detection and pedestrian detection in urban scenes where occlusions often happen. We show the effectiveness of our approach by creating a model called SG-Det (short for Semantics and Geometry Detection) and testing SG-Det on two widely-adopted datasets, KITTI and CityPersons for which it achieves state-of-the-art performance."
Towards causal benchmarking of bias in face analysis algorithms
Authors: Yuanjun Xiong, Wei Xia, Pietro Perona Guha Balakrishnan
Published in: European Conference on Computer Vision (ECCV), 2020 (Main Conference Track), 2020

Abstract: Measuring algorithmic bias is crucial both to assess algorithmic fairness, and to guide the improvement of algorithms. Current bias measurement methods in computer vision are based on observational datasets, and conflate algorithmic bias with dataset bias. To address this problem we develop an experimental method for measuring algorithmic bias of face analysis algorithms, which directly manipulates the attributes of interest, e.g., gender and skin tone, in order to reveal causal links between attribute variation and performance change. Our method is based on generating synthetic image grids that differ along specific attributes while leaving other attributes constant. A crucial aspect of our approach is relying on the perception of human observers, both to guide manipulations, and to measure algorithmic bias. We validate our method by comparing it to a traditional observational bias analysis study in gender classification algorithms. The two methods reach different conclusions. While the observational method reports gender and skin color biases, the experimental method reveals biases due to gender, hair length, age, and facial hair. We also show that our synthetic transects allow for more straightforward bias analysis on minority and intersectional groups."
Learning and Memorizing Representative Prototypes for 3D Point Cloud Semantic and Instance Segmentation
Authors: Dong Gong, Zhi Tian, Chunhua Shen Tong He
Published in: European Conference on Computer Vision (ECCV), 2020 (Main Conference Track), 2020

Abstract: 3D point cloud semantic and instance segmentation are crucial and fundamental for 3D scene understanding. Due to the complex structure, point sets are distributed off-balance and diversely, appearing as both category and pattern imbalance. It has been proved that deep networks can easily forget the non-dominant cases during training, which influences the model generalization and leads to unsatisfactory performance. Although re-weighting on instances may reduce the influence, it is hard to find a balance between the dominant and the non-dominant cases. To tackle the above issue, we propose a memory-augmented network that learns and memorizes the representative prototypes that encode both geometry and semantic information. The prototypes are shared by diverse 3D points and recorded in a universal memory module. During training, the memory slots are dynamically associated with both dominant and non-dominant cases, alleviating the forgetting issue. In testing, the distorted observations and rare cases can thus be augmented by retrieving the stored prototypes, leading to better generalization. Experiments on the benchmarks, i.e., S3DIS and ScanNetV2, show the superiority of our method on both effectiveness and efficiency, which substantially improves the accuracy not only on the entire dataset but also on non-dominant classes and samples."
Knowledge-Based Video Question Answering with Unsupervised Scene Descriptions
Authors: Yuta Nakashima Noa Garcia
Published in: European Conference on Computer Vision (ECCV), 2020 (Main Conference Track), 2020

Abstract: To understand movies, humans constantly reason over the dialogues and actions shown in specific scenes and relate them to the overall storyline already seen. Inspired by this behaviour, we design ROLL, a model for knowledge-based video story question answering that leverages three crucial aspects of movie understanding: dialog comprehension, scene reasoning, and storyline recalling. In ROLL, each of these tasks is in charge of extracting rich and diverse information by 1) processing scene dialogues, 2) generating unsupervised video scene descriptions, and 3) obtaining external knowledge in a weakly supervised fashion. To answer a given question correctly, the information generated by each inspired-cognitive task is encoded via Transformers and fused through a modality weighting mechanism, which balances the information from the different sources. Exhaustive evaluation demonstrates the effectiveness of our approach, which yields a new state-of-the-art on two challenging video question answering datasets: KnowIT VQA and TVQA+."
Transformation Consistency Regularization – A Semi-Supervised Paradigm for Image-to-Image Translation
Authors: Rafal K. Mantiuk Aamir Mustafa
Published in: European Conference on Computer Vision (ECCV), 2020 (Main Conference Track), 2020

Abstract: Scarcity of labeled data has motivated the development of semi-supervised learning methods, which learn from large portions of unlabeled data alongside a few labeled samples. Consistency Regularization between model's predictions under different input perturbations, particularly has shown to provide state-of-the art results in a semi-supervised framework. However, most of these method have been limited to classification and segmentation applications. We propose Transformation Consistency Regularization, which delves into a more challenging setting of image-to-image translation, which remains unexplored by semi-supervised algorithms. The method introduces a diverse set of geometric transformations and enforces the model's predictions for unlabeled data to be invariant to those transformations. We evaluate the efficacy of our algorithm on three different applications: image colorization, denoising and super-resolution. Our method is significantly data efficient, requiring only around 10-20% of labeled samples to achieve similar image reconstructions to its fully-supervised counterpart. Furthermore, we show the effectiveness of our method in video processing applications, where knowledge from a few frames can be leveraged to enhance the quality of the rest of the movie. "
LIRA: Lifelong Image Restoration from Unknown Blended Distortions
Authors: Jianxin Lin, Xin Li, Wei Zhou, Sen Liu, Zhibo Chen Jianzhao Liu
Published in: European Conference on Computer Vision (ECCV), 2020 (Main Conference Track), 2020

Abstract: Most existing image restoration networks are designed in a disposable way and catastrophically forget previously learned distortions when trained on a new distortion removal task. To alleviate this problem, we raise the novel lifelong image restoration problem for blended distortions. We first design a base fork-join model in which multiple pre-trained expert models specializing in individual distortion removal task work cooperatively and adaptively to handle blended distortions. When the input is degraded by a new distortion, inspired by adult neurogenesis in human memory system, we develop a neural growing strategy where the previously trained model can incorporate a new expert branch and continually accumulate new knowledge without interfering with learned knowledge. Experimental results show that the proposed approach can not only achieve state-of-the-art performance on blended distortions removal tasks in both PSNR/SSIM metrics, but also maintain old expertise while learning new restoration tasks."
HDNet: Human Depth Estimation for Multi-Person Camera-Space Localization
Authors: Gim Hee Lee Jiahao Lin
Published in: European Conference on Computer Vision (ECCV), 2020 (Main Conference Track), 2020

Abstract: Current works on multi-person 3D pose estimation mainly focus on the estimation of the 3D joint locations relative to the root joint and ignore the absolute locations of each pose. In this paper, we propose the Human Depth Estimation Network (HDNet), an end-to-end framework for absolute root joint localization in the camera coordinate space. Our HDNet first estimates the 2D human pose with heatmaps of the joints. These estimated heatmaps serve as attention masks for pooling features from image regions corresponding to the target person. A skeleton-based Graph Neural Network (GNN) is utilized to propagate features among joints. We formulate the target depth regression as a bin index estimation problem, which can be transformed with a soft-argmax operation from the classification output of our HDNet. We evaluate our HDNet on the root joint localization and root-relative 3D pose estimation tasks with two benchmark datasets, i.e., Human3.6M and MuPoTS-3D. The experimental results show that we outperform the previous state-of-the-art consistently under multiple evaluation metrics. Our source code is available at: https://github.com/jiahaoLjh/HumanDepth ."
SOLO: Segmenting Objects by Locations
Authors: Tao Kong, Chunhua Shen, Yuning Jiang, Lei Li Xinlong Wang
Published in: European Conference on Computer Vision (ECCV), 2020 (Main Conference Track), 2020

Abstract: We present a new, embarrassingly simple approach to instance segmentation in images. Compared to many other dense prediction tasks, e.g., semantic segmentation, it is the arbitrary number of instances that have made instance segmentation much more challenging. In order to predict a mask for each instance, mainstream approaches either follow the 'detect-thensegment' strategy as used by Mask R-CNN, or predict category masks first then use clustering techniques to group pixels into individual instances. We view the task of instance segmentation from a completely new perspective by introducing the notion of ""instance categories"", which assigns categories to each pixel within an instance according to the instance's location and size, thus nicely converting instance mask segmentation into a single-shot classification-solvable problem. We demonstrate a much simpler and flexible instance segmentation framework with strong performance, achieving on par accuracy with Mask R-CNN and outperforming recent single-shot instance segmenters in accuracy. We hope that this very simple and strong framework can serve as a baseline for many instance-level recognition tasks besides instance segmentation."
Learning to See in the Dark with Events
Authors: Yu Zhang, Zhe Jiang, Dongqing Zou, Jimmy Ren, Bin Zhou Song Zhang
Published in: European Conference on Computer Vision (ECCV), 2020 (Main Conference Track), 2020

Abstract: Imaging in the dark environment is important for many real-world applications like video surveillance. Recently, the development of Event Cameras raises promising directions in solving this task thanks to its High Dynamic Range (HDR) and low requirement of computational sources. However, such cameras record sparse, asynchronous intensity changes of the scene (called events), instead of canonical images. In this paper, we propose learning to see in the dark by translating the HDR events in low light to canonical sharp images as if captured in day light. Since it is extremely challenging to collect paired event-image training data, a novel unsupervised domain adaptation network is proposed which explicitly separates domain-invariant features (e.g. scene structures) from the domain-specific ones (e.g. detailed textures) to ease representation learning. A detail enhancing branch is proposed to reconstruct day light-specific features from the domain-invariant representations in a residual manner, regularized by a ranking loss. To evaluate the proposed approach, a novel large-scale dataset is captured with a DAVIS240C camera with both day/low light events and intensity images. Experiments on this dataset show that the proposed domain adaptation approach achieves superior performance than various state-of-the-art architectures."
Trajectron++: Dynamically-Feasible Trajectory Forecasting With Heterogeneous Data
Authors: Boris Ivanovic, Punarjay Chakravarty, Marco Pavone Tim Salzmann
Published in: European Conference on Computer Vision (ECCV), 2020 (Main Conference Track), 2020

Abstract: Reasoning about human motion is an important prerequisite to safe and socially-aware robotic navigation. As a result, multi-agent behavior prediction has become a core component of modern human-robot interactive systems, such as self-driving cars. While there exist many methods for trajectory forecasting, most do not enforce dynamic constraints and do not account for environmental information (e.g., maps). Towards this end, we present Trajectron++, a modular, graph-structured recurrent model that forecasts the trajectories of a general number of diverse agents while incorporating agent dynamics and heterogeneous data (e.g., semantic maps). Trajectron++ is designed to be tightly integrated with robotic planning and control frameworks; for example, it can produce predictions that are optionally conditioned on ego-agent motion plans. We demonstrate its performance on several challenging real-world trajectory forecasting datasets, outperforming a wide array of state-of-the-art deterministic and generative methods."
Context-Gated Convolution
Authors: Lin Ma, Wei Liu, Shih-Fu Chang Xudong Lin
Published in: European Conference on Computer Vision (ECCV), 2020 (Main Conference Track), 2020

Abstract: As the basic building block of Convolutional Neural Networks (CNNs), the convolutional layer is designed to extract local patterns and lacks the ability to model global context in its nature. Many efforts have been recently devoted to complementing CNNs with the global modeling ability, especially by a family of works on global feature interaction. In these works, the global context information is incorporated into local features before they are fed into convolutional layers. However, research on neuroscience reveals that the neurons' ability of modifying their functions dynamically according to context is essential for the perceptual tasks, which has been overlooked in most of CNNs. Motivated by this, we propose one novel Context-Gated Convolution (CGC) to explicitly modify the weights of convolutional layers adaptively under the guidance of global context. As such, being aware of the global context, the modulated convolution kernel of our proposed CGC can better extract representative local patterns and compose discriminative features. Moreover, our proposed CGC is lightweight and applicable with modern CNN architectures, and consistently improves the performance of CNNs according to extensive experiments on image classification, action recognition, and machine translation. Our code of this paper is available at https://github.com/XudongLinthu/context-gated-convolution."
Polynomial Regression Network for Variable-Number Lane Detection
Authors: Zilei Wang, Yixin Zhang Bingke Wang
Published in: European Conference on Computer Vision (ECCV), 2020 (Main Conference Track), 2020

Abstract: Lane detection is a fundamental yet challenging task in autonomous driving and intelligent traffic systems due to perspective projection and occlusion. Most of previous methods utilize semantic segmentation to identify the regions of traffic lanes in an image, and then adopt some curve-fitting method to reconstruct the lanes. In this work, we propose to use polynomial curves to represent traffic lanes and then propose a novel polynomial regression network (PRNet) to directly predict them, where semantic segmentation is not involved. Specifically, PRNet consists of one major branch and two auxiliary branches: (1) polynomial regression to estimate the polynomial coefficients of lanes, (2) initialization classification to detect the initial retrieval point of each lane, and (3) height regression to determine the ending point of each lane. Through the cooperation of three branches, PRNet can detect variable-number of lanes and is highly effective and efficient. We experimentally evaluate the proposed PRNet on two popular benchmark datasets: TuSimple and CULane. The results show that our method significantly outperforms the previous state-of-the-art methods in terms of both accuracy and speed. "
Structural Deep Metric Learning for Room Layout Estimation
Authors: Jiwen Lu, Jie Zhou Wenzhao Zheng
Published in: European Conference on Computer Vision (ECCV), 2020 (Main Conference Track), 2020

Abstract: In this paper, we propose a structural deep metric learning (SDML) method for room layout estimation, which aims to recover the 3D spatial layout of a cluttered indoor scene from a monocular RGB image. Different from existing room layout estimation methods that solve a regression or per-pixel classification problem, we formulate the room layout estimation problem from a metric learning perspective where we explicitly model the structural relations across different images. We propose to learn a latent embedding space where the Euclidean distance can characterize the actual structural difference between the layouts of two rooms. We then minimize the discrepancy between an image and its ground-truth layout in the learned embedding space. We employ a metric model and a layout encoder to map the RGB images and the ground-truth layouts to the embedding space, respectively, and a layout decoder to map the embeddings to the corresponding layouts, where the whole framework is trained in an end-to-end manner. We perform experiments on the widely used Hedau and LSUN datasets and achieve state-of-the-art performance. "
Adaptive Task Sampling for Meta-Learning
Authors: Zhihao Wang, Doyen Sahoo, Yuan Fang Kun Zhang, Steven C.H. Hoi Chenghao Liu
Published in: European Conference on Computer Vision (ECCV), 2020 (Main Conference Track), 2020

Abstract: Meta-learning methods have been extensively studied and applied in computer vision, especially for few-shot classification tasks. The key idea of meta-learning for few-shot classification is to mimic the few-shot situations faced at test time by randomly sampling classes in meta-training data to construct few-shot tasks for episodic training. While a rich line of work focuses solely on how to extract meta-knowledge across tasks, we exploit the complementary problem on how to generate informative tasks. We argue that the randomly sampled tasks could be sub-optimal and uninformative (e.g., the task of classifying ``dog"" from ``laptop"" is often trivial) to the meta-learner. In this paper, we propose an adaptive task sampling method to improve the generalization performance. Unlike instance based sampling, task based sampling is much more challenging due to the implicit definition of the task in each episode. Therefore, we accordingly propose a greedy class-pair based sampling method, which selects difficult tasks according to class-pair potentials. We evaluate our adaptive task sampling method on two few-shot classification benchmarks, and it achieves consistent improvements across different feature backbones, meta-learning algorithms and datasets."
Deep Complementary Joint Model for Complex Scene Registration and Few-shot Segmentation on Medical Images
Authors: Tiantian Li, Guanyu Yang, Youyong Kong, Yang Chen, Huazhong Shu, Jean-Louis Coatrieux, Jean-Louis Dillenseger, Shuo Li Yuting He
Published in: European Conference on Computer Vision (ECCV), 2020 (Main Conference Track), 2020

Abstract: Deep learning-based medical image registration and segmentation joint models utilize the complementarity (augmentation data or weakly supervised data from registration, region constraints from segmentation) to bring mutual improvement in complex scene and few-shot situation. However, further adoption of the joint models are hindered: 1) the diversity of augmentation data is reduced limiting the further enhancement of segmentation, 2) misaligned regions in weakly supervised data disturb the training process, 3) lack of label-based region constraints in few-shot situation limits the registration performance. We propose a novel Deep Complementary Joint Model (DeepRS) for complex scene registration and few-shot segmentation. We embed a perturbation factor in the registration to increase the activity of deformation thus maintaining the augmentation data diversity. We take a pixel-wise discriminator to extract alignment confidence maps which highlight aligned regions in weakly supervised data so the misaligned regions' disturbance will be suppressed via weighting. The outputs from segmentation model are utilized to implement deep-based region constraints thus relieving the label requirements and bringing fine registration. Extensive experiments on the CT dataset of MM-WHS 2017 Challenge show great advantages of our DeepRS that outperforms the existing state-of-the-art models."
Improving Multispectral Pedestrian Detection by Addressing Modality Imbalance Problems
Authors: Linsen Chen, Xun Cao Kailai Zhou
Published in: European Conference on Computer Vision (ECCV), 2020 (Main Conference Track), 2020

Abstract: Multispectral pedestrian detection is capable of adapting to insufficient illumination conditions by leveraging color-thermal modalities. On the other hand, it is still lacking of in-depth insights on how to fuse the two modalities effectively. Compared with traditional pedestrian detection, we find multispectral pedestrian detection suffers from modality imbalance problems which will hinder the optimization process of dual-modality network and depress the performance of detector. Inspired by this observation, we propose Modality Balance Network (MBNet) which facilitates the optimization process in a much more flexible and balanced manner. Firstly, we design a novel Differential Modality Aware Fusion (DMAF) module to make the two modalities complement each other. Secondly, an illumination aware feature alignment module selects complementary features according to the illumination conditions and aligns the two modality features adaptively. Extensive experimental results demonstrate MBNet outperforms the state-of-the-arts on both the challenging KAIST and CVC-14 multispectral pedestrian datasets in terms of the accuracy and the computational efficiency."
High-Resolution Image Inpainting with Iterative Confidence Feedback and Guided Upsampling
Authors: Zhe Lin, Jimei Yang, Jianming Zhang, Eli Shechtman, Huchuan Lu Yu Zeng
Published in: European Conference on Computer Vision (ECCV), 2020 (Main Conference Track), 2020

Abstract: Existing image inpainting methods often produce artifacts when dealing with large holes in real applications. To address this challenge, we propose an iterative inpainting method with a feedback mechanism. Specifically, we introduce a deep generative model which not only outputs an inpainting result but also a corresponding confidence map. Using this map as feedback, it progressively fills the hole by trusting only high-confidence pixels inside the hole at each iteration and focuses on the remaining pixels in the next iteration. As it reuses partial predictions from the previous iterations as known pixels, this process gradually improves the result. In addition, we propose a guided upsampling network to enable generation of high-resolution inpainting results. We achieve this by extending the Contextual Attention module to borrow high-resolution feature patches in the input image. Furthermore, to mimic real object removal scenarios, we collect a large object mask dataset and synthesize more realistic training data that better simulates user inputs. Experiments show that our method significantly outperforms existing methods in both quantitative and qualitative evaluations. "
Online Ensemble Model Compression using Knowledge Distillation
Authors: Zhiqiang Shen, Marios Savvides Devesh Walawalkar
Published in: European Conference on Computer Vision (ECCV), 2020 (Main Conference Track), 2020

Abstract: This paper presents a novel knowledge distillation based model compression framework consisting of a student ensemble. It enables distillation of simultaneously learnt ensemble knowledge onto each of the compressed student models. Each model learns unique representations from the data distribution due to its distinct architecture. This helps the ensemble generalize better by combining every model’s knowledge. The distilled students and ensemble teacher are trained simultaneously without requiring any pretrained weights. Moreover, our proposed method can deliver multi-compressed students with single training, which is efficient and flexible for different scenarios. We provide comprehensive experiments using state-of-the-art classification models to validate our framework’s effectiveness. Notably, using our framework a 97% compressed ResNet110 student model managed to produce a 10.64% relative accuracy gain over its individual baseline training on CIFAR100 dataset. Similarly a 95% compressed DenseNet-BC (k=12) model managed a 8.17% relative accuracy gain."
Deep Learning-based Pupil Center Detection for Fast and Accurate Eye Tracking System
Authors: Jung Ho Jeon, Byung Cheol Song Kang Il Lee
Published in: European Conference on Computer Vision (ECCV), 2020 (Main Conference Track), 2020

Abstract: In augmented reality (AR) or virtual reality (VR) systems, eye tracking is a key technology and requires significant accuracy as well as real-time operation. Many techniques for detecting pupil centers with error range of iris radius have been developed, but few techniques have precise performance with error range of pupil radius. In addition, the conventional methods rarely guarantee real-time pupil center detection in a general-purpose computer environment due to high complexity. Thus, we propose more accurate pupil center detection by improving the representation quality of the network in charge of pupil center detection. This is realized by representation learning based on mutual information. Also, the latency of the entire system is greatly reduced by using non-local block and self-attention block with large receptive field, which makes it accomplish real-time operation. The proposed system not only shows real-time performance of 52 FPS in a general-purpose computer environment but also provides state-of-the-art accuracy in terms of fine level index of 96.71%, 99.84% and 96.38% for BioID, GI4E and Talking Face Video datasets, respectively."
Efficient Residue Number System Based Winograd Convolution
Authors: Matthew Mattina Zhi-Gang Liu
Published in: European Conference on Computer Vision (ECCV), 2020 (Main Conference Track), 2020

Abstract: Prior research has shown that Winograd algorithm can reduce the computational complexity of convolutional neural networks (CNN) with weights and activations represented in floating point. However it is difficult to apply the scheme to the inference of low-precision quantized (e.g. INT8) networks. Our work extends the Winograd algorithm to Residue Number System (RNS). The minimal complexity convolution is computed precisely over large transformation tile (e.g. 10x10 to 16x16) of filters and activation patches using the Winograd transformation and low cost (e.g. 8-bit) arithmetic without degrading the prediction accuracy of the networks during inference. The arithmetic complexity reduction is up to 7.03x while the performance improvement is up to 2.30x to 4.69x. for 3x3 and 5x5 filters respectively.
Robust Tracking against Adversarial Attacks
Authors: Chao Ma, Yibing Song, Xiaokang Yang Shuai Jia
Published in: European Conference on Computer Vision (ECCV), 2020 (Main Conference Track), 2020

Abstract: While deep convolutional neural networks (CNNs) are vulnerable to adversarial attacks, considerably few efforts have been paid to construct robust deep tracking algorithms against adversarial attacks. Current studies on adversarial attack and defense mainly rest in single images. In this work, we first attempt to generate adversarial examples on top of video sequences to improve the tracking robustness against adversarial attacks. To this end, we take temporal motion into consideration when generating lightweight perturbations over the estimated tracking results frame-by-frame. On one hand, we add the temporal perturbations into the original video sequences as adversarial examples to greatly degrade the tracking performance. On the other hand, we sequentially estimate the perturbations from input sequences and learn to eliminate their effect for performance restoration. We apply the proposed adversarial attack and defense approaches to state-of-the-art deep tracking algorithms. Extensive evaluations on the benchmark datasets demonstrate that the proposed defense method not only eliminates the large performance drops caused by adversarial attacks, but also achieves additional performance gains when deep trackers are not under adversarial attacks."
Single-Shot Neural Relighting and SVBRDF Estimation
Authors: Manmohan Chandraker Shen Sang
Published in: European Conference on Computer Vision (ECCV), 2020 (Main Conference Track), 2020

Abstract: We present a novel physically-motivated deep network for joint shape and material estimation, as well as relighting under novel illumination conditions, using a single image captured by a mobile phone camera. Our physically-based modeling leverages a deep cascaded architecture trained on a large-scale synthetic dataset that consists of complex shapes with microfacet SVBRDF. In contrast to prior works that train rendering layers subsequent to inverse rendering, we propose deep feature sharing and joint training that transfer insights across both tasks, to achieve significant improvements in both reconstruction and relighting. We demonstrate in extensive qualitative and quantitative experiments that our network generalizes very well to real images, achieving high-quality shape and material estimation, as well as image-based relighting. Code, models and data will be publicly released."
Unsupervised 3D Human Pose Representation with Viewpoint and Pose Disentanglement
Authors: Ziwei Liu , Yunhui Liu Qiang Nie
Published in: European Conference on Computer Vision (ECCV), 2020 (Main Conference Track), 2020

Abstract: Learning a good 3D human pose representation is important for human pose related tasks, e.g. human 3D pose estimation and action recognition. Within all these problems, preserving the intrinsic pose information and adapting to view variations are two critical issues. In this work, we propose a novel Siamese denoising autoencoder to learn a 3D pose representation by disentangling the pose-dependent and view-dependent feature from the human skeleton data, in a fully unsupervised manner. These two disentangled features are utilized together as the representation of the 3D pose. To consider both the kinematic and geometric dependencies, a sequential bidirectional recursive network (SeBiReNet) is further proposed to model the human skeleton data. Extensive experiments demonstrate that the learned representation 1) preserves the intrinsic information of human pose, 2) shows good transferability across datasets and tasks. Notably, our approach achieves state-of-the-art performance on two inherently different tasks: pose denoising and unsupervised action recognition. Code and models are available at: https://github.com/NIEQiang001/unsupervised-human-pose.git."
Angle-based Search Space Shrinking for Neural Architecture Search
Authors: Yuding Liang, Zichao Guo, Ruosi Wan, Xiangyu Zhang, Yichen Wei, Qingyi Gu, Jian Sun Yiming Hu
Published in: European Conference on Computer Vision (ECCV), 2020 (Main Conference Track), 2020

Abstract: In this work, we present a simple and general search space shrinking method, called Angle-Based search space Shrinking (ABS), for Neural Architecture Search (NAS). Our approach progressively simplifies the original search space by dropping unpromising candidates, thus can reduce difficulties for existing NAS methods to find superior architectures. In particular, we propose an angle-based metric to guide the shrinking process. We provide comprehensive evidences showing that, in weight-sharing supernet, the proposed metric is more stable and accurate than accuracy-based and magnitude-based metrics to predict the capability of child models. We also show that the angle-based metric can converge fast while training supernet, enabling us to get promising shrunk search spaces efficiently. ABS can easily apply to most of NAS approaches (e.g. SPOS, FairNAS, ProxylessNAS, DARTS and PDARTS). Comprehensive experiments show that ABS can dramatically enhance existing NAS approaches by providing a promising shrunk search space."
RobustScanner: Dynamically Enhancing Positional Clues for Robust Text Recognition
Authors: Zhanghui Kuang, Chenhao Lin, Hongbin Sun, Wayne Zhang Xiaoyu Yue
Published in: European Conference on Computer Vision (ECCV), 2020 (Main Conference Track), 2020

Abstract: The attention-based encoder-decoder framework has recently achieved impressive results for scene text recognition, and many variants have emerged with improvements in recognition quality. However, it performs poorly on contextless texts (e.g., random character sequences) which is unacceptable in most of real application scenarios. In this paper, we first deeply investigate the decoding process of the decoder. We empirically find that a representative character-level sequence decoder utilizes not only context information but also positional information. Contextual information, which the existing approaches heavily rely on, causes the problem of attention drift. To suppress such side-effect, we propose a novel position enhancement branch, and dynamically fuse its outputs with those of the decoder attention module for scene text recognition. Specifically, it contains a position aware module to enable the encoder to output feature vectors encoding their own spatial positions, and an attention module to estimate glimpses using the positional clue (i.e., the current decoding time step) only. The dynamic fusion is conducted for more robust feature via an element-wise gate mechanism. Theoretically, our proposed method, dubbed RobustScanner, decodes individual characters with dynamic ratio between context and positional clues, and utilizes more positional ones when the decoding sequences with scarce context, and thus is robust and practical. Empirically, it has achieved new state-of-the-art results on popular regular and irregular text recognition benchmarks while without much performance drop on contextless benchmarks, validating its robustness in both contextual and contextless application scenarios."
Towards Fast, Accurate and Stable 3D Dense Face Alignment
Authors: Xiangyu Zhu, Yang Yang, Fan Yang, Zhen Lei, Stan Z. Li Jianzhu Guo
Published in: European Conference on Computer Vision (ECCV), 2020 (Main Conference Track), 2020

Abstract: Accurate and Stable 3D Dense Face Alignment","Existing methods of 3D dense face alignment mainly concentrate on accuracy, thus limiting the scope of their practical applications. In this paper, we propose a novel regression framework which makes a balance among speed, accuracy and stability. Firstly, on the basis of a lightweight backbone, we propose a meta-joint optimization strategy to dynamically regress a small set of 3DMM parameters, which greatly enhances speed and accuracy simultaneously. To further improve the stability on videos, we present a virtual synthesis method to transform one still image to a short-video which incorporates in-plane and out-of-plane face moving. On the premise of high accuracy and stability, our model runs at over 50fps on a single CPU core and outperforms other state-of-the-art heavy models simultaneously. Experiments on several challenging datasets validate the efficiency of our method. The code and models will be available at \url{https://github.com/cleardusk/3DDFA_V2}."
Iterative Feature Transformation for Fast and Versatile Universal Style Transfer
Authors: Danna Gurari Tai-Yin Chiu
Published in: European Conference on Computer Vision (ECCV), 2020 (Main Conference Track), 2020

Abstract: The general framework for fast universal style transfer consists of an autoencoder and a feature transformation at the bottleneck. We propose a new transformation that iteratively stylizes features with analytical gradient descent. Experiments show this transformation is advantageous in part because it is fast. With control knobs to balance content preservation and style effect transferal, we also show this method can switch between artistic and photo-realistic style transfers and reduce distortion and artifacts. Finally, we show it can be used for applications requiring spatial control and multiple-style transfer. "
CATCH: Context-based Meta Reinforcement Learning for Transferrable Architecture Search
Authors: Yawen Duan, Zewei Chen, Hang Xu, Zihao Chen, Xiaodan Liang, Tong Zhang, Zhenguo Li Xin Chen
Published in: European Conference on Computer Vision (ECCV), 2020 (Main Conference Track), 2020

Abstract: Neural Architecture Search (NAS) achieved many breakthroughs in recent years. In spite of its remarkable progress, many algorithms are restricted to particular search spaces. They also lack efficient mechanisms to reuse knowledge when confronting multiple tasks. These challenges preclude their applicability, and motivate our proposal of CATCH, a novel Context-bAsed meTa reinforcement learning (RL) algorithm for transferrable arChitecture searcH. The combination of meta-learning and RL allows CATCH to efficiently adapt to new tasks while being agnostic to search spaces. CATCH utilizes a probabilistic encoder to encode task properties into latent context variables, which then guide CATCH' s controller to quickly ""catch"" top-performing networks. The contexts also assist a network evaluator in filtering inferior candidates and speed up learning. Extensive experiments demonstrate CATCH' s universality and search efficiency over many other widely-recognized algorithms. It is also capable of handling cross-domain architecture search as competitive networks on ImageNet, COCO, and Cityscapes are identified. This is the first work to our knowledge that proposes an efficient transferrable NAS solution while maintaining robustness across various settings."
Toward Faster and Simpler Matrix Normalization via Rank-1 Update
Authors: Yunfeng Cai, Ping Li Tan Yu
Published in: European Conference on Computer Vision (ECCV), 2020 (Main Conference Track), 2020

Abstract: Bilinear pooling has achieved an impressive improvement in many computer vision tasks. Recent studies discover that matrix normalization is vital for improving the performance of bilinear pooling. Nevertheless, traditional matrix square-root or logarithm normalization methods need singular value decomposition (SVD), which is not well suited in the GPU platform, limiting its efficiency in training and inference. To boost the efficiency in the GPU platform, recent methods rely on Newton-Schulz (NS) iteration to approximate the matrix square-root. Despite NS iteration is GPU-friendly, it takes several times matrix-matrix multiplications, which is still expensive in computation. Besides, the proposed RUN is much simpler than NS iteration and thus much easier for implementation in practice. Meanwhile, NS iteration can not well support the compact bilinear feature obtained from tensor sketch or random projection. To overcome these limitations, we propose a rank-1 update normalization (RUN), which only needs matrix-vector multiplications and thus is significantly more efficient than NS iteration using matrix-matrix multiplications. Moreover, our RUN readily supports the normalization on compact bilinear features, making it more flexible to be deployed compared with NS iteration. The proposed RUN is differentiable and can be plugged in a neural network for an end-to-end training. Comprehensive experiments on four public benchmarks show that, for the full bilinear pooling, the proposed RUN achieves comparable accuracies with a $330 imes$ speedup over NS iteration. For the compact bilinear pooling, our RUN achieves comparable accuracies with a $5400 imes$ speedup over SVD-based normalization."
Accurate Polarimetric BRDF for Real Polarization Scene Rendering
Authors: Taishi Ono, Legong Sun, Yasutaka Hirasawa, Jun Murayama Yuhi Kondo
Published in: European Conference on Computer Vision (ECCV), 2020 (Main Conference Track), 2020

Abstract: Polarization has been used to solve a lot of computer vision tasks such as Shape from Polarization (SfP). But existing methods suffer from ambiguity problems of polarization. To overcome such problems, some research works have suggested to use Convolutional Neural Network (CNN). But acquiring large scale dataset with polarization information is a very difficult task. If there is an accurate model which can describe a complicated phenomenon of polarization, we can easily produce synthetic polarized images with various situations to train CNN. In this paper, we propose a new polarimetric BRDF (pBRDF) model. We prove its accuracy by fitting our model to measured data with variety of light and camera conditions. We render polarized images using this model and use them to estimate surface normal. Experiments show that the CNN trained by our polarized images has more accuracy than one trained by RGB only."
Lensless Imaging with Focusing Sparse URA Masks in Long-Wave Infrared and its Application for Human Detection
Authors: Hideki Oyaizu, Kenichiro Nakamura, Ryuta Satoh, Suguru Ushiki, Ryuichi Tadano, Atsushi Ito, Jun Murayama Ilya Reshetouski
Published in: European Conference on Computer Vision (ECCV), 2020 (Main Conference Track), 2020

Abstract: We introduce a lensless imaging framework for contemporary computer vision applications in long-wavelength infrared (LWIR). The framework consists of two parts: a novel lensless imaging method that utilizes the idea of local directional focusing for optimal binary sparse coding, and lensless imaging simulator based on Fresnel-Kirchhoff diffraction approximation. Our lensless imaging approach, besides being computationally efficient, is calibration-free and allows for wide FOV imaging. We employ our lensless imaging simulation software for optimizing reconstruction parameters and for synthetic images generation for CNN training. We demonstrate the advantages of our framework on a dual-camera system (RGB-LWIR lensless), where we perform CNN-based human detection using the fused RGB-LWIR data."
Topology-Preserving Class-Incremental Learning
Authors: Xinyuan Chang, Xiaopeng Hong, Xing Wei, Yihong Gong Xiaoyu Tao
Published in: European Conference on Computer Vision (ECCV), 2020 (Main Conference Track), 2020

Abstract: A well-known issue for class-incremental learning is the catastrophic forgetting phenomenon, where the network's recognition performance on old classes degrades severely when incrementally learning new classes. To alleviate forgetting, we put forward to preserve the old class knowledge by maintaining the topology of the network's feature space. On this basis, we propose a novel topology-preserving class-incremental learning (TPCIL) framework. TPCIL uses an elastic Hebbian graph (EHG) to model the feature space topology, which is constructed with the competitive Hebbian learning rule. To maintain the topology, we develop the topology-preserving loss (TPL) that penalizes the changes of EHG's neighborhood relationships during incremental learning phases. Comprehensive experiments on CIFAR100, ImageNet, and subImageNet datasets demonstrate the power of the TPCIL for continuously learning new classes with less forgetting. The code will be released."
Inter-Image Communication for Weakly Supervised Localization
Authors: Yunchao Wei, Yi Yang Xiaolin Zhang
Published in: European Conference on Computer Vision (ECCV), 2020 (Main Conference Track), 2020

Abstract: Weakly supervised localization aims at finding target object regions using only image-level supervision. However, localization maps extracted from classification networks are often not accurate due to the lack of fine pixel-level supervision. In this paper, we propose to leverage pixel-level similarities across different objects for learning more accurate object locations in a complementary way. Particularly, two kinds of constraints are proposed to prompt the consistency of object features within the same categories. The first constraint is to learn the stochastic feature consistency among discriminative pixels that are randomly sampled from different images within a batch. The discriminative information embedded in one image can be leveraged to benefit its counterpart with inter-image communication. The second constraint is to learn the global consistency of object features throughout the entire dataset. We learn a feature center for each category and realize the global feature consistency by forcing the object features to approach class-specific centers. The global centers are actively updated with the training process. The two constraints can benefit each other to learn consistent pixel-level features within the same categories, and finally improve the quality of localization maps. We conduct extensive experiments on two popular benchmarks, i.e., ILSVRC and CUB-200-2011. Our method achieves the Top-1 localization error rate of 45.17% on the ILSVRC validation set, surpassing the current state-of-the-art method by a large margin. The code is available at https://github.com/xiaomengyc/I2C."