GenAI4ELab/papercli-papers huggingface dataset:
Fairness by Learning Orthogonal Disentangled Representations
Authors: Mhd Hasan Sarhan; Nassir Navab; Abouzar Eslami; Shadi Albarqouni
Venue: ECCV 2020 (main)

Abstract: Learning discriminative powerful representations is a crucial step for machine learning systems. Introducing invariance against arbitrary nuisance or sensitive attributes while performing well on specific tasks is an important problem in representation learning. This is mostly approached by purging the sensitive information from learned representations. In this paper, we propose a novel disentanglement approach to invariant representation problem. We disentangle the meaningful and sensitive representations by enforcing orthogonality constraints as a proxy for independence. We explicitly enforce the meaningful representation to be agnostic to sensitive information by entropy maximization. The proposed approach is evaluated on five publicly available datasets and compared with state of the art methods for learning fairness and invariance achieving the state of the art performance on three datasets and comparable performance on the rest. Further, we perform an ablative study to evaluate the effect of each component."
Self-supervision with Superpixels: Training Few-shot Medical Image Segmentation without Annotation
Authors: Cheng Ouyang; Carlo Biffi; Chen Chen; Turkay Kart; Huaqi Qiu; Daniel Rueckert
Venue: ECCV 2020 (main)

Abstract: Few-shot semantic segmentation (FSS) has great potential for medical imaging applications. Most of the existing FSS techniques require abundant annotated semantic classes for training. However, these methods may not be applicable for medical images due to the lack of annotations. To address this problem we make several contributions: (1) A novel self-supervised FSS framework for medical images in order to eliminate the requirement for annotations during training. Additionally, superpixel-based pseudo-labels are generated to provide supervision; (2) An adaptive local prototype pooling module plugged into prototypical networks, to solve the common challenging foreground-background imbalance problem in medical image segmentation; (3) We demonstrate the general applicability of the proposed approach for medical images using three different tasks: abdominal organ segmentation for CT and MRI, as well as cardiac segmentation for MRI. Our results show that, for medical image segmentation, the proposed method outperforms conventional FSS methods which require manual annotations for training."
On Diverse Asynchronous Activity Anticipation
Authors: He Zhao; Richard P. Wildes
Venue: ECCV 2020 (main)

Abstract: We investigate the joint anticipation of long-term activity labels and their corresponding times with the aim of improving both the naturalness and diversity of predictions. We address these matters using Conditional Adversarial Generative Networks for Discrete Sequences. Central to our approach is a reexamination of the unavoidable sample quality vs. diversity tradeoff of the recently emerged Gumbel-Softmax relaxation based GAN on discrete data. In particular, we ameliorate this trade-off with a simple but effective sample distance regularizer. Moreover, we provide a unified approach to inference of activity labels and their times so that a single integrated optimization succeeds for both. With this novel approach in hand, we demonstrate the effectiveness of the resulting discrete sequential GAN on multimodal activity anticipation. We evaluate the approach on three standard datasets and show that it outperforms previous approaches in terms of both accuracy and diversity, thereby yielding a new state-of-the-art in activity anticipation."
Representative-Discriminative Learning for Open-set Land Cover Classification of Satellite Imagery
Authors: Razieh Kaviani Baghbaderani; Ying Qu; Hairong Qi; Craig Stutts
Venue: ECCV 2020 (main)

Abstract: Land cover classification of satellite imagery is an important step toward analyzing the Earth's surface. Existing models assume a closed-set setting where both the training and testing classes belong to the same label set. However, due to the unique characteristics of satellite imagery with an extremely vast area of versatile cover materials, the training data are bound to be non-representative. In this paper, we study the problem of open-set land cover classification that identifies the samples belonging to unknown classes during testing, while maintaining performance on known classes. Although inherently a classification problem, both representative and discriminative aspects of data need to be exploited in order to better distinguish unknown classes from known. We propose a representative-discriminative open-set recognition (RDOSR) framework, which 1) projects data from the raw image space to the embedding feature space that facilitates differentiating similar classes, and further 2) enhances both the representative and discriminative capacity through transformation to a so-called abundance space. Experiments on multiple satellite benchmarks demonstrate the effectiveness of the proposed method. We also show the generality of the proposed approach by achieving promising results on open-set classification tasks using RGB images."
Structure-Aware Human-Action Generation
Authors: Ping Yu; Yang Zhao; Chunyuan Li; Junsong Yuan; Changyou Chen
Venue: ECCV 2020 (main)

Abstract: Generating long-range skeleton-based human actions has been a challenging problem since small deviations of one frame can cause an malformed action sequence. Most existing methods borrow ideas from video generation that naively treat skeleton nodes/joints as pixels of images without considering the rich inter-frame and intra-frame structure information, leading to potential distorted actions. Graph convolutional networks (GCNs) could leverage structure information to learn structure representations. However, adopting GCNs to tackle such continuous action sequences both in spatial and temporal space is challenging as the action graph could be huge. To overcome this challenge, we propose a variant of GCNs to leverage the self-attention mechanism to prune a complete action graph in the temporal space. Our method could dynamically attend to past important frames and construct a sparse graph to apply in the GCN framework, well capturing the structure information in action sequences. Extensive experimental results demonstrate the superiority of our method on two standard human action datasets compared with existing methods."
Towards Efficient Coarse-to-Fine Networks for Action and Gesture Recognition
Authors: Niamul Quader; Juwei Lu; Peng Dai; Wei Li
Venue: ECCV 2020 (main)

Abstract: State-of-the-art approaches to video-based action and gesture recognition often employ two key concepts: First, they employ multistream processing; second, they use an ensemble of convolutional networks. We improve and extend both aspects. First, we systematically yield enhanced receptive fields for complementary feature extraction via coarse-to-fine decomposition of input imagery along the spatial and temporal dimensions, and adaptively focus on training important feature pathways using a reparameterized fully connected layer. Second, we develop a `use when needed' scheme with a `coarse-exit' strategy that allows selective use of expensive high-resolution processing in a data-dependent fashion to retain accuracy while reducing computation cost. Our C2F learning approach builds ensemble networks that outperform most competing methods in terms of both reduced computation cost and improved accuracy on the Something-Something V1, V2, and Jester datasets, while also remaining competitive on the Kinetics-400 dataset. Uniquely, our C2F ensemble networks can operate at varying computation budget constraints."
S³Net: Semantic-Aware Self-supervised Depth Estimation with Monocular Videos and Synthetic Data
Authors: Bin Cheng; Inderjot Singh Saggu; Raunak Shah; Gaurav Bansal; Dinesh Bharadia
Venue: ECCV 2020 (main)

Abstract: Solving depth estimation with monocular cameras enables the possibility of widespread use of cameras as low-cost depth estimation sensors in applications such as autonomous driving and robotics. In order to learn such a scalable depth estimation model, we require a ton of data and labels which are targeted towards specific use-cases. Acquiring these labels is expensive and often requires a calibrated research platform to collect data, which can be unfeasible especially in situations where the terrain is unknown. There are two popular approaches that do not require annotated depth maps:(i) using labelled synthetic and unlabeled real data in an adversarial framework to predict more accurate depth, and (ii) unsupervised models which exploit geometric structure across space and time in monocular video frames. Ideally, we would like to leverage features provided by both approaches as they complement each other; however, existing methods do not adequately exploit these additive benefits. We present a self-supervised framework which combines these complementary features: We use synthetic as well as real images for training while exploiting geometric and temporal constraints. Our novel consolidated architecture performs better than existing state-of-the-art literature. We present a unique way to train this self-supervised framework, and achieve over 15% improvement over the previous supervised approaches with domain adaption and 10% over the previous self-supervised approaches."
Leveraging Seen and Unseen Semantic Relationships for Generative Zero-Shot Learning
Authors: Maunil R Vyas; Hemanth Venkateswara; Sethuraman Panchanathan
Venue: ECCV 2020 (main)

Abstract: Zero-shot learning (ZSL) addresses the unseen class recognition problem by leveraging semantic information to transfer knowledge from seen classes to unseen classes. Generative models synthesize the unseen visual features and convert ZSL into a classical supervised learning problem. These generative models are trained using the seen classes and are expected to implicitly transfer the knowledge from seen to unseen classes. However, their performance is stymied by overfitting, which leads to substandard performance on Generalized Zero-Shot learning (GZSL). To address this concern, we propose the novel LsrGAN, a generative model that Leverages the Semantic Relationship between seen and unseen categories and explicitly performs knowledge transfer by incorporating a novel Semantic Regularized Loss (SR-Loss). The SR-loss guides the LsrGAN to generate visual features that mirror the semantic relationships between seen and unseen classes. Experiments on seven benchmark datasets, including the challenging Wikipedia text-based CUB and NABirds splits, and Attribute-based AWA, CUB, and SUN, demonstrates the superiority of the LsrGAN compared to previous state-of-the-art approaches under both ZSL and GZSL. Our code will be released after the review period. "
Weight Excitation: Built-in Attention Mechanisms in Convolutional Neural Networks
Authors: Niamul Quader; Md Mafijul Islam Bhuiyan; Juwei Lu; Peng Dai; Wei Li
Venue: ECCV 2020 (main)

Abstract: We propose novel approaches for simultaneously identifying important weights of a convolutional neural network (ConvNet) and providing more attention to the important weights during training. More formally, we identify two characteristics of a weight, its magnitude and its location, which can be linked with the importance of the weight. By targeting these characteristics of a weight during training, we develop two separate weight excitation (WE) mechanisms via weight reparameterization-based backpropagation modifications. We demonstrate significant improvements over popular baseline ConvNets on multiple computer vision applications using WE (e.g. 1.3% accuracy improvement over ResNet50 baseline on ImageNet image classification, etc.). These improvements come at no extra computational cost or ConvNet structural change during inference. Additionally, including WE methods in a convolution block is straightforward, requiring few lines of extra code. Lastly, WE mechanisms can provide complementary benefits when used with external attention mechanisms such as the popular Squeeze-and-Excitation attention block."
UNITER: UNiversal Image-TExt Representation Learning
Authors: Yen-Chun Chen; Linjie Li; Licheng Yu; Ahmed El Kholy Faisal Ahmed; Zhe Gan; Yu Cheng; Jingjing Liu
Venue: ECCV 2020 (main)

Abstract: Joint image-text embedding is the bedrock for most Vision-and-Language (V+L) tasks, where multimodality inputs are simultaneously processed for joint visual and textual understanding. In this paper, we introduce UNITER, a UNiversal Image-TExt Representation, learned through large-scale pre-training over four image-text datasets (COCO, Visual Genome, Conceptual Captions, and SBU Captions), which can power heterogeneous downstream V+L tasks with joint multimodal embeddings. We design four pre-training tasks: Masked Language Modeling (MLM), Masked Region Modeling (MRM, with three variants), Image-Text Matching (ITM), and Word-Region Alignment (WRA). Different from previous work that applies joint random masking to both modalities, we use conditional masking on pre-training tasks (i.e., masked language/region modeling is conditioned on full observation of image/text). In addition to ITM for global image-text alignment, we also propose WRA via the use of Optimal Transport (OT) to explicitly encourage fine-grained alignment between words and image regions during pre-training. Comprehensive analysis shows that both conditional masking and OT-based WRA contribute to better pre-training. We also conduct a thorough ablation study to find an optimal combination of pre-training tasks. Extensive experiments show that UNITER achieves new state of the art across six V+L tasks (over nine datasets), including Visual Question Answering, Image-Text Retrieval, Referring Expression Comprehension, Visual Commonsense Reasoning, Visual Entailment, and NLVR$^2$."
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
Authors: Xiujun Li; Xi Yin; Chunyuan Li; Pengchuan Zhang; Xiaowei Hu; Lei Zhang; Lijuan Wang; Houdong Hu; Li Dong; Furu Wei; Yejin Choi; Jianfeng Gao
Venue: ECCV 2020 (main)

Abstract: Large-scale pre-training methods of learning cross-modal representations on image-text pairs are becoming popular for vision-language tasks. While existing methods simply concatenate image region features and text features as input to the model to be pre-trained and use self-attention to learn image-text semantic alignments in a brute force manner, in this paper, we propose a new learning method Oscar, which uses object tags detected in images as anchor points to significantly ease the learning of alignments. Our method is motivated by the observation that the salient objects in an image can be accurately detected, and are often mentioned in the paired text. We pre-train an Oscar model on the public corpus of 6.5 million text-image pairs, and fine-tune it on downstream tasks, creating new state-of-the-arts on six well-established vision-language understanding and generation tasks."
Improving Face Recognition from Hard Samples via Distribution Distillation Loss
Authors: Yuge Huang; Pengcheng Shen; Ying Tai; Shaoxin Li; Xiaoming Liu; Jilin Li; Feiyue Huang; Rongrong Ji
Venue: ECCV 2020 (main)

Abstract: Large facial variations are the main challenge in face recognition. To this end, previous variation-specific methods make full use of task-related prior to design special network losses, which are typically not general among different tasks and scenarios. In contrast, the existing generic methods focus on improving the feature discriminability to minimize the intra-class distance while maximizing the interclass distance, which perform well on easy samples but fail on hard samples. To improve the performance on those hard samples for general tasks, we propose a novel Distribution Distillation Loss to narrow the performance gap between easy and hard samples, which is a simple, effective and generic for various types of facial variations. Specifically, we first adopt state-of-the-art classifiers such as ArcFace to construct two similarity distributions: teacher distribution from easy samples and student distribution from hard samples. Then, we propose a novel distribution-driven loss to constrain the student distribution to approximate the teacher distribution, which thus leads to smaller overlap between the positive and negative pairs in the student distribution. We have conducted extensive experiments on both generic large-scale face benchmarks and benchmarks with diverse variations on race, resolution and pose. The quantitative results demonstrate the superiority of our method over strong baselines, e.g., Arcface and Cosface. "
Extract and Merge: Superpixel Segmentation with Regional Attributes
Authors: Jianqiao An; Yucheng Shi; Yahong Han; Meijun Sun; Qi Tian
Venue: ECCV 2020 (main)

Abstract: For a certain object in an image, the relationship between its central region and the peripheral region is not well utilized in existing superpixel segmentation methods. In this work, we propose the concept of regional attribute, which indicates the location of a certain region in the object. Based on the regional attributes, we propose a novel superpixel method called Extract and Merge (EAM). In the extracting stage, we design square windows with a side length of a power of two, named power-window, to extract regional attributes by calculating boundary clearness of objects in the window. The larger windows are for the central regions and the smaller ones correspond to the peripheral regions. In the merging stage, power-windows are merged according to the defined attraction between them. Specifically, we build a graph model and propose an efficient method to make the large windows merge the small ones strategically, regarding power-windows as vertices and the adjacencies between them as edges. We demonstrate that our superpixels have fine boundaries and are superior to the respective state-of-the-art algorithms on multiple benchmarks."
Spatial-Adaptive Network for Single Image Denoising
Authors: Meng Chang; Qi Li; Huajun Feng; Zhihai Xu
Venue: ECCV 2020 (main)

Abstract: Previous works have shown that convolutional neural networks can achieve good performance in image denoising tasks. However, limited by the local rigid convolutional operation, these methods lead to oversmoothing artifacts. A deeper network structure could alleviate these problems, but at the cost of additional computational overhead. In this paper, we propose a novel spatial-adaptive denoising network (SADNet) for effcient single image blind noise removal. To adapt to changes in spatial textures and edges, we design a residual spatial-adaptive block. Deformable convolution is introduced to sample the spatially related features for weighting. An encoder-decoder structure with a context block is introduced to capture multiscale information. By conducting noise removal from coarse to fine, a high-quality noise-free image is obtained. We apply our method to both synthetic and real noisy image datasets. The experimental results demonstrate that our method outperforms the state-of-the-art denoising methods both quantitatively and visually."
Physics-based Feature Dehazing Networks
Authors: Jiangxin Dong; Jinshan Pan
Venue: ECCV 2020 (main)

Abstract: We propose a physics-based feature dehazing network for image dehazing. In contrast to most existing end-to-end trainable network-based dehazing methods, we explicitly consider the physics model of the haze process in the network design and remove haze in a deep feature space. We propose an effective feature dehazing unit (FDU), which is applied to the deep feature space to explore useful features for image dehazing based on the physics model. The FDU is embedded into an encoder and decoder architecture with residual learning, so that the proposed network can be trained in an end-to-end fashion and effectively help haze removal. The encoder and decoder modules are adopted for feature extraction and clear image reconstruction, respectively. The residual learning is applied to increase the accuracy and ease the training of deep neural networks. We analyze the effectiveness of the proposed network and demonstrate that it can effectively dehaze images with favorable performance against state-of-the-art methods."
Learning Surrogates via Deep Embedding
Authors: Yash Patel; Tomáš Hodaň; Jiří Matas
Venue: ECCV 2020 (main)

Abstract: This paper proposes a technique for training neural networks by minimizing surrogate losses that approximate the target evaluation metric, which may be non-differentiable. The surrogates are learned via a deep embedding where the Euclidean distance between the prediction and the ground truth corresponds to the value of the evaluation metric. The effectiveness of the proposed technique is demonstrated in a post-tuning setup, where a trained model is tuned on the learned surrogate. The scores on the evaluation metric are improved without any significant computational overhead. Without any bells and whistles, improvements are demonstrated on challenging and the practical tasks of scene-text recognition (training with the edit distance metric) and scene-text detection (training with the intersection over union metric for rotated bounding boxes). Relative improvements of up to $38\%$ (in the total edit distance) and $4.25\%$ (in the $F_{1}$ score) were achieved in the recognition and detection tasks respectively."
An Asymmetric Modeling for Action Assessment
Authors: Jibin Gao; Wei-Shi Zheng; Jia-Hui Pan; Chengying Gao; Yaowei Wang; Wei Zeng; Jianhuang Lai
Venue: ECCV 2020 (main)

Abstract: Action assessment is a task of assessing the performance of an action. It is widely applicable to many real-world scenarios such as medical treatment and sporting events. However, existing methods for action assessment are mostly limited to individual actions, especially lacking modeling of the asymmetric relations among agents (e.g., between persons and objects); and this limitation undermines their ability to assess actions containing asymmetrically interactive motion patterns, since there always exists subordination between agents in many interactive actions. In this work, we model the asymmetric interactions among agents for action assessment. In particular, we propose an asymmetric interaction module (AIM), to explicitly model asymmetric interactions between intelligent agents within an action, where we group these agents into a primary one (e.g., human) and secondary ones (e.g., objects). We perform experiments on JIGSAWS dataset containing surgical actions, and additionally collect a new dataset, TASD-2, for interactive sporting actions. The experimental results on two interactive action datasets show the effectiveness of our model, and our method achieves state-of-the-art performance. The extended experiment on AQA-7 dataset also demonstrates the generalization capability of our framework to conventional action assessment."