Context-Agnostic Learning Using Synthetic Data

1 INTRODUCTION . Despite recent advances in deep learning , one central challenge is the large amount of labelled training data required to achieve state-of-the-art performance . Procuring such volumes of high quality , reliably annotated data can be costly or even close to impossible ( e.g. , obtaining data to train an autonomous navigation system for a lunar probe ) . Additional hurdles include hidden biases in large datasets ( Tommasi et al. , 2017 ) and maliciously perturbed training data ( Biggio et al. , 2012 ) . Synthetically generated data has seen growing adoption in response to these problems , since the marginal cost of producing new training data is generally very low , and one has full control over the generation process . This is particularly true for applications with a physical component , such as autonomous navigation ( Gaidon et al. , 2016 ) or robotics ( Todorov et al. , 2012 ) . However , training with purely synthetic data suffers from the so-called “ reality gap ” , whereby good performance on synthetic data does not necessarily yield good performance in the real world ( Jakobi et al. , 1995 ) . In particular , the difficulty of generating realistic training images scales not just with the objects of interest , but also the real-world contexts in which the learned model is expected to operate . This work begins with the simple observation that , for many classification tasks , the label of an input is determined entirely by the object ; however , this additional structure is discarded by current synthetic data pipelines . Our goal is to leverage this decomposition to develop more efficient methods for the related problems of generating training data and learning from a synthetic domain . Our contributions are two-fold : first , we formally introduce the setting of context-agnostic learning , where the input space is decomposed into object and context spaces , and the labels are independent of contexts when conditioned on the objects . Second , we propose an algorithm to efficiently train a classifier in the context-agnostic setting , which relies on the ability to sample from the object and context spaces independently . We apply our methods to train deep neural networks for real-world image classification using only a single synthetic example of each class , obtaining performance comparable to existing methods for domain adaptation and few-shot learning while using substantially less data . Our results show that it is possible to train classifiers in the absence of any contextual training data that nonetheless generalize to real world domains . 2 RELATED WORK . Domain shift refers to the problem that occurs when the training set ( source domain ) and test set ( target domain ) are drawn from different distributions . In this setting , a classifier which performs well on the source domain may not generalize well in the target domain . A standard method for addressing this challenge is domain adaptation , which leverages a small amount of data from the target domain to adapt a function that is learned over the source domain ( Blitzer et al. , 2006 ) . In the context of learning from synthetic data , the domain shift that occurs between synthetic and real world data is known as the reality gap ( Jakobi et al. , 1995 ) . State-of-the-art rendering engines , such as those used for video games , can help narrow this gap by generating photorealistic data for training ( Dosovitskiy et al. , 2017 ; Johnson-Roberson et al. , 2016 ; Qiu and Yuille , 2016 ) . Another technique is using domain randomization to generate the source domain with more variability than is expected in the target domain ( e.g. , extreme lighting conditions and camera angles ) , so as to make real images appear as just another variant ( Tobin et al. , 2017 ; Tremblay et al. , 2018 ) ; in particular , Torres et al . ( 2019 ) apply domain randomization to traffic sign detection and find that arbitrary natural images suffice for the task . Another body of work exploits generative adversarial networks ( Goodfellow et al. , 2014a ) to generate synthetic domains ( Hoffman et al. , 2017 ; Liu et al. , 2017 ; Shrivastava et al. , 2016 ; Taigman et al. , 2016 ; Tzeng et al. , 2017 ) . Finally , several works have explored using synthetic data for natural image text recognition ( Gupta et al. , 2016 ; Jaderberg et al. , 2014 ) . These works use an approach that is roughly analogous to our baseline models , and test their techniques on the target domain of street signs rather than handwritten characters ( as we do ) . A different paradigm for the low-data regime is few-shot learning . In contrast to domain adaptation , few-shot learning operates under the assumption that the target and source distributions are the same , but the ability to sample certain classes is limited in the source domain . Early approaches emphasized capturing knowledge in a Bayesian framework ( Fe-Fei et al. , 2003 ) , which was later formulated as Bayesian program learning ( Lake et al. , 2015 ) . Another approach based on metric learning is to find a nonlinear embedding for objects where closeness in the geometry of the embedding generalizes to unseen classes ( Koch , 2015 ; Snell et al. , 2017 ; Sung et al. , 2018 ; Vinyals et al. , 2016 ) . Meta-learning approaches aim to extract higher level concepts which can be applied to learn new classes from a few examples ( Finn et al. , 2017 ; Munkhdalai and Yu , 2017 ; Nichol et al. , 2018 ; Ravi and Larochelle , 2016 ) . A conceptually-related method that leverages synthetic training data is learning how to generate new data from a few examples of unseen classes ; in contrast to our work , however , these methods still require a large number of samples to learn the synthesizer ( Schwartz et al. , 2018 ; Zhang et al. , 2019 ) . Finally , some works combine domain adaptation with few-shot learning to learn under domain shift and limited samples ( Motiian et al . ( 2017 ) ) . The main characteristic that differentiates our work from these approaches is that we are interested in learning classifiers that are context-agnostic , i.e. , do not rely on background signals . As such , while we find our approach is applicable to many of the same tasks as the aforementioned works , our theoretical setting and objectives differ significantly . From a practical perspective , we demonstrate our techniques when the entire training set consists solely of a single synthetic image of each class , though our techniques can certainly be applied when more data is available ; however we do not expect the reverse to hold for domain adaptation or few-shot learning in our setting . Indeed , we consider this work to be complementary in that we are concerned with exploiting the additional structure that is inherent in certain source domains , while the goal of domain adaptation and fewshot learning is to achieve good performance under various downstream domain shift assumptions . 3 SETTING . The standard supervised learning setting consists of an input space X , an output space Y , and a hypothesis space H of functions mapping X to Y . A domain PD is a probability distribution over ( X , Y ) . Given a target domain PT and a loss function ` , the goal is to learn a classifier h ∈ H that minimizes the risk , i.e. , the expected loss RPT ( h ) : = EPT [ ` ( h ( x ) , y ) ] . The training procedure consists of n samples ( x1 , y1 ) , ... , ( xn , yn ) from a source domain PS . A standard approach is empirical risk minimization , which takes the classifier that minimizes Remp ( h ) = 1n ∑ i ` ( h ( xi ) , yi ) ; if PS is close to PT , then with enough samples , such a classifier also achieves low risk in the target domain . 3.1 CONTEXT-AGNOSTIC LEARNING . In general , we can frame the goal of classification as learning to extract reliable signals for the label y from points x ∈ X . This task is often complicated by the presence of noise or other spurious signals . However , for input spaces generated by physical processes , such signals are generally produced by distinct physical entities and can thus be thought of as independent signals that become mixed via the observation process . We aim to capture this additional structure in our setting . Concretely , we have an object space O , a context space C , and an observation function γ on O × C. The input space X is defined as the image of γ : O × C → X . We will assume that points in O are associated with a unique label in Y , and require that γ preserves this property when passing to X . Note that this setting can be easily generalized to a case when the image of γ is a subdomain of X . In this work , we will consider the special case when X ⊆ C. Conceptually , the context space is an “ ambient space ” containing not only valid inputs , but also random noise or irrelevant classes ; the input space is a subset of the context space for which there exists a well-defined label . For example , in our experiments we explore such a decomposition for the task of traffic sign recognition , where the object space O consists of traffic signs viewed from different angles , the context space C is unconstrained pixel space , and the input space X is the set of images that contain a traffic sign . Recall that the standard objective of learning is to find a good classifier for an unknown subdomain XPT ⊆ X . We consider instead the task of learning a classifier on the entire input space X . To sample from X we are given oracle access to the observation function and draw ( labelled ) samples from O and C independently . Clearly , if this problem is realizable , i.e. , there exists h∗ ∈ H for which RX ( h∗ ) = 0 , then we do not even need to know the target domain PT , since XPT ⊆ X =⇒ [ RX ( h ∗ ) = 0 =⇒ RPT ( h∗ ) = 0 ] Assuming access toX through γ , we can learn h∗ simply by taking the number of samples to infinity . Unfortunately , learning a classifier on X generally requires many more samples than learning a classifier on XPT . Thus we aim to learn h∗ using as few samples as possible . Our new goal will be to learn a classifier over X which depends only on signals from O ; more precisely , we have the following definitions : Definition 3.1 . A function f on X is context-agnostic if Pr [ f ◦ γ ( o , c ) = x ] = Pr [ f ◦ γ ( o , c′ ) = x ] ∀c , c′ ∈ C , o ∈ O , x ∈ Im ( f ) Definition 3.2 . Given a context-agnostic label function y∗ , the objective of context-agnostic learning is to find h ∈ H such that h achieves the lowest risk of all context-agnostic classifiers . The hope is that , since y∗ is context-agnostic , we can learn y∗ through the lower dimensional structure of O using fewer samples . Note , however , that while we only need max ( |O| , |C| ) samples to observe every object and context once , we need |O| ∗ |C| samples to observe every object in every context . Hence the main challenge when the number of samples is low will be avoiding spurious signals , i.e. , statistical correlations between context and objects ( and by extension , labels ) which are artifacts of the sampling process and do not generalize outside the training set . We conclude with some high-level remarks about this setting . First , note that if the problem is realizeable , then the lowest risk classifier is also context-agnostic . Second , we recover the standard supervised setting for the trivial context space C = ∅ . Conversely , classification remains welldefined even in the trivial object space O = { yi } , the set of classes ; however , this pushes all the complexity to the observation function γ , which may be hard to define or intractable to compute . Finally , we do not preclude the existence of useful signals originating from the context for certain domains . For instance , a great deal of information can often be gleaned from the backgrounds of photos , e.g. , stop signs are more often found in cities than on highways . Our theoretical setting avoids this issue by assuming realizability and uniqueness of labels ; more practically , we argue that a “ good ” classifier should nonetheless recognize stop signs on the highway , and our experimental results provide evidence that over-reliance on such background signals leads to brittle classifiers .

Summary: The paper defines the task of context-agnostic learning and proposes an algorithm to solve the problem while assuming the ability to sample objects and contexts independently. They propose to decompose factors contributing to the risk into two, context bias and object error. Based on this interpretation, an algorithm is designed to 'greedily correct bias' while employing adversarial training (or robustness training) for 'local refinement'. The method achieves high accuracy on two synthetic visual tasks, digits and traffic sign classification, when a model is trained using one sample per class from the source domain and tested on an unseen target domain.