Context-Agnostic Learning Using Synthetic Data

1 INTRODUCTION . Despite recent advances in deep learning , one central challenge is the large amount of labelled training data required to achieve state-of-the-art performance . Procuring such volumes of high quality , reliably annotated data can be costly or even close to impossible ( e.g. , obtaining data to train an autonomous navigation system for a lunar probe ) . Additional hurdles include hidden biases in large datasets ( Tommasi et al. , 2017 ) and maliciously perturbed training data ( Biggio et al. , 2012 ) . Synthetically generated data has seen growing adoption in response to these problems , since the marginal cost of producing new training data is generally very low , and one has full control over the generation process . This is particularly true for applications with a physical component , such as autonomous navigation ( Gaidon et al. , 2016 ) or robotics ( Todorov et al. , 2012 ) . However , training with purely synthetic data suffers from the so-called “ reality gap ” , whereby good performance on synthetic data does not necessarily yield good performance in the real world ( Jakobi et al. , 1995 ) . In particular , the difficulty of generating realistic training images scales not just with the objects of interest , but also the real-world contexts in which the learned model is expected to operate . This work begins with the simple observation that , for many classification tasks , the label of an input is determined entirely by the object ; however , this additional structure is discarded by current synthetic data pipelines . Our goal is to leverage this decomposition to develop more efficient methods for the related problems of generating training data and learning from a synthetic domain . Our contributions are two-fold : first , we formally introduce the setting of context-agnostic learning , where the input space is decomposed into object and context spaces , and the labels are independent of contexts when conditioned on the objects . Second , we propose an algorithm to efficiently train a classifier in the context-agnostic setting , which relies on the ability to sample from the object and context spaces independently . We apply our methods to train deep neural networks for real-world image classification using only a single synthetic example of each class , obtaining performance comparable to existing methods for domain adaptation and few-shot learning while using substantially less data . Our results show that it is possible to train classifiers in the absence of any contextual training data that nonetheless generalize to real world domains . 2 RELATED WORK . Domain shift refers to the problem that occurs when the training set ( source domain ) and test set ( target domain ) are drawn from different distributions . In this setting , a classifier which performs well on the source domain may not generalize well in the target domain . A standard method for addressing this challenge is domain adaptation , which leverages a small amount of data from the target domain to adapt a function that is learned over the source domain ( Blitzer et al. , 2006 ) . In the context of learning from synthetic data , the domain shift that occurs between synthetic and real world data is known as the reality gap ( Jakobi et al. , 1995 ) . State-of-the-art rendering engines , such as those used for video games , can help narrow this gap by generating photorealistic data for training ( Dosovitskiy et al. , 2017 ; Johnson-Roberson et al. , 2016 ; Qiu and Yuille , 2016 ) . Another technique is using domain randomization to generate the source domain with more variability than is expected in the target domain ( e.g. , extreme lighting conditions and camera angles ) , so as to make real images appear as just another variant ( Tobin et al. , 2017 ; Tremblay et al. , 2018 ) ; in particular , Torres et al . ( 2019 ) apply domain randomization to traffic sign detection and find that arbitrary natural images suffice for the task . Another body of work exploits generative adversarial networks ( Goodfellow et al. , 2014a ) to generate synthetic domains ( Hoffman et al. , 2017 ; Liu et al. , 2017 ; Shrivastava et al. , 2016 ; Taigman et al. , 2016 ; Tzeng et al. , 2017 ) . Finally , several works have explored using synthetic data for natural image text recognition ( Gupta et al. , 2016 ; Jaderberg et al. , 2014 ) . These works use an approach that is roughly analogous to our baseline models , and test their techniques on the target domain of street signs rather than handwritten characters ( as we do ) . A different paradigm for the low-data regime is few-shot learning . In contrast to domain adaptation , few-shot learning operates under the assumption that the target and source distributions are the same , but the ability to sample certain classes is limited in the source domain . Early approaches emphasized capturing knowledge in a Bayesian framework ( Fe-Fei et al. , 2003 ) , which was later formulated as Bayesian program learning ( Lake et al. , 2015 ) . Another approach based on metric learning is to find a nonlinear embedding for objects where closeness in the geometry of the embedding generalizes to unseen classes ( Koch , 2015 ; Snell et al. , 2017 ; Sung et al. , 2018 ; Vinyals et al. , 2016 ) . Meta-learning approaches aim to extract higher level concepts which can be applied to learn new classes from a few examples ( Finn et al. , 2017 ; Munkhdalai and Yu , 2017 ; Nichol et al. , 2018 ; Ravi and Larochelle , 2016 ) . A conceptually-related method that leverages synthetic training data is learning how to generate new data from a few examples of unseen classes ; in contrast to our work , however , these methods still require a large number of samples to learn the synthesizer ( Schwartz et al. , 2018 ; Zhang et al. , 2019 ) . Finally , some works combine domain adaptation with few-shot learning to learn under domain shift and limited samples ( Motiian et al . ( 2017 ) ) . The main characteristic that differentiates our work from these approaches is that we are interested in learning classifiers that are context-agnostic , i.e. , do not rely on background signals . As such , while we find our approach is applicable to many of the same tasks as the aforementioned works , our theoretical setting and objectives differ significantly . From a practical perspective , we demonstrate our techniques when the entire training set consists solely of a single synthetic image of each class , though our techniques can certainly be applied when more data is available ; however we do not expect the reverse to hold for domain adaptation or few-shot learning in our setting . Indeed , we consider this work to be complementary in that we are concerned with exploiting the additional structure that is inherent in certain source domains , while the goal of domain adaptation and fewshot learning is to achieve good performance under various downstream domain shift assumptions . 3 SETTING . The standard supervised learning setting consists of an input space X , an output space Y , and a hypothesis space H of functions mapping X to Y . A domain PD is a probability distribution over ( X , Y ) . Given a target domain PT and a loss function ` , the goal is to learn a classifier h ∈ H that minimizes the risk , i.e. , the expected loss RPT ( h ) : = EPT [ ` ( h ( x ) , y ) ] . The training procedure consists of n samples ( x1 , y1 ) , ... , ( xn , yn ) from a source domain PS . A standard approach is empirical risk minimization , which takes the classifier that minimizes Remp ( h ) = 1n ∑ i ` ( h ( xi ) , yi ) ; if PS is close to PT , then with enough samples , such a classifier also achieves low risk in the target domain . 3.1 CONTEXT-AGNOSTIC LEARNING . In general , we can frame the goal of classification as learning to extract reliable signals for the label y from points x ∈ X . This task is often complicated by the presence of noise or other spurious signals . However , for input spaces generated by physical processes , such signals are generally produced by distinct physical entities and can thus be thought of as independent signals that become mixed via the observation process . We aim to capture this additional structure in our setting . Concretely , we have an object space O , a context space C , and an observation function γ on O × C. The input space X is defined as the image of γ : O × C → X . We will assume that points in O are associated with a unique label in Y , and require that γ preserves this property when passing to X . Note that this setting can be easily generalized to a case when the image of γ is a subdomain of X . In this work , we will consider the special case when X ⊆ C. Conceptually , the context space is an “ ambient space ” containing not only valid inputs , but also random noise or irrelevant classes ; the input space is a subset of the context space for which there exists a well-defined label . For example , in our experiments we explore such a decomposition for the task of traffic sign recognition , where the object space O consists of traffic signs viewed from different angles , the context space C is unconstrained pixel space , and the input space X is the set of images that contain a traffic sign . Recall that the standard objective of learning is to find a good classifier for an unknown subdomain XPT ⊆ X . We consider instead the task of learning a classifier on the entire input space X . To sample from X we are given oracle access to the observation function and draw ( labelled ) samples from O and C independently . Clearly , if this problem is realizable , i.e. , there exists h∗ ∈ H for which RX ( h∗ ) = 0 , then we do not even need to know the target domain PT , since XPT ⊆ X =⇒ [ RX ( h ∗ ) = 0 =⇒ RPT ( h∗ ) = 0 ] Assuming access toX through γ , we can learn h∗ simply by taking the number of samples to infinity . Unfortunately , learning a classifier on X generally requires many more samples than learning a classifier on XPT . Thus we aim to learn h∗ using as few samples as possible . Our new goal will be to learn a classifier over X which depends only on signals from O ; more precisely , we have the following definitions : Definition 3.1 . A function f on X is context-agnostic if Pr [ f ◦ γ ( o , c ) = x ] = Pr [ f ◦ γ ( o , c′ ) = x ] ∀c , c′ ∈ C , o ∈ O , x ∈ Im ( f ) Definition 3.2 . Given a context-agnostic label function y∗ , the objective of context-agnostic learning is to find h ∈ H such that h achieves the lowest risk of all context-agnostic classifiers . The hope is that , since y∗ is context-agnostic , we can learn y∗ through the lower dimensional structure of O using fewer samples . Note , however , that while we only need max ( |O| , |C| ) samples to observe every object and context once , we need |O| ∗ |C| samples to observe every object in every context . Hence the main challenge when the number of samples is low will be avoiding spurious signals , i.e. , statistical correlations between context and objects ( and by extension , labels ) which are artifacts of the sampling process and do not generalize outside the training set . We conclude with some high-level remarks about this setting . First , note that if the problem is realizeable , then the lowest risk classifier is also context-agnostic . Second , we recover the standard supervised setting for the trivial context space C = ∅ . Conversely , classification remains welldefined even in the trivial object space O = { yi } , the set of classes ; however , this pushes all the complexity to the observation function γ , which may be hard to define or intractable to compute . Finally , we do not preclude the existence of useful signals originating from the context for certain domains . For instance , a great deal of information can often be gleaned from the backgrounds of photos , e.g. , stop signs are more often found in cities than on highways . Our theoretical setting avoids this issue by assuming realizability and uniqueness of labels ; more practically , we argue that a “ good ” classifier should nonetheless recognize stop signs on the highway , and our experimental results provide evidence that over-reliance on such background signals leads to brittle classifiers .

Summary: This paper proposed a context-agnostic learning approach that combines an object area and a context image (s.t. background image) to generate input synthetic image and train the model as context-independent. The proposed method is made more efficient by including this generation process in the training loop, compared to a exhaustive sampling method that randomly selects from a set of combinations of object areas and context images. When applying this method on the task of traffic sign recognition, character recognition, it provides enhanced performance in a setting where model is trained using synthetic data and evaluated on real-world dataset.