Towards Context-Agnostic Learning Using Synthetic Data

1 Introduction . We study the problem of learning to classifying images of known objects when placed in context , given only a single synthetic example of each object ; our empirical evaluation considers the tasks of traffic sign and handwritten character recognition . Our methods also enable us to explore the extent to which deep neural networks trained using realworld data to perform image classification can over-rely on signals from the backgrounds of images , even when no such information is necessary for classification . Intuitively , even though contextual clues may be required in some settings , given an input that unambiguously contains the object of interest in the foreground , we expect a robust classifier to be invariant to the rest of the image . We present two main contributions . First , we introduce a formal setting for our study , where the input space is decomposed into object and context spaces , and the labels are independent of contexts when conditioned on the objects . We introduce the goal of learning a context-agnostic classifier , i.e. , a classifier whose predictions are invariant under perturbations of the context . We derive a new risk bound for this setting that is tight up to a factor of two but also independent of the true labels , which holds so long as the classifier outperforms random guessing . Second , we present a new technique for automatically generating data to train a deep neural network for image classification . The technique works by sampling independently from the object space , which contains the transformed views of the objects , and the context space , which is designed to include challenging backgrounds that make the resulting images difficult to classify . The hypothesis is that training the network to accurately classify objects against these challenging backgrounds should produce a trained network that robustly generalizes to accurately classify objects against the range of backgrounds that it may encounter when deployed in more natural settings . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We empirically validate our methods by training deep neural networks for a variety of real-world image classification tasks using only a single synthetic example of each class , obtaining robust performance in the context-agnostic setting on natural data . Conversely , we find that classifiers trained without our techniques using only natural data achieve negligible accuracy even under relatively benign perturbations that leave a well-defined object in the foreground completely untouched . These results demonstrate the ability of our technique to train accurate and robust classifiers using only small amounts of high quality synthetic data , while also highlighting the need for future work in understanding the performance of deep learning systems in the context-agnostic setting when trained on natural data . 2 Related work . Domain shift refers to the problem that occurs when the training set ( source domain ) and test set ( target domain ) are drawn from different data distributions . In this setting , a classifier which performs well on the source domain may not generalize well to the target domain . A standard method for addressing this challenge is domain adaptation , which leverages a small amount of data from the target domain to adapt a function that is learned over the source domain [ Blitzer et al. , 2006 ] . Approaches can be further categorized as supervised , using small amounts of labelled data from the target domain [ Donahue et al. , 2014 , Motiian et al. , 2017b , Saenko et al. , 2010 , Tzeng et al. , 2015 ] ; unsupervised , typically requiring large amounts of unlabelled data from the target domain [ Baktashmotlagh et al. , 2013 , Fernando et al. , 2013 , Ganin and Lempitsky , 2014 , Gopalan et al. , 2011 ] ; or semi-supervised , using a mixture of both types of data [ Gong et al. , 2012 , Pan et al. , 2010 , Yao et al. , 2015 ] . In the context of learning from synthetic data , the domain shift that occurs between synthetic and real world data is known as the reality gap [ Jakobi et al. , 1995 ] . State-of-the-art rendering engines , such as those used for video games , can help narrow this gap by generating photorealistic data for training [ Dosovitskiy et al. , 2017 , Johnson-Roberson et al. , 2016 , Qiu and Yuille , 2016 ] . Another technique , particularly prevalent in the robotics community , is known as domain randomization , where the synthetic training data is generated with more variability than expected in the actual test environment ( e.g. , extreme lighting conditions and camera angles ) , so that the additional robustness also transfers across the reality gap [ Tobin et al. , 2017 , Tremblay et al. , 2018 ] ; in particular , Torres et al . [ 2019 ] apply domain randomization to traffic sign detection and find that arbitrary natural images suffice for the task . Another body of work exploits generative adversarial networks ( GANs ) [ Goodfellow et al. , 2014a ] to generate synthetic domains [ Hoffman et al. , 2017 , Liu et al. , 2017 , Shrivastava et al. , 2016 , Taigman et al. , 2016 , Tzeng et al. , 2017 ] . In particular , Shetty et al . [ 2019 ] use a GAN trained to perform in-painting and replace extraneous objects in images as a data-augmentation technique to reduce the trained model ’ s dependence on co-occurring classes . A different paradigm for the low-data regime is few-shot learning . In contrast to domain adaptation , the goal of few-shot learning is to generalize to new classes given only a few examples , given the ability to train on large amounts of data containing related classes . Early approaches emphasized capturing knowledge in a Bayesian framework [ Fe-Fei et al. , 2003 ] , which was later formulated as Bayesian program learning [ Lake et al. , 2015 ] . Another approach based on metric learning is to find a nonlinear embedding for objects where closeness in the geometry of the embedding generalizes to unseen classes [ Koch , 2015 , Snell et al. , 2017 , Sung et al. , 2018 , Vinyals et al. , 2016 ] . Meta-learning approaches aim to extract higher level concepts which can be applied to learn new classes from a few examples [ Finn et al. , 2017 , Munkhdalai and Yu , 2017 , Nichol et al. , 2018 , Ravi and Larochelle , 2016 ] . A conceptually-related method that leverages synthetic training data is learning how to generate new data from a few examples of unseen classes ; in contrast to our work , however , these methods still require a large number of samples to learn the synthesizer [ Schwartz et al. , 2018 , Zhang et al. , 2019 ] . Finally , some works combine domain adaptation with few-shot learning to learn under domain shift and limited samples ( Motiian et al . [ 2017a ] ) . The main characteristic that differentiates our work from these approaches is that we are interested in learning classifiers that are context-agnostic , i.e. , do not rely on background signals . As such , while we find our approach is applicable to many of the same tasks as the aforementioned works , our theoretical setting and objectives differ significantly . From a practical perspective , we demonstrate our techniques when the entire training set consists solely of a single synthetic image of each class , though our techniques can certainly be applied when more data is available ; however the reverse does not hold , i.e. , existing approaches for domain adaptation or few-shot learning can not be applied to our setting . Indeed , we consider this work to be complementary in that we are concerned with exploiting the additional structure that is inherent in certain domains , while the goal of domain adaptation and few-shot learning is to achieve good performance on a downstream target distribution given data from a related source distribution . 3 Context-agnostic learning . In this section , we introduce the formal setting of context-agnostic learning . We begin with some notation from the standard supervised learning setting . We are given an input space X , an output space Y , and a hypothesis space H of functions mapping X to Y . A domain PD is a probability distribution over ( X , Y ) . Given a target domain PT and a loss function ` , the goal is to learn a classifier h 2 H that minimizes the risk , i.e. , the expected loss RPT ( h ) : = EPT [ ` ( h ( x ) , y ) ] . The training procedure is given as input a set of n training samples ( x1 , y1 ) , ... , ( xn , yn ) drawn from a source domain PS . The standard approach is empirical risk minimization , which takes the classifier that minimizes Remp ( h ) = 1n P i ` ( h ( xi ) , yi ) . This technique is known to converge to the optimal classifier over PS as the number of samples increases ; furthermore , if PS is sufficiently “ close ” to PT ( e.g. , if PS = PT , as is the case when there is no domain adaptation ) , then such a classifier also achieves low risk in the target domain . 3.1 Formal setting . In general , we can frame the goal of classification as learning to extract reliable signals for the label y from points x 2 X . This task is often complicated by the presence of noise or other spurious signals . However , for input spaces generated by physical processes , such signals are generally produced by distinct physical entities and can thus be thought of as independent signals that become mixed via an observation process . We aim to capture this additional structure in our setting . Concretely , we have an object space O , a context space C , and an observation function on O ⇥ C. The input space X is defined as the image of : O ⇥ C ! X . We will assume that points in O are associated with a unique label in Y , and points in X inherit labels from O via ; in particular , we require that be injective with respect to labels , i.e. , always maps objects with different labels in O to distinct points in X . In this work , we will consider the special case when X ✓ C. Conceptually , the context space is an “ ambient space ” containing not only valid inputs , but also random noise or irrelevant classes ; the input space is a subset of the context space for which there exists a well-defined label . For example , in our experiments we explore such a decomposition for the task of traffic sign recognition , where the object space O consists of traffic signs viewed from different angles , the context space C is unconstrained pixel space , and the input space X is the set of images that contain a traffic sign . Recall that the standard objective of learning is to find a good classifier for an unknown subdomain XPT ✓ X . We consider instead the task of learning a classifier on the entire input space X . To sample from X we are given oracle access to the observation function and draw ( labelled ) samples from O and C independently . Clearly , if this problem is realizable , i.e. , there exists h⇤ 2 H for which RX ( h⇤ ) = 0 , then we do not even need to know the target domain PT , since XPT ✓ X = ) ⇥ RX ( h ⇤ ) = 0 = ) RPT ( h⇤ ) = 0 ⇤ ( 1 ) Hence our new goal will be to learn a classifier over X which depends only on signals from O ; more precisely , we have the following definitions : Definition 3.1 . A function f on X is context-agnostic if Pr [ f ( o , c ) = y ] = Pr [ f ( o , c0 ) = y ] 8c , c0 2 C , o 2 O , y 2 Im ( f ) ( 2 ) Definition 3.2 . The objective of context-agnostic learning is to find h 2 H such that h achieves the lowest risk of all context-agnostic classifiers . Remark . First , note that if the true label function y⇤ is realizeable , then the lowest risk classifier is also context-agnostic . Second , we recover the standard supervised setting for the trivial context space C = ; . Conversely , our setting remains well-defined even in the trivial object space O = { yi } i , the set of classes ; however , this pushes all the complexity to the observation function , which may be hard to define or intractable to compute . Finally , we do not preclude the existence of useful signals originating from the context for certain domains . For instance , a great deal of information can often be gleaned from the backgrounds of photos , e.g. , stop signs are more often found in cities than on highways . Our theoretical setting avoids this issue by assuming realizability and uniqueness of labels ; more practically , we argue that a “ good ” classifier should nonetheless recognize stop signs on the highway , and our experimental results provide evidence that over-reliance on such background signals leads to brittle classifiers .

Summary: This paper proposes a data augmentation scheme that synthesizes image background so that models can achieve better generalization by learning from one synthetic image and generalizing to real natural images. The process involves using previous images as background and adding adversarial noises. Experimental results show that it can generalize from synthetic traffic signs and digit images to real traffic sign images and handwritten digit images, outperforming a baseline that directly trains from the synthetic images without augmentations or with random background augmentations.