Towards Context-Agnostic Learning Using Synthetic Data
Authors: Charles Jin
Martin Rinard
Venue: NeurIPS 2021 Poster (2021)
Keywords: synthetic data
context agnostic learning

Abstract: We propose a novel setting for learning, where the input domain is the image of a map defined on the product of two sets, one of which completely determines the labels. We derive a new risk bound for this setting that decomposes into a bias and an error term, and exhibits a surprisingly weak dependence on the true labels. Inspired by these results, we present an algorithm aimed at minimizing the bias term by exploiting the ability to sample from each set independently. We apply our setting to visual classification tasks, where our approach enables us to train classifiers on datasets that consist entirely of a single synthetic example of each class. On several standard benchmarks for real-world image classification, we achieve robust performance in the context-agnostic setting, with good generalization to real world domains, whereas training directly on real world data without our techniques yields classifiers that are brittle to perturbations of the background.

Towards Context-Agnostic Learning
Using Synthetic Data
Charles Jin
Martin Rinard CSAIL MIT
CSAIL MIT
Cambridge, MA 02139 cej@csail. mit.
Cambridge, MA 02139 rinard@csail
Abstract
We propose a novel setting for learning, where the input domain the image of a map defined the product of two sets, of which completely determines the labels. We derive a new risk bound for this setting that decomposes into a bias and an error term, and exhibits surprisingly weak dependence on the true labels. Inspired by these results, we present an algorithm aimed at minimizing the bias term by exploiting the ability to sample from each set independently. We apply setting to visual classification tasks, where our approach enables us to train classi- fiers on datasets that consist entirely of single synthetic example of each class. On several standard benchmarks for real-world image classification, we achieve robust performance the context-agnostic setting, with good generalization to real world domains, whereas training directly on real world data without our techniques yields classifiers that brittle to perturbations of the background.
Introduction
We study the problem of learning classifying images known objects when placed in context, given only single synthetic example of each object; our empirical evaluation considers the tasks of traffic sign and handwritten character recognition.
Our methods also enable us to explore the extent to which deep neural networks trained using real- world data to perform image classification
over-rely signals from the backgrounds of images, even when no such information is necessary for classification. Intuitively, even though contextual clues may be required in some settings, given an input that anambiguously contains the object of interest in the foreground, we expect a robust classifier to be invariant to the rest of the image.
We present two main contributions. First, we introduce formal setting for our study, where the input space is decomposed into object and context spaces, and the labels are independent of contexts when conditioned the objects. We introduce the goal of learning the context. We derive a new risk
context-agnosie classifier, i.c.,
classifier whose predictions are invariant under perturbations bound for this setting that is fight up factor of two but also independent of the true labels, which holds so long as the classifier outperforms random guessing.
Second, we present new technique for automatically generating data to train a deep neural network for image classification. The technique works by sampling independently from the object space, which contains the transformed views of the objects, and the context space, which is designed to include challenging backgrounds that make the resulting images difficult classify. The hypothesis is that training the network to accurately classify objects against these challenging backgrounds should produce a trained network that robustly generalizes to accurately classify objects against the range of backgrounds that it may encounter when deployed in more natural settings.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

We empirically validate our methods by training deep neural networks for variety of real-world image classification tasks using only a single synthetic example of each class, obtaining robust performance in the context-agnostic setting on natural data. Conversely, we find that classifiers trained without our techniques using only natural data achieve negligible accuracy even under relatively benign perturbations that leave a well-defined object in the foreground completely untouched. These results demonstrate the ability of our technique to train accurate and robust classifiers using only small amounts of high quality synthetic data, while also highlighting the need for future work in understanding the performance of deep learning systems in the context-agnostic setting when trained on natural data.
2 Related work
Domain shift refers to the problem that occurs when the training set (source domain) and test set (target domain) drawn from different data distributions. In this setting, a classifier which performs well on the source domain may not generalize well to the target domain. A standard method for addressing this challenge is domain adaptation, which leverages small amount of data from the target domain to adapt a function that is learned over the source domain [Blitzer et al. |006]. Approaches can be further categorized supervised, using small amounts of labelled data from the target domain
(Donahue et al., 2014, Motiian et al., 2017b Saenko et al. 2010 Tzeng et al. 2015]: unsupervised. typically requiring large amounts of unlabelled data from the target domain Baktashmotlagh et al.)
Fernando et al. 2013 Ganin and Lempitsky 2014 Gopalan et al. 201 or semi-supervised
using a mixture of both types of data [[Gong et al. 2012 Pan et al., 2010 Yao et al. 2015).
In the context of learning from synthetic data, the domain shift that occurs between synthetic and real world data is known the reality gap (Jkobbi al.] [995]. State-of-the-arr rendering engines, such as those used for video games, can help narrow this gap by generating photorealistic data for training al. 2016 Qiu and Yuille 2016]. Another technique,
(Dosovisskiy et al. 2017 Johnson-Roberson particularly prevalent in the robotics community, is known as domain randomization, where the synthetic training data is generated with more variability than expected in the actual test environment (e.g., extreme lighting conditions and camera angles), so that the additional robustness also transfers across the reality gap [Tobin et al. 2017 Tremblay al. 2018 ;; in particular, Torres et al. 20199 apply domain randomization to traffic sign detection and find that arbitrary natural images suffice for the task. Another body of work exploits generative adversarial networks (GANs) [Goodfellow
et al.... 2014a] to generate synthetic domains [Hoffman et al. 2017 Liu et al. 2017 Shrivastava et al.
2016 Taigman al. 2016 Tzeng et al. 2017]. particular, Shetty et al. [2019] use GAN trained
to perform in- paintinii and replace extraneous objects in images as a data augmenattion technique to reduce the trained model's dependence on co-occurring classes.
A different paradigm for the low-data regime is few-shot learning. In contrast to domain adaptation, the goal of few-shot learning is generalize to new classes given only a few examples, given the ability to train on large amounts of data containing related classes. Early approaches emphasized capturing knowledge in Bayesian framework (Fe-Fei al. 2003], which was later formulated as Bayesian program learning [Lake et al. 2015). J. Another approach based on metric learning is to find a nonlinear embedding for objects where closeness in the geometry of the embedding generalizes to unseen classes [Koch] 2015 Snell et al.] 2017 Sung et al. 2018 Vinyals et al. 2016). Meta-learning approaches aim to extract higher level concepts which can be applied to learn new classes from few examples [Finn et al. [017]] Munkhdalai and Yu! 2017 Nichol et al. 2018 Ravi and Larochelle, 2016). conceptually-reltte method that leverages synthetic training data is learning how to generate new data from few examples of unseen classes; in contrast to our work, however, these methods still require a large number of samples to Iearn the synthesizer [Schwartz et al. 2018 Zhang et al. 2019). Finally, some works combine domain adaptation with few-shot learning to learn under domain shift and limited samples Motiian et al. 2017al).
The main characteristic that differentiates our work from these approaches is that we are interested in learning classifiers that are contexi- aontext-agnottic, i.e., do not rely on background signals. As such, while we find our approach is applicable to many of the same tasks as the aforementioned works, our theoretical setting and objectives differ significantly. From a practical perspective, we demonstrate our techniques when the entire training sel consists solely of a single synthetic image of euch class, though our techniques can certainly be applied when more data is available; however the reverse does not hold, i.c., existing approaches for domain adaptation or few-shot learning cannot be applied to

---

setting. Indeed, we consider this work to be complementary in that we are concerned with exploiting the additional structure that is inherent in certain domains, while the goal of domain adaptation and few-shot learning is to achieve good performance on a downstream target distribution given data from a related source distribution.
3 Context-agnostic learning
In this section, we introduce the formal setting of context-agnostic learning. We begin with some notation from the standard supervised learning setting. We are given an input space X, an output space V. and hypothesis space H. of functions mapping X' to V. A domain Pp is probability distribution over (?). Given a target domain Pr and a loss function the goal to learn a classifier & E H that minimizes the risk, i.e., the expected loss R2y (h) E,, (e((()) 2)]. The training procedure is given as input a set of n training samples (I1, 31 11..8))..... (In: yn. drawn from a source domain Pg. The standard approach is empirical risk minimization, which takes the classifier that minimizes Remp(h)
1 E. (1;). y4). This technique is known to converge to the optimal classifier over Ps the number of samples increases; furthermore, if P3 is sufficiently "close" to Pr (e.g., if Ps Pr, as is the case when there is no domain adaptation), then such classifier also achieves low risk in the target domain.
3.1 Formal setting
In general, we can frame the goal of classification as learning to extract reliable signals for the label from points x E X. This task is often complicated by the presence of noise other spurious signals. However, for input spaces generated by physical processes, such signals are generally produced by distinct physical entities and can thus be thought of as independent signals that become mixed via an observation process. We aim to capture this additional structure in our setting.
Concretely, we have an object space O, a context space C, and an observation function y on O x C. The input space X' defined as the image of y O -> X. We will assume that points in O are associated with unique label in V, and points in X inherit labels from O via 7; in particular, require that y be injective with respect to labels, i.e., y always maps objects with different labels in O to distinct points in X.
In this work, we will consider the special case when X' S C. Conceptually, the context space is an "ambient space" containing not only valid inputs, but also random noise or irrelevant classes; the input space a subset of the context space for which there exists well-defined label. For example, in our experiments we explore such decomposition for the task of traffic sign recognition, where the object space O consists of traffic signs viewed from different angles, the context space is unconstrained pixel space, and the input space A' is the set of images that contain traffic sign.
Recall that the standard objective of learning is find a good classifier for an unknown subdomain Xp, C X. We consider instead the task of learning a classifier on the entire input space X. To sample from X' we are given oracle access the observation function and draw (labelled) samples from O and C' independently. Clearly, if this problem is realizable, i.c., there exists h" E H. for which Rx(h")
then we do not even need know the target domain Pr, since
Xp, X
[Rx)'') =
Rp, (h")
Hence our new goal will be learn classifier over X' which depends only on signals from O; more precisely, we have the following definitions:
Definition 3.1. A function f on is context-agnostic if
Pr[f ((0,e) v] Pr[f ((0,c) y]
Ve, E C, O,y Im(f)
(2)
Definition 3.2. The objective of context-agnostic learning is to find H. such that h achieves the lowest risk of all context-agnostic classifiers.
Remark. First, note that if the true label function y" realizeable, then the lowest risk classifier also context-agnostic. Second, we recover the standard supervised setting for the trivial context space 6. Conversely, our setting remains well-defined even in the trivial object space O = {W:} the set of classes; however, this pushes all the complexity to the observation function 7, which may be
3

---

hard to define or intractable to compute. Finally, we do not preclude the existence of useful signals originating from the context for certain domains. For instance, a great deal of information can often be gleaned from the backgrounds of photos, e.g., stop signs are more often found in cities than on highways. Our theoretical setting avoids this issue by assuming realizability and uniqueness labels; more practically, we argue that "good" classifier should nonetheless recognize stop signs on the highway, and our experimental results provide evidence that over-reliance on such background signals leads to brittle classifiers.
3.2 A new risk bound for the context-agnostic setting
Our central tool in the context-agnostic setting is new risk bound that decomposes into separate terms over the context and object spaces, We first develop a formal notion of contextual bias. For clarity we will assume binary classification task and slightly abuse notation, denoting the classifier as h instead of h o 7. i..., h : O x C + {-1,1}. We will denote the true label function as y"
Definition 3.3. For an object o E O the expected classification and object error â are defined as
ō = E (,fo,)( c)] ô (o) σ
(3)
(4)
Definition 3.4. The context bias B(h, c) of a classifier h on the context e defined as
sgn(B(h, c)) = sgn(Eono -) ō]) [B c)ll = E.-o [.-o(((((o.c),,) where E is the hinge loss E(i,j) max(0, 1 j)
(5)
(6)
Intuitively, the sign of the bias corresponds to the label toward which the classifier is biased by a given context, the magnitude measures the strength of this bias. Clearly, the classifier is context-agnostic exactly when the bias is zero. We are now ready to state our main theoretical result, which gives an upper bound on the risk in terms the context bias on and object error over O.
Theorem 3.1. Let h be a classifier with average bias K, and denote the true risk as R(h). Then the risk can be lower bounded as
K/2 S R(h)
(7)
with equality if and only if R(h) K Furthermore, if the object error for all objects is bounded from above by < 1, then we may also upper bound the risk as
R(h) K/(2-a) aa)
(8)
with equality if and only if all object errors equal a.
The proof is deferred to Appendix A Since < 1, we can further upper bound K/(2 aa) as K,
which yields the following concise corollary
K/2 S R(h) < <K
(9)
(under the same assumptions).
Remark, Both the learning objective and the standard definition of risk depend crucially on the true labels y"; then given the setting of context-agnostic learning, one might be tempted focus on minimizing some objective over the object space, which determines the labels exclusively. However from the theorem we see that measuring the context bias of a classifier gives bound on the risk that is tight up to constant factor of two without access to any labels. In fact, the dependence on the object space fairly weak in that the labels only enter in via the assumption a 1, which is achieved exactly as soon as the classifier outperforms random guessing. Note that the error bound and the bias bound K are not independent; the conclusion rather that the risk depends more strongly on good estimate of the bias. In particular, a = 0 if and only if K = 0 and a < 1; observe also that when C = then = 0 holds trivially, but in this case we can identify O with X, so the assumption that < 1 equivalently yields that the classifier is correct on all inputs.

---

Algorithm 1: Greedy Bias Correction
Input: Object space O, context space C, observation function 7, number of rounds R. resample probability classifier update subroutine Fit, binary classifier
Output: Trained classifier h
// initialize random context and label
c c C;
y ~ 1-1,]};
for r + 1 to do
0 O(y): // sample object
I 4- c(o,e);// observe object and context
h Fit(h, y): perform classifier update
// update context and label
p' Uniform(0. 1);
if p' < then
// resample random context and label
e ~ C:
y {-1,1};
else
c c x: // previous image becomes new context y + y, // flip label
end
end
Context-agnostic learning of visual tasks using synthetic data
We now present a pair of algorithms for the setting of context-agnostic learning. The first algorithm is generic approach to context agnostic learning that attempts to minimize the context bias in line with Theorem 3.1 The second algorithm is specialization of our framework to the visual domain, where contexts. Both algorithms assume
the observation function is given by superposition of objects unlimited independent samples from C and O, along with black-box access to the true labelling function h- and the observation function 7.
These assumptions allow us to learn h* simply by taking the number samples to infinity. Unfortu- nately, learning a classifier the entire input space X' generally requires many more samples than learning a classifier on a smaller target domain X',, Thus we should aim to learn using as few samples possible. Our main strategy will be exploit the priori knowledge that the true label function h² is context-agnostic and thus learn ht through the decomposed object and context spaces. To that end, note that while only need max(1O), [C]) samples to observe every object and context once, we need [O|[C) samples to observe every object in every context. Hence, the main challenge when the number of samples is low will be avoiding spurious signals, i.e., statistical correlations between context and objects (and by extension, labels) which are artifacts of the sampling process and do generalize outside the training set. This intuition corresponds the formal notion of contextual bias introduced in the previous section.
4.1 Greedy bias correction
The central idea behind Theorem is leveraging the fact that labels depend only on objects to factor the risk into separate terms for object eиror and context bias. This factorization enables us exploit our ability to sample independently from the object and context spaces. More specifically, we can use samples from O to minimize the object error, and samples from C to minimize the context bias. Since we only need a < 1 (i.e., any performance that exceeds random guessing), we continue to draw objects randomly; however given object o, aim to observe it with the context for which the classifier has the strongest opposing bias. Intuitively, this allows the classifier to "correct" its bias and unlearn the spurious signals, thereby minimizing the bias and also the risk.
Adopting this approach without modification requires computing the bias of every context in C. In most cases, however, even estimating single bias may be prohibitively expensive. Thus, rather than
5

---

(5)
(2b)
(2a)
(3)
(4)
Figure 1: A graphical representation of the generative loop in Algorithm using real training data. (1) Sample from object space, (2) Observe object and context. (3) Perform local refinement. (4) Add to training set. (5) Previous image becomes next context (resample from C with probability p).
solve for the maximum bias explicitly, we instead propose heuristic for identifying contexts with large biases. Note that since X' C C, reasonable assumption is that the classifier learns strong bias on recent training imputs when taken contexts. This suggests a simple greedy approach for correcting biases by repurposing recent training inputs as contexts; we call this algorithm Greedy Bias Correction and present a description in Algorithm 1
4.2 Learning visual tasks with synthetic data
We next introduce an instantiation of Greedy Bias Correction for learning visual tasks using synthetic data. We given function which takes label and outputs rendering of the corresponding class in random pose without any background. The context the background of the image, on which we place restrictions. The observation function y superimposes object
background.
Local refinement via robustness training We note that our observation function 7 is insufficient to capture the true range of possible inputs for given object; for instance, do not support occlusions. Because our ultimate goal will be to perform on data taken from a real-world context, we aim to capture this discrepancy using robustness training In particular, y is an e-covering of X, where a set A is said to be an e-covering of another set B iff for all points b E B, there exists point A such that I|a bll S 6. Then for given sample æ, will instead add the point in the e-neighborhood of a which maximizes the training loss, i.e., for a classifier h and sample ((, c). we instead aim identify
assume that the image of
argmax (((x'))))) z'EN((x)
(10)
This formulation is often used to train models which are robust against local perturbations. An empirically effective method for finding approximations to x' is known as Projected Gradient Descent (PGD) [Goodfellow al. 2014b Madry al. 2017]. The algorithm can be summarized as
30 + x +5
(11)
x; < (o (zi-1 + y sgn [_(((r:)))))))
1= 72
(12)
where 5 is a small amount of random noise, II is projection back onto the c-ball, is the step size, and n is the number of iterations. As is standard for robustness training, we take the e-ball with respect to the lo -nomm defined as (( Fn Il max; 2y. Our choice of E will depend on the task hand, and we also use different e for the portions of the image corresponding to the object and context.
Additionally, since we are no longer in binary context, we sample random permutation on labels instead of flipping the label deterministically. The full algorithm is presented as Algorithm 2 in Appendix B. Figure ! provides visualization of the key generative process, with images taken from a real step of training deep neural network to perform classification of traffic signs.
From practical standpoint, this algorithm makes concrete several benefits of our approach. First, rendering object classes, i.e. sampling from O, is often relatively easy. In the case of -wo-dimensional
Robustness training is more commonly referred to as adversarial training in the adversarial robustness community whence we borrow this technique. We use the nonstandard term to avoid confusion with the unrelated (generative) adversarial methods found in the few-shot learning literature.

---

30
70
3
88
4
5
6
9
*
/
2
4
6
7
6 9
Figure 2: Images from the training (top) and test (bottom) set for GTSRB (left) and MNIST (right).
Table I: Performance of Algorithm 2 on various benchmarks, plus ablation studies.
Approach
Picto GTSRB
Digit + MNIST
Omnifont Omniglot
baseline
72.0
81.9
71.9
random-context
72.1
88.3
69.8
+ refinementoonly
86.4
89.7
90.8
bias-correction
87.3
89.2
80.5
+ full
95.9
90.2
92.2
rigid body objects, this can be captured using standard data augmentation such as rotations, flips, and perspective distortions. Indeed, in this setting, work can be viewed form of minimal one-shot learning, where the training data consists solely of a single unobstructed straight-on shot for each object class. Second, since we allow the context space C to be unconstrained, there is requirement to perform realistic rendering of backgrounds, avoiding an additional layer of complexity.
Finally, because our approach is context agnostic, our classifiers are learned without any reference to target domains. the formal setting, we assumed that target domain was contained in the image of the observation function; however, synthetic images will always be subject to the reality gap. Our experiments suggest that our approach overcomes this barrier and successfully generalizes natural images while training synthetic data only.
5 Experiments
We evaluate our approach to learning visual tasks using synthetic data on three benchmarks for image recognition. Our training sets consist of single synthetic image for each object class with no additional information about the target domain; Figure 2 shows examples of the training and test images from of the datasets. On all three benchmarks, our models perform comparably with previous state-of-the-art results from related settings using few-shot learning and domain adaptation. Table | provides summary of results; comprehensive results and comparisons Appendix D Appendix C provides the full experimental setup and training details. Sample images from all datasets referenced below, including examples of rendered training data from the experiments and ablation studies, are shown in Appendix
compiled in
5.1 GTSRB
For the traffic sign recognition task, our training set consists of single, canonical pictogram of each class taken from the visualization software accompanying the target dataset, which we refer to as Picto. The target dataset is the German Traffic Sign Recognition Benchmark (GTSRB) (Stallkamp) ct al. 2012], which has 39,209 training and 12,630 test images of 43 classes of German traffic signs taken from the real world. We achieve 95.9% accuracy the GTSRB test set training only on Picto, against a human baseline of 98.8%. comprehensive comparison with existing approaches can be found in Appendix D Table
As a baseline, we also consider approaches using the SynSign Moiseev al. 2013 dataset of synthetic images designed to provide realistic training data for traffic sign recognition. The dataset comprises 100,000 synthetically generated images of signs from Sweden, Germany, and Belgium in variety of poses, rendered against comain-appropriate backgrounds (e.g. trees, roads, sky) taken from real-world images. The dataset contains a superset of the GTSRB classes; as result, Saito et al. 2017 report 79.2% accuracy by training directly on SynSign.
7

---

For domain adaptation, all approaches train on the full 100,000 images in SynSign plus part of the GTSRB training set. ATT [Saito et al. 2017 is the only method with better performance than ours, achieving 0.3% higher accuracy; however they use 31,367 unlabelled images from the GTSRB training set (in addition to SynSign). Methods using few-shot learning train on roughly half of the data (22 classes) from the GTSRB training set. The leading few-shot learning approach, VPE [Kim et al. 2019], adds a pictographic dataset similar to Picto, but achieves only 83.79% accuracy, In comparison, our training set consists of only 43 images, none of which are from GTSRB.
5.2 Handwritten character recognition
We consider two subtasks for handwritten character recognition. For the first subtask of classifying images of Arabic numerals, our training set, Digit, consists single example of each digit taken from standard digital font. The target dataset, MNIST [LeCun]. consists of 60,000 training and 10,000 test images of handwritten Arabic numerals in grayscale against a blank background. We achieve 90.2% accuracy on MNIST by training only on Digit, compared to human accuracy of 98%.
On MNIST, every approach using domain adaptation uses the full Street View House Numbers (SVHN) training set of 73,257 images of house numbers obtained from Google Street View [Netzer et al. 2. plus varying amounts of data from MNIST. The domain shift problem faces a similar challenge as Digit, namely, handwriting exhibits different characteristics than house numbers fonts. Nevertheless, we note that SVHN contains far more examples of each digit. The only non-baseline approach to exceed our performance is CyCADA [Hoffman et al. 2017]. which achieves 0.2% better performance by performing domain adaptation using 60,000 unlabelled images from the MNIST training set (in addition to training SVHN). In contrast, we use only 10 images, none of which are from MNIST.
For the second subtask, we use the Omniglot [akke et al. 2015 challenge, which consists of 1623 hand-written characters from 50 different alphabets, with 20 samples each. The samples were sourced online from 20 workers on Amazon's 's Mechanical Turk, who were asked to copy each character from a single font-based example using digital input (e.g., a mouse). We obtained the original representations (one per character) for our training images, which we call OmniFont. We achieve 92.2% on the 20-way Omniglot classification task training only on Omnifont, compared to human accuracy of 95.5%. Tables 3 and 4 in Appendix compare our results on the handwritten character tasks with approaches using few-shot learning and domain adaptation.
Omniglot is often described as an MNIST-transpose, where the goal is learn handwriting rather than specific symbols, and is widely used as a benchmark for few-shot learning. We reproduce the most common split given in Lake et al. [2015], which uses a predefined set of 30 alphabets, with 19,280 images for training. Test performance is reported as an average over random subsets of n = 5, 20 unseen classes for the n-way task (given one labelled example). In comparison, for each test run, we retrain a model using only the corresponding n images from OmniFont. As expected, our method finds 5-way classification easier than 20-way classification (95.8% vs 92.2%). In both cases, our performance lags behind the -fhth--rt for few-shot learning (>99%), ), though we emphasize that experimental setup differs significantly in both the type and amount of training data used.
Finally, several approaches apply few-shot learning from Omniglot to MNIST, with the idea of transferring extracted features from human handwriting. We hypothesize that in comparison to Omniglot, where all the samples come from the same 20 subjects, MNIST may be particularly difficult for transfer one-shot learning, since any two examples will likely exhibit high "variance"; conversely, our approach benefits from using a canonical form which might be closer to the "mean" representation.
Remark. Handwritten characters and GTSRB present conceptually opposed challenges for learning: in GTSRB, the objects are rigid two-dimensional objects and backgrounds are complex settings in the natural world; Omniglot and MNIST, backgrounds are uniform, but classes no longer have a strict specification and individual examples exhibit high variability. Thus, the main challenge of these tasks is learning how to generalize over the object class. Despite the inherent variation, a baseline model trained on Digit with plain data augmentation was able to achieve 81.9% accuracy on MNIST, exceeding many domain adaptation approaches and all the one-shot learning results; Omniglot more difficult, with an Omnifont plus data augmentation baseline accuracy of 71.9%.
8

---

100
75
Accuracy (%)
50
25
odlaenent-dly
0
0
16
64
255
0
16
64
255
a) Random initialization (c/255)
Bias heuristic (c/255)
Figure 3: Context-agnostic performance Picto using a PGD adversary on the background.
5.3
Ablation studies
We conduct two sets of ablation studies to better understand approach to context- aonnext-agnostic learning. The first study tests the individual components of our algorithm for their contributions to generalization over the real world dataset. All strategies employ the same data augmentation and use the following sampling procedures: baseline picks a fresh random background for each training point, and measures the performance of training random-context reuses random backgrounds as contexts; bias-correction reuses previous training images contexts; refinement-only the same random-context with the addition of PGD-based refinement; full is the full algorithm as described in Algorithm 2 The results are in Table |
synthetic dataset with plain data augmentation;
In all cases, we observe that both bias correction and local refinement contribute individually and jointly to the performance of our models. For GTSRB, a particularly interesting comparison is training SynSign, dataset designed to provide synthetic training data with realistic backgrounds for GTSRB, which yields 79.2% accuracy (Saito et al.) 2017]. Though this is an improvement over baseline of using random backgrounds at 72.0% accuracy, refinement-only and bias-correction achieve higher accuracy at 86.4% and 87.3%, respectively. Both methods leverage the background of training images to combat spurious signals, generating completely unrealistic backgrounds; this suggests that learning context-agnostic features is more effective than using realistic backgrounds.
The second study measures classification performance in context-agnosic setting the synthetic Picto dataset. By definition, the performance of a context-agnostic classifier should not degrade under perturbations of the background. We thus the foreground pixels, and ranges from fixed to unbounded on the background pixels, effectively searching the context space for background that causes misclassification the given object. We also consider two initialization strategies for the PGD adversary: a standard random initialization, and initializing the previous image, inspired by our bias heuristic.
adaptive attack using PGD adversary which fixes
We evaluate the same set of ablated strategies as before, plus a classifier trained directly on the GTSRB training set (discussed in the following section). Appendix 1 contains samples of the generated images, and results are plotted Figure Across all experiments, the models have worse (or very close) performance when using our bias heuristic for initialization. We believe this supports our usage of the bias heuristic for context-ag learning. Additionally, in the last column of Figure 3b, only full method maintains passable accuracy, which suggests the gap between models larger than performance on the GTSRB test set would indicate.
5.4 Comparison with training on real data
In this section, we compare classifier trained using our method on only synthetic data with model of the same architecture but trained directly on the GTSRB training set and achieving 98% performance on the GTSRB test set, which we refer to as real2sim. We first note that Figure B indicates the real2sim method seems to suffer from "synthetic gap" even at = 0/255, which not entirely unexpected. However, in both settings, the performance of the real2sim model degrades very

---

80
20
30
70
STOP
Figure 4: Test images from the second ablation study using the Picto dataset. Examples of test images generated using a PGD adversary initialized randomly (top) and with the bias heuristic (bottom) at e 255/255.. Note that only backgrounds are perturbed, while foregrounds remain unambiguous.
<E
Figure 5:
Guided Grad-CAM visualizations for the full (left) and real2sim (right) methods on the GTSRB test set. Misclassified images are marked with red boxes.
quickly as e increases: the effect is most pronounced when the bias heuristic is used to initialize the PGD adversary, though in both cases the accuracy eventually drops to O. We emphasize that all of the experiments leave the foreground objects completely unperturbed (and easily human-identfiablle;; Figure $ presents examples of the generated test images.
To better understand the differences between a full classifier trained using our method and the real2sim classifier trained on real data, used the Guided BraddCAMM method (Selvaraju et al., 2019], which localizes regions of fine-grained features that are important to the classifier's 's output. Figure 5 presents visualizations for the real2sim and full classifiers
images from the GTSRB test set. We see that the real2sim classifier (on the right) trained on real images of traffic signs has more diffuse activation map with clear interpretation, whereas the classifier trained using our method using purely synthetic data (on the left) is more focused, with more semantically-aligeee features. Our results thus suggest that classifiers trained on natural images become over-reliant on contextual signals, leading to surprisingly brittle behavior even given unambiguous foregrounds.
6 Conclusion
We introduce the task of context-agnostic learning, a theoretical setting for learning models whose predictions are independent of background signals. Leveraging the ability to sample objects and contexts independently. we propose an approach to context-agnostic learning by minimizing a formally defined notion of context bias. Our algorithm has natural interpretation for training classifiers on vision-based tasks using synthetic data, with the distinct advantage that we do not need to model the background. We evaluate our methods on several real-world domains; our results suggest that our approach succeeds in learning context-agnostic classifiers that generalize to natural images using only a single synthetic image of each class, while training with natural images can lead to brittleness in the context-agnostic setting. Our performance is competitive with existing methods for learning when data is limited, while using significantly less data. More broadly, the ability to learn from single synthetic examples of each class also affords fine- grained control over the data used to train our models, allowing us to sidestep issues of data provenance and integrity entirely.
10

---

Acknowledgments and Disclosure of Funding
We gratefully acknowledge support from DARPA Grant HR001 120C0015. The views expressed those of the authors and do not reflect the official policy or position of the Department of Defense or the U.S. Government.
References
Antreas Antoniou, Harrison Edwards, and Amos J. Storkey. How to train your MAML. CoRR, abs/1810.09502, 2018. URL http://arxiv.. http://...rgbbb/////eee 1810. 09502
Mahsa Baktashmotlagh, Mehrtash Harandi, Brian C Lovell, and Mathieu Salzmann. Unsupervised domain adaptation by domain invariant projection. In Proceedings of the IEEE International Conference on Computer Vision, pages 769-776, 2013.
John Blitzer, Ryan McDonald, and Fernando Pereira. Domain adaptation with structural correspon- dence learning. In Proceedings of the 2006 conference on empirical methods in natural language processing, pages 120-128, 2006.
Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishman, and Dumitru Erhan. Domain separation networks. In Advances in neural information processing systems, pages 343-351, 2016.
G. Bradski. The OpenCV Library. Dr. Dobb's Journal of Software Tools, 2000.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In International conference on machine learning, pages 647-655, 2014.
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. arXiv preprint arXiv: 2017.
Li Fe-Fei et al. bayesian approach to unsupervised one-shot learning of object categories. In Proceedings Ninth IEEE International Conference on Computer Vision, pages 1134-1141. IEEE, 2003.
Basura Fernando, Amaury Habrard, Mare Sebban, and Tinne Tuytelaars. Unsupervised visual domain adaptation using subspace alignment. In Proceedings of the IEEE international conference on computer vision, pages 2960-2967, 2013.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference Machine Learning-Voin Volume 70, pages 1126-1135. JMLR. org, 2017.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. arXiv preprint /409.7 2014.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The Journal of Machine Learning Research, 17(1):2096 2006622000,,0 2016.
Muhammad Ghifary, W. Bastiaan Kleijn, Mengjie Zhang, David Balduzzi, and Wen Li. Deep
reconstruction-Classffeee
networks for unsupervised domain adaptation, 2016.
Jacob Gildenblat and contributors. Pytorch library for cam methods.
https://github.co com/
jacobgil/pytoron granod---rrd-aarra 2021.
Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 2066-2073. IEEE, 2012.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural informa- tion processing systems, pages 2672-2680, 2014a.

---

Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014b.
Raghuraman Gopalan, Ruonan Li, and Rama Chellappa. Domain adaptation for object recognition: An unsupervised approach. In 20// international conference on computer vision, pages 999-1006. IEEE, 2011.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun- Yan Zhu, Phillip Isola, Kate Saenko, Alexei A Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. arXiv preprint arXiv:171..02215 2017.
Nick Jakobi, Phil Husbands, and Inman Harvey, Noise and the reality gap: The use of simulation in evolutionary robotics. In European Conference on Artificial Life, pages 704-720. Springer, 1995.
Matthew Johnson- Charles Barto, Rounak Mehta, Sharath Nittur Sridhar, Karl Rosaen, and Ram Vasudevan. Driving in the matrix: Can virtual worlds replace numan-generann annotations for real world tasks? arXiv preprini 1r10..1600.099. 2016.
Junsik Kim, Seokju Lee, Tae-Hyun Oh, and So Kweon. Co-domain embedding using deep quadruplet networks for unseen traffic sign recognition. In Thirty-Second AAAI Conference Artificial Intelligence, 2018.
Junsik Kim, Tae-Hyun Oh, Seokju Lee, Fei Pan, and In So Kweon, Variational prototyping-encoder: One-shot learning with prototypical images. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 9462-9470, 2019.
Gregory Koch. Siamese neural networks for one-shot image recognition. 2015.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350K6266) 2015.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/.
Aoxue Li, Tiange Luo, Tao Xiang, Weiran Huang, and Liwei Wang. Few-shot learning with global class representations. In Proceedings of the IEEE International Conference on Computer Vision, pages 9715-9724, 2019.
Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. In Advances in neural information processing systems, pages 700-708, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 1706.06088 2017.
Boris Moiseev, Artem Konev, Alexander Chigorin, and Anton Konushin. Evaluation of traffic sign recognition methods trained on synthetically generated data. In International Conference on Advanced Concepts for Intelligent Vision Systems, pages 576-583. Springer, 2013.
Saeid Motiian, Quinn Jones, Seyed Iranmanesh, and Gianfranco Doretto. Few-shot adversarial domain adaptation. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 6670-6680. Curran Associates, Inc., 2017a. URL http::/papeer. nips cctpaper/ shot- - -ww.--ddaaaaaaaaatttiiooo pdf
Saeid Motiaan, Marco Piccirilli, Donald A Adjeroh, and Gianfranco Doretto. Unified deep supervised domain adaptation and generalization. In Proceedings of the IEEE International Conference on Computer Vision, pages 5715-5725, 2017b.
Tsendsuren Munkhdalai and Hong Yu. Meta networks. Proceedings of the 34th International Conference Machine Learning- Volume 70, pages 2554-2563, JMLR. org, 2017.
12

---

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv: 1803. 2018.
Sinno Jialin Pan, Ivor W Tsang. James T Kwok, and Qiang Yang. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks, 22(2):199-2100 2010.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-perfornance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024-8035, Curran Associates, Inc., 2019. URL http::/papprrr.aapppm//// http:/pppprr....uippppppp. www.ipe.rrccomm pttpurippcccpppppe... library....
an aantoorchham-impratiiiin htw.://ppaaaaateeeeeeeeeeerrrrrr
pdf
Zhongyi Pei, Zhangjie Cao, Mingsheng Long, and Jianmin Wang. Multi-adversarial domain adapta- tion. In Thirty-Secone AAAI Conference on Artificial Intelligence, 2018.
Weichao Qiu and Alan Yuille. Unrealev: Connecting computer vision unreal engine. In European Conference on Computer Vision, pages 909-916. Springer, 2016.
Tiago Ramalho and Marta Garnelo. Adaptive posterior learning: few-shot learning with a surprise- based memory module, 2019.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2016.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European conference on computer vision, pages 213-226. Springer, 2010.
Kuniaki Saito, Yoshitaka Ushiku, and Tatsuya Harada. Asymmetric tri-training for unsupervised domain adaptation. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 2988-2997. JMLR. org, 2017.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory-augmented neural networks. In International conference machine learning, pages 1842-1850, 2016.
Alice Schoenauer-Sehag Louise Heinrich, Mare Schoenauer, Michele Sehag, Lani F. Wu, and Steve J. Altschuler. Multi-domain adversarial learning, 2019.
Eli Schwartz, Leonid Karlinsky, Joseph Shtok, Sivan Harary, Mattias Marder, Rogerio Feris, Abhishek Kumar, Raja Giryes, and Alex M. Bronstein. Delta-encoder: an effective sample synthesis method for few-shot object recognition, 2018
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- based localiza- tion. International Journal of Computer Vision, 128(2):33--359, Oct 2019. ISSN 1573-1405. doi: 10. 0012260--9--12---777 URL
doi
Rakshith Shetty, Bernt Schiele, and Mario Fritz. Not using the car to see the sidewalk-quaaniffying and controlling the effects of context in classification and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8218-8226, 2019.
Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Josh Susskind, Wenda Wang, and Russ Webb. Learning from simulated and unsupervised images through adversarial training, 2016.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, pages 4077-4087, 2017.
13

---

J. Stallkamp, M. Schlipsing, J. Salmen, and C. Igel. Man vs, computer: Benchmarking machine learning algorithms for traffic sign recogmition. Neural Nenworks, (0): 2012. ISSN 0893-6080. doi: 10. 00.100666.eeuuet22222.......1 URL http: htttpp//ww..cienccdiree..coom http://wew..eecceccccccererrriia p:1/$08936081200045577
Flood Sung, Yongxin Yang. Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1199-1208, 2018.
Yaniv Taigman, Adam Polyak, and Lior Wolf. Unsupervised cross-domain image generation. arXiv preprint arXiv:161 1.02200, 2016.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 23-30 IEEE, 2017.
Lucas Tabelini Torres, Thiago M. Paixão, Rodrigo F. Berriel, Alberto F. De Souza, Claudine Badue, Nicu Sebe, and Thiago Oliveira-Santos, Effortless deep training for traffic sign detection using templates and arbitrary natural images, 2019.
Jonathan Tremblay, Aayush Prakash, David Acuna, Mark Brophy, Varun Jampani, Cem Anil, Thang To, Eric Cameracci, Shaad Boochoon, and Stan Birchfield. Training deep networks with synthetic data: Bridging the reality gap by domain randomization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 969-977, 2018.
Eric Tzeng, Judy Hoffman, Trevor Darrell, and Kate Saenko. Simultancous deep transfer across domains and tasks. In Proceedings of the IEEE International Conference on Computer Vision, pages 4068 4076, 2015.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 7167-7176, 2017.
Stéfan van der Walt, Johannes L. Schönberger. Juan Nunez-Iglesias, François Boulogne, Joshua D. Warner, Neil Yager, Emmanuelle Gouillart, Tony Yu, and the scikit-image contributors. scikit- image: image processing in Python. PeerJ, 2:3453, 6 2014. ISSN 2167-8359. doi: 10.7717/peerj. 453. https://///// ////1100771//perr// .43
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pages 3630-3638, 2016.
Ting Yao, Yingwei Pan, Chong Wah Ngo, Houqiang Li, and Tao Mei. Semi-supervised domain adaptation with subspace learning for visual recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2142-2150, 2015.
Sung Whan Yoon, Jun Seo, and Jackyun Moon. Tapnet: Neural network augmented with task-adaptive projection for fhw-shot learning, 2019.
Jian Zhang, Chenglong Zhao, Bingbing Ni, Minghao Xu, and Xiaokang Yang. Variational few-shot learning. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
14


Reviews:
[{"review_id": "ZwlLDylKoyR", "reviewer": "Reviewer_gTVT", "summary": "This paper proposes a data augmentation scheme that synthesizes image background so that models can achieve better generalization by learning from one synthetic image and generalizing to real natural images. The process involves using previous images as background and adding adversarial noises. Experimental results show that it can generalize from synthetic traffic signs and digit images to real traffic sign images and handwritten digit images, outperforming a baseline that directly trains from the synthetic images without augmentations or with random background augmentations.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "-------------------------------------------------------\n### Strengths\n\n- **Problem setup:** Recognizing new concepts with one synthetic image is an interesting problem to study.\n\n- **Good performance:** The proposed method achieves good performance on the proposed benchmarks.\n\n- **Ablations:** The set of baselines is helpful for studying the effect of each component of the algorithm.\n\n\n-------------------------------------------------------\n### Weaknesses\n\n- **Clarity:** It would be good to do some reorganization to make the paper clearer. Table-1 shows up the first but the description does not show up until section 5.3, and I was searching for the definition for bias correction. It seems like this term should show up in the model section too for defining what is bias correction. It is also not clear what does the “random initialization” or “bias heuristic” mean in Figure 3, and what is the horizontal axis.\n\n- **Additional assumption:** The model assumes the knowledge of how to separate foreground and background in an image and explicitly synthesize augmented background to make it context-agnostic. Such information may not be available to some data and will require additional segmentation labels.\n\n- **Lack of literature review:** There are some papers in the computer vision literature that study the contextual effect for classification, detection and segmentation, that can be probably mentioned. E.g.\n[1] Shetty et al. Not Using the Car to See the Sidewalk -- Quantifying and Controlling the Effects of Context in Classification and Segmentation. CVPR 2019.\n[2] Ntavelis et al. SESAME: Semantic Editing of Scenes by Adding, Manipulating or Erasing Objects. ECCV 2020.\n\n- **More realistic benchmarks:** Currently, all the training images are synthetic. I wonder if such a method can be applied to more realistic benchmarks, such as the WILDS benchmark (Koh et al., 2021)?\n\n- **Effect of representation pretraining:** It would be good to study a baseline that pretrains its representation from a large dataset such as ImageNet and then performs a transfer learning in this one-shot setting. Such a paradigm is very commonly used in few-shot learning these days. It can provide a useful representation to start with, instead of training from scratch from the one-shot dataset, which might lead to degeneracy.\n\n- **IID test accuracy:** One drawback of the current design of the baselines is that the number of examples is perhaps too few for the network to learn a good hypothesis for the original distribution, let alone to learn a context-agnostic hypothesis for the new data distribution. So for example, what if the test set is not realistic images, but other traffic sign synthetic images with different transformations of the foreground only? Does the baseline perform well on these test images with just a blank background?\n\n- **Upper bound performance:** What would be a performance upper bound, of using all the natural images to train the model? It would be good to set up such an experiment to understand the gap in between.\n\n- **Random natural image background:** Instead of using the proposed procedure, another baseline to test is to use a random natural image (e.g. from ImageNet) as background. Although it uses more data, it is a very practical baseline method that can be easily adopted.\n\n-------------------------------------------------------\n### Questions for clarification\n\n- **Foreground augmentation:** In Figure 1, it seems that the full model samples from the “object space” in step 1, which performs a certain level of foreground augmentation. Is this procedure also included in the baseline? If not, which of the baselines in Table 1 includes foreground augmentation? This seems like a very useful procedure and may introduce significant improvement over just training on one sample. This comment, combined with the previous comment, is trying to clarify the individual contribution of the context augmentation and foreground augmentation, and whether it is merely a question of the number of examples. If the “random context” does not have foreground augmentation, then it suggests that the foreground augmentation is playing a bigger role here. If it does have foreground augmentation, then I would suggest revising the text to make the augmentation procedure clearer.\n\n- **Baseline vs. random:** It seems that both baseline and random use random background. Then what are the main differences between the two and what is the purpose of the random baseline?\n\n-------------------------------------------------------\n### Conclusion\n\nIn conclusion, this paper studies an interesting problem and proposes a simple and working method. However, since it compares to only baselines it creates and evaluates on synthetic benchmarks it proposes, I have to apply some extra scrutiny here, to make sure that the proposed method is widely applicable and robust, instead of just relying on some strict assumptions. In the above text, I mentioned a few more experiments and baselines that will help getting a more thorough picture on the empirical side. In addition, the clarity and literature review parts of the paper can also be improved.\n\n-------------------------------------------------------\n### Update post-rebuttal\nI thank the authors for their response to my comments. The rebuttal has addressed some of my concerns regarding iid accuracy and the foreground augmentation, and therefore I have increased my rating from 5 to 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "X7_DUkq0NM", "reviewer": "Reviewer_XVQW", "summary": "The authors mathematically show that the risk of a classifier is bounded by K / (2 - \\alpha) if the object error of the classifier is bounded by \\alpha. This means that decreasing K (context bias) can improve the performance. Inspired by this and by assuming that we have access to a function that generates an image based on object O and context C, the authors propose an algorithm that is context-agnostic. Their algorithm updates the model based on generated image and then uses this new image to generate the context for the next image. This way, the classifier is enforced to ignore the context. They evaluate their method on three different datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is an innovative work, and the mathematical proof of showing that classifier accuracy is bounded by context bias is new and very interesting. The Proposed method on object recognition inspired by this mathematical proof is also novel. The authors study the related work and cover a reasonable number of related studies. They discuss few-shot learning and domain shift. I would suggest that the authors try to cite a couple of more related works that look at both domain shift and few-shot learning. Currently, they only cite one related work[1], but I suspect that there is just one work that studies both domain shift and few-shot learning. The paper is technically sound, and claims are supported in experimental results to a good degree. However, The authors could do experiments on more datasets like the Office dataset that is used by the related work they mention[1]. It is interesting to see if it is possible to extend this work to other objects such as chairs or monitors? Because it is not as easy to apply transformations on these objects as it is on characters of MNIST or Omniglot or traffic signs. For example, the authors apply blur or threshold transformation on images. However, chairs have different colours/shapes. Can the proposed algorithm learn that the colour belongs to the context? In addition, the authors can visualize the network activation on a given image from MNIST, GTSRB or Omniglot. This visualization can show how the network's decision comes from the pixels that belong to the object rather than pixels that belong to the context. I would increase my rating if these experiments or visualizations were added to the paper. Overall, I think the paper has substantial contributions and recommend a score of 6. In addition, The paper is well-structured and easy to follow. The idea is interesting and can be useful in different scenarios. This paper opens some interesting directions for learning classifiers and object detectors such that we can generalize better to domain shifts. This can help us advance the state-of-the-art by focusing on how we can recognize objects or train context-free classifiers on larger datasets. By itself, this work still is useful by some practitioners. For example, self-driving cars need to have a very robust context-free classifier to recognize road signs. \nFinally, in line 100: Can you elaborate more? How can we generalize this to a case when the image \\gamma is a subdomain of X?\n\n=======Post Rebuttal=========\nI found the authors' responses convincing and strongly suggest modifying the manuscript to include the new suggestions by the reviewers. As a result, I will increase my score to 7 since I think the community can benefit from this work.\n\n[1]: Motiian S, Jones Q, Iranmanesh S, Doretto G. Few-Shot Adversarial Domain Adaptation. Advances in Neural Information Processing Systems.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Maybe the authors can add a paragraph and describe what could be the good and bad impacts of having classifiers or object detectors that generalize to context shifts. This could be beneficial for many research areas like robotics, self-driving cars, etc. It would be good to also study negative impacts and limitations. For example, the authors can describe what are the challenges when we face with more difficult datasets and what do they think are the way to overcome those challenges.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "Fulv15E_Orw", "reviewer": "Reviewer_PrQD", "summary": "The submission proposes a motified setting where the task is context-agnostic learning and proposes an algorithm to solve the problem while assuming the ability to sample objects and contexts independently. The method achieves high accuracy on two synthetic visual tasks, digits and traffic sign classification, when a model is trained using one sample per class from the source domain and tested on an unseen target domain.\n\n", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My main concern is experiments.  Although the method shows promising results under this specific setting where only one image is needed, it is hard to conclude that the proposed method will generalize other settings, such as when there are more classes, more images per classes (since one image per class is a very extreme setting), and other more complicated images (currently seems the images are simple and objects always sits in the middle). \n\nI think evaluation on additional datasets would be necessary. It would be interesting to know if the proposed method could use more than one sample in order to make the comparison with SoA methods fair, while assessing performance in comparative terms with Sota, to give value to the method also in relation to performance. \n\nSynthetic images are small iconic images with objects in the center. Although the method shows promising results under this specific setting, more datasets can have stronger support. It would also be more solid if there's more comparison with the generation based method  as described in related works A conceptually-related method that leverages synthetic training data is learning how to generate new\ndata from a few examples of unseen classes...\n\nAlso, I don't see many improvements compared to the last submission.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "NA", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a data augmentation scheme that synthesizes image background so that models can achieve better generalization by learning from one synthetic image and generalizing to real natural images.

All reviewers recommended to accept.

Accept.
Decision: Accept (Poster)