Context-Agnostic Learning Using Synthetic Data
Authors: Charles Jin
Martin Rinard
Venue:  (2021)
Keywords: machine learning
synthetic data
few-shot learning
domain adaptation

Abstract: We propose a novel setting for learning, where the input domain is the image of a map defined on the product of two sets, one of which completely determines the labels. Given the ability to sample from each set independently, we present an algorithm that learns a classifier over the input domain more efficiently than sampling from the input domain directly. We apply this setting to visual classification tasks, where our approach enables us to train classifiers on datasets that consist entirely of a single example of each class. On several standard benchmarks for real-world image classification, our approach achieves performance competitive with state-of-the-art results from the few-shot learning and domain transfer literature, while using significantly less data.

Under review as a conference paper at ICLR 2021
CONTEXT-AGNOSTH LEARNING USING SYNTHETIC DATA
Anonymous authors Paper under double-blind review
ABSTRACT
We propose a novel setting for learning, where the input domain is the image map defined the product of two sets, one of which completely determines the labels. Given the ability to sample from each set independently, we present algorithm that learns classifier over the input domain more efficiently than sampling from the input domain directly. We apply this setting to visual classi- fication tasks, where our approach enables us to train classifiers on datasets that consist entirely of a single example of each class, On several standard benchmarks for real-world image classification, our approach achieves performance compet- itive with state-of-the-ae results from the few-shot learning and domain transfer literature, while using significantly less data.
INTRODUCTION
Despite recent advances in deep learning, one central challenge is the large amount of labelled train- ing data required to achieve state-of-the-ar performance. Procuring such volumes of high quality, reliably annotated data can be costly or even close to impossible (e.g., obtaining data to train an autonomous navigation system for lunar probe). Additional hurdles include hidden biases in large datasets (Toomaai et al. 2017) and maliciously perturbed training data (Biggio et al. 2012).
Synthetically generated data has seen growing adoption in response these problems, since the marginal cost of producing new training data is generally very low, and one has full control over the generation process. This is particularly true for applications with a physical component, such as autonomous navigation (Gaidon et al., 2016) or robotics (Todorov et al. 2012). However, training with purely synthetic data suffers from the so-called "reality gap", whereby good performance on synthetic data does not necessarily yield good performance in the real world (Jakobi et al.) 1995). In particular, the difficulty of generating realistic training images scales not just with the objects of interest, but also the real-world contexts in which the learned model is expected to operate.
This work begins with the simple observation that, for many classification tasks, the label of input is determined entirely by the object; however, this additional structure is discarded by current syn- thetic data pipelines. Our goal is to leverage this decomposition to develop more efficient methods for the related problems of generating training data and learning from synthetic domain.
Our contributions are two-fold: where the input space is decomposed into object and context spaces, and the labels are independent of contexts when conditioned on the objects. Second, we propose an algorithm to efficiently train classifier in the context-agnostic setting, which relies the ability sample from the object and context spaces independently. We apply our methods to train deep neural networks for real-world image classification using only a single synthetic example of each class, obtaining performance com- parable to existing methods for domain adaptation and few-shot learning while using substantially less data. Our results show that possible train classifiers in the absence of any contextual training data that nonetheless generalize to real world domains.
first, we formally introduce the setting of context-agnostic learning,
2 RELATED WORK
Domain shift refers to the problem that occurs when the training set (source domain) and test set (target domain) drawn from different distributions. In this setting, classifier which performs

---

Under review as a conference paper at ICLR 2021
well on the source domain may not generalize well in the target domain. A standard method for addressing this challenge is domain adaptation, which leverages a small amount of data from the target domain to adapt function that is learned over the source domain (Blitzer et al. 2006).
In the context of learning from synthetic data, the domain shift that occurs between synthetic and real world data is known the reality gap (Jakobi al. 1995). State-of- -ateeoo--te-rrtttt rendering engines, such as those used for video games, can help narrow this gap by generating photorealistic data for training et al. 2016 Qiu and Yuille| 2016). Another technique
Dossvvisskyy et al. 2017 Bochnson-Roberee is using domain randomization to generate the source domain with more variability than is expected in the target domain (e.g., extreme lighting conditions and camera angles). so as to make real images appear as just another variant (Tobin et al. [2177] (Trmblay et al.] 2018); ) in particular, (Torres et al. (2019) apply domain randomization to traffic sign detection and hind that arbitrary natural images suffice for the task. Another body of work exploits generative adversarial networks (Goodfellow jet al., 2014a) to generate synthetic domains (Hoffman et al., 2017, Liu et al., 2017] [hhrivattaaa et al. 2016; Taigman et al.) 2016; Tzeng et al. 2017). Finally, several works have explored using synthetic data for natural image text recognition (Gupla et al. 2016 Jaderberg et al. 2014). These works an approach that is roughly analogous our baseline models, and test their techniques on the target domain of street signs rather than handwritten characters (as we do).
A different paradigm for the low-data regime is few-shot learning. In contrast to domain adapta- tion, few-shot learning operates under the assumption that the target and source distributions are the same, but the ability to sample certain classes is limited in the source domain. Early approaches emphasized capturing knowledge in Bayesian framework (Fe-Fei et al.) (003)), which was later formulated as Bayesian program learning (Lake al. 2015). Another approach based on metric learning is to find a nonlinear embedding for objects where closeness in the geometry of the embed- ding generalizes unseen classes (Koch) 2015 Snell et al. 2017: Sung et al. 2018; Vinyals al.] 2016). Meta-learning approaches aim to extract higher level concepts which can be applied to learn new classes from a few examples (Finn et al., 2017 Munkhdalai and Yu 2017 Nichol et al. 2018 Ravi and Larochelle] 2016). A conceptually-relee raleedd method that leverages synthetic training data is learning how to generate new data from a few examples of unseen classes; in contrast to our work, however, these methods still require a large number of samples to learn the synthesizer (Schwartz ct al. 2018: Zhang et al. 2019). Finally, some works combine domain adaptation with few-shot learning to learn under domain shift and limited samples (Motiaa et al. (2017).
The main characteristic that differentiates our work from these approaches is that we are interested in learning classifiers that are context-agnostic, i.e., do not rely on background signals. As such, while we find our approach is applicable to many of the same tasks as the aforementioned works, our theoretical setting and objectives differ significantly. From a practical perspective, we demonstrate our techniques when the entire training set consists solely of a single synthetic image of each class, though our techniques can certainly be applied when more data is available; however we do not expect the reverse to hold for domain adaptation or few-shot learning in our setting, Indeed, we consider this work to be complementary structure that is inherent in certain source domains, while the goal of domain adaptation and few- shot learning is to achieve good performance under various downstream domain shift assumptions.
that we are concerned with exploiting the additional
SETTING
The standard supervised learning setting consists of an input space X, an output space V. and a hypothesis space H of functions mapping X y. A domain PD is probability distribution over (X' )). Given a target domain Pr and a loss function f, the goal is to learn a classifier h E H that minimizes the risk, i.c., the expected loss Rp, (h) == [e,[[((h [((h(z), ))), v)]. The training procedure con- sists of n samples (e1,)1),.... (In, Vn.) from a source domain Ps. A standard approach is empirical risk minimization, which takes the classifier that minimizes Remp (h) = 1
a Σ, E(h(x;); v1); if Ps is close to Pr. then with enough samples, such a classifier also achieves low risk in the target domain.
3.1 CONTEXT-AGNOSTIC LEARNING
In general, we can frame the goal of classification as learning to extract reliable signals for the label from points & E X. This task is often complicated by the presence of noise other spurious signals.
2

---

Under review as a conference paper at ICLR 2021
However, for input spaces generated by physical processes, such signals generally produced by distinct physical entities and can thus be thought of as independent signals that become mixed via the observation process. We aim to capture this additional structure in setting.
Concretely, we have an object space O, a context space C, and an observation function y on x C. The input space X' is defined as the image of y O C + X. We will assume that points in O are associated with a unique label in y, and require that preserves this property when passing to X. Note that this setting can be easily generalized a case when the image of is a subdomain of X.
In this work, we will consider the special case when C C. Conceptually, the context space is an "ambient space" containing not only valid inputs, but also random noise or irrelevant classes; the input space is subset of the context space for which there exists a well-defined label. For example, in our experiments we explore such decomposition for the task of traffic sign recognition, where the object space O consists of traffic signs viewed from different angles, the context space C is unconstrained pixel space, and the input space X' the set of images that contain traffic sign.
Recall that the standard objective of learning is to find a good classifier for an unknown subdomain Xp, S X. We consider instead the task of learning a classifier on the entire input space X. To sample from X' we are given oracle access to the observation function and draw (labelled) samples from O and C independently. Clearly, if this problem is realizable, i.e., there exists h² E H for which Rx (h*) = O, then do not even need to know the target domain Pr, since
Xpr X [Rx(h')
(h*) =
Assuming access through y, we can learn h simply by taking the number of samples to infinity. Unfortunately. learning a classifier on X' generally requires many more samples than learning classifier on X'y,- Thus we aim to learn h" using as few samples as possible.
Our new goal will be to learn a classifier over X' which depends only on signals from O more precisely, we have the following definitions:
Definition 3.1. A function f on is context-agnostic if
Pr[s o ro = x] Pr(s x(, =
Ve,d EC, Im(J)
Definition 3.2. Given a context-agnostic label function yy, the objective of context-agnostic learn- ing is to find h E H. such that l achieves the lowest risk of all context-agnostic classifiers.
The hope is that, since y" is context-agnostic, we can learn y" through the lower dimensional struc- ture of using fewer samples. Note, however, that while we only need max( (C]) samples to observe every object and context once, we need |O| |C] samples to observe every object in every context. Hence the main challenge when the number samples is low will be avoiding spurious signals, i.e., statistical correlations between context and objects (and by extension, labels) which are artifacts of the sampling process and do not generalize outside the training set.
We conclude with some high-level remarks about this setting. First, note that if the problem is realizeable, then the lowest risk classifier also context-agnostic. Second, we recover the standard supervised setting for the trivial context space C (1 Conversely, classification remains well- defined even in the trivial object space O { the set of classes; however, this pushes all the complexity to the observation function 7, which may be hard define or intractable to compute. Finally, we do not preclude the existence of useful signals originating from the context for certain domains. For instance, great deal of information can often gleaned from the backgrounds of photos, e.g., stop signs are more often found in cities than on highways. Our theoretical setting avoids this issue by assuming realizability and uniqueness of labels; more practically, we argue that "good" classifier should nonetheless recognize stop signs the highway, and our experimental results provide evidence that over-reliance on such background signals leads to brittle classifiers.
3.2 EFFICIENT SAMPLING FOR OBJECT-CONTEXT DECOMPOSED INPUT SPACES
In this section, we present an algorithm for context-agnostic learning. We first develop a formal notion of contextual bias for this setting. We assume binary classifier h and slightly abuse notation, writing h for o 7, i..., h : O x + {-1,1}, 1,. For an object o, denote the correct label o*, the expected classification 0 Ee (h((,, e)), and the object error = lo" o
3

---

Under review as a conference paper at ICLR 2021
Definition 3.3. The context bias B(h, c) of classifier h the context is defined
sgn(B(h, c)) = sgn( (b),c - [| c)ll = Eoro (((o))((((.cc),)) where E is the hinge loss E(i,j) = max(0, j.
Intuitively, sign of the bias corresponds the label toward which the classifier is biased by a given context; the magnitude measures the strength of this bias. Clearly, the classifier is context-agnostic exactly when the bias is zero. We are now ready to state our main theoretical result, which gives an upper bound on the risk in terms of the context bias on C' and object error over O.
Theorem 3.1. Let h be classifier with average bias K and object error for all objects bounded from above by a < 1. Then the risk is bounded from above by K/(2 - oz). Furthermore, equality holds if and only if all object errors equal a.
We give a proof in Appendix A The assumption a < is fairly weak, being equivalent to the classifier performing better than random guessing. Note that the error bound and bias bound K are not independent; in particular, a 0 if and only if K
and a < 1. Observe also that when C == 0, K = 0 holds trivially, but < 1 for all objects means the classifier is correct on all inputs.
The central idea behind Theorem is leveraging the fact that labels depend only on objects to factor the risk into separate terms for object error and context bias. This factorization enables us to exploit our ability to sample independently from the object and context spaces. More specifically, we can use samples from O to minimize the object error, and samples from C to minimize the context bias. Since we only need a < 1, we continue to draw objects randomly; however given an object we aim to observe it with the context for which the classifier has the strongest opposing bias. Intuitively, this allows the classifier to "correct" its bias and unlearn the spurious signals, thereby minimizing the bias and also the risk.
Adopting this approach without modification requires computing the bias of every context in C. In most cases, however, even estimating single bias may be prohibitively expensive. Thus, rather than solve for the maximum bias explicitly, we instead propose a heuristic for identifying contexts with large biases. Note that since X' C C, reasonable assumption is that the classifier learns strong bias on recent training inputs when taken as contexts. This suggests a simple greedy approach for correcting biases by repurposing recent training inputs as contexts; we call this algorithm Greedy Bias Correction and present a description in Algorithm
4 LEARNING VISUAL TASKS USING CONTEXT-AGNOSTU SYNTHETIC DATA
We introduce an instantiation of Greedy Bias Correction for learning visual tasks using synthetic data. We are given a function which takes a label and outputs a rendering of the corresponding class in random pose without any background. The context is the background of the image, on which we place no restrictions. The observation function fy superimposes an object over a background.
Local refinement via robustness training We note that our observation function is fairly re- strictive; for instance, we do not support occlusions. Because our ultimate goal will be to perform on data taken from real-world context, we aim to capture this discrepancy using robustness train- ing ! In particular, we assume that the image of y is an e-covering of X, where a set A is said to be an e-covering of another set B iff for all points b B, there exists a point E A such that ||a - bll & Then for given sample, we will instead add the point in the e-neighborhood of which maximizes the training loss, i.e., for classifier h and sample = 7(0,c), c), we x' = arg max reN f(h(z'), y)- This formulation is often used to train models which are robust against local perturbations. An empirically effective method for finding approximations to x' is known as Projected Gradient Descent (PGD) (Goodfellow et al., 2014b. Madry et al., 2017). The
Robustness training is more commonly referred to as adversarial training in the adversarial robustness com- munity whence we horrow this technique. We use the nonstandard term to avoid confusion with the unrelated (generative) adversarial methods found in the few-shot learning literature.
4

---

Under review as a conference paper at ICLR 2021
Algorithm 1: Greedy Bias Correction
Input:
Object space O, context space C, observation function y, number of rounds R,
resample probability classifier update subroutine F Fit, binary classifier h.
Output: Trained classifier h
// initialize random context and label
C C;
y ~ { 1111
for +- 1 to R do
0 O(y); sample object
y(0, // observe object and context
h - Fit I, y): // perform classifier update
11 update context and label
p' < Uni form(0,
if p' < then
// resample random context and label C:
y ~ {-1,1};
else
C - x; previous image becomes new context
y y:: flip label
end
end
(5)
(2b)
(2a)
Figure 1: graphical representation of the generative loop in Algorithm 2 using real training data. (1) Sample from object space. (2) Observe object and context. (3) Perform local refinement. (4) Add to training set. (5) Previous image becomes next context (resample from with probability p).
algorithm can be summarized as
20 +
I; I1 (1;-
sgn (,((((,;))))))))))
=
where 8 is small amount of random noise, 11 is the projection back onto to the e-ball, is the step size, and the number of iterations. As is standard for robustness training, we use the foo norm defined as [Z, F,), max; z;, Our choice of e will depend on the task at hand, and we also use different for the portions of the image corresponding the object and context.
Additionally, since we are no longer in binary context, we sample random permutation on labels instead of flipping the label deterministically. The full algorithm is presented as Algorithm g in Appendix B Figure [ provides visualization of the key generative process, with images taken from real step of training deep neural network to perform classification traffic signs.
From a practical standpoint, this algorithm makes concrete several benefits of our approach. First, rendering object classes, i.e. sampling from O, is often relatively easy. In the case of two- dimensional rigid body objects, this can be captured using standard data augmentation such as rotations, flips, and perspective distortions. Indeed, in this setting, our work can be viewed as a form of minimal one-shot learning, where the training data consists solely of single unobstructed straight-on shot for each object class. Second, there no requirement to perform realistic rendering of contexts C, avoiding additional layer of complexity.

---

Under review as a conference paper at ICLR 2021
30
#
70
0 5
8
6
7
9
0 2 3
4 6
7 6 9
Figure 2: Images from the training (top) and test (bottom) set for GTSRB (left) and MNIST (right).
Table I: Performance of Algorithm 2 on various benchmarks, plus ablation studies.
Approach
Picto + GTSRB
Digit + MNIST
Omnifont + Omniglot
baseline
72.0
81.9
71.9
random-context
72.1
88.3
69.8
+ refinement-only
86.4
89.7
90.8
bias-correction
87.3
89.2
80.5
+ full
95.9
90.2
92.2
Finally, because our approach context agnostic, our functions are learned without any reference to target domains. In the formal setting, we assumed that the target domain was contained in the image of the observation function; however, synthetic images will always be subject to the reality gap. Our experiments suggest that approach overcomes this barrier and successfully generalizes to natural images while training on synthetic data only.
EXPERIMENTS
We evaluate our approach to learning visual tasks using synthetic data on three benchmarks for im- age recognition. Our training sets consist of single synthetic image for each object class with no additional information about the target domain: Figure 2 shows examples of the training and test images from two of the datasets. On all three benchmarks, our models perform comparably with previous sfate-of-the-arr results from related settings using few-shot learning and domain adaptation. Table provides summary of our results; comprchensive results and comparisons Appendix D Appendix
compiled in C provides the full experimental setup and training details. Sample images from all datasets referenced below, including examples of rendered training data from the experi- ments and ablation studies, are shown in Appendix
5.1 GTSRB
The German Traffic Sign Recognition Benchmark (GTSRB) (Stalkampp al. 2012) contains 39,209 training and 12,630 test images of 43 classes of German traffic signs taken from the real world. Our training set consists of single, canonical pictogram of each class taken from the visual- ization software accompanying the dataset, which we refer to as Picto. We achieve 95.9% accuracy on the GTSRB
training only on Picto, against a human bascline of 98.8%. A comprehensive comparison with existing approaches can be found in Appendix D Table
SynSign (Moiseev et al. 2013) is a synthetic dataset designed provide realistic training data for traffic sign recognition. The dataset comprises 100,000 synthetically generated images of signs from Sweden, Germany, and Belgium in a variety of poses, rendered against domain -ammainapppopriate real-world backgrounds (e.g. trees, roads, sky). The dataset contains superset of the GTSRB classes; as a result, [Saio et al. (2017) report 79.2% accuracy by training directly on SynSign.
For domain adaptation, all approaches train on the full 100,000 images in SynSign plus part of the GTSRB training set. ATT (Saito et al. 2017 the only method with better performance than ours, achieving 0.3% higher accuracy; however they use 31,367 unlabelled images from the GTSRB training set (in addition to SynSign). Methods using few-shot learning train on roughly half of the data (22 classes) from the GTSRB training set. The leading few-shot learning approach, VPE (Kim (et al. 2019), adds a pictographic dataset similar to Picto, but achieves only 83.79% accuracy. In comparison, training set consists of only 43 images, none of which are from GTSRB.

---

Under review as a conference paper at ICLR 2021
5.2 HANDWRITTEN CHARACTER RECOGNITION
MNIST (LeCun) consists of 60,000 training and 10,000 test images of handwritten Arabic numerals in grayscale against a blank background. Our training set, Digit, consists of a single example of each digit taken from a standard digital font. Omniglot (Lake et al., 2015) consists of 1623 hand-written characters from 50 different alphabets, with 20 samples each. The samples were sourced online from 20 workers on Amazon's Mechanical Turk, who were asked to copy each character from a single font-based example using digital input (e.g., a mouse). We obtained the original representations for our dataset, OmniFont. On MNIST, we achieve 90.2% accuracy training only on Digit, compared to human accuracy of 98%; on Omniglot, we achieve 92.2% 20-way accuracy training only on Omnifont, compared to human accuracy of 95.5%. Tables 3 and in Appendix D compare these results with approaches using few-shot learning and domain adaptation.
Handwritten characters and GTSRB present conceptually opposed challenges for learning: in GTSRB, the objects are rigid two-dimensional objects and backgrounds are complex settings in the natural world; in Omniglot and MNIST, backgrounds a strict specification and individual examples exhibit high variability. Thus, the main challenge of these tasks is learning how to generalize over the object class. Despite the inherent variation, baseline model trained on Digit with plain data augmentation was able to achieve 81 81.9% accuracy on MNIST, exceeding many domain adaptation approaches and all the one-shot learning results; Omniglot is more difficult, with an Omnifont plus data augmentation baseline accuracy of 7 71.9%.
uniform, but classes longer have
On MNIST, every approach using domain adaptation uses the full Street View House Numbers (SVHN) training set of 73,257 images of house numbers obtained from Google Street View (Netzer) et al. 2011), plus varying amounts of data from MNIST. The domain transfer problem faces a similar challenge as Digit, namely, handwriting exhibits different characteristics than house numbers fonts. Nevertheless, we note that SVHN contains far more examples of each digit. The only non- baseline approach to exceed our performance is CyCADA (Hoffman et al. 2017). which achieves 0.2% better performance by performing domain adaptation using 60,000 unlabelled images from the MNIST training set (in addition to training SVHN). All approaches using few-shot learning (except FADA) train on 32,460 images from Omniglot and use as few as one image per class from MNIST; the best result achieves accuracy 3% below ours using 70 images from MNIST. In contrast, we use only 10 images, none of which are from MNIST.
Omniglot is often described as an MNIST-transpose, where the goal is learn handwriting rather than specific symbols, and is widely used as a benchmark for few-shot learning. We reproduce the most common split given in Lake et al. (2015), which uses a predefined set of 30 alphabets, with 19,280 images for training. Test performance is reported as an average over random subsets of n = 5, 20 unseen classes for the n-way task (given one labelled example). In comparison, for each test run, we retrain model using only the corresponding n images from OmniFont. As expected, our method finds 5-way classification easier than 20-way classification (95.8% vs 92.2%). In both cases, our performance lags behind the state-of-the for few-shot learning (>99%), though we emphasize that our experimental setup differs significantly in both the type and amount of training data used.
Finally, several approaches apply few-shot learning from Omniglot to MNIST, with the idea of trans- ferring extracted features from human handwriting. However the one-shot experiments all perform worse than even baseline approach. We hypothesize that in comparison to Omniglot, where all the samples come from the same 20 subjects, MNIST may be particularly difficult for transfer one- shot learning, since any two examples will likely exhibit high "variance", conversely, our approach benefits from using a canonical form which might be closer to the "mean" representation.
5.3 ABLATION STUDIES
We conduct two sets of ablation studies to better understand our approach to context-agnostic learn- ing. The first study tests the individual components of our algorithm for their contributions to gener- alization over the real world dataset. All strategies employ the same data augmentation and use the following sampling procedures: baseline picks a fresh random background for each training point, and measures the performance of training on our synthetic dataset with plain data augmentation; random-contex reuses random backgrounds images as contexts; refinement-only is the same as random-conten with the addition of PGD-based refinement; full is the full algorithm as described in Algorithm 2 The results are in Table T
contexts; bias-correction reuses previous training
7

---

Under review as a conference paper at ICLR 2021
100
75
Accuracy (%)
50
25
leacine
mndon-cmites
miizenetionly
0
realiian
0
16
64
255
0
4
16
64
255
a) Random initialization (c/255)
b) Bias heuristic (c/255)
Figure 3: Context-agnostic performance Picto using PGD adversary the background.
In all cases, we observe that both bias correction and local refinement contribute individually and jointly to the performance of our models. For GTSRB, a particularly interesting comparison training on SynSign, dataset designed to provide synthetic training data with realistic backgrounds for GTSRB, which yields 79.2% accuracy (Saito et al. 2017). Though this is an improvement over baseline of using random backgrounds at 72.0% accuracy, refinement-only and bias-correction achieve higher accuracy at 86.4% and 87.3%. respectively. Both methods leverage the background of training images combat spurious signals, generating completely unrealistic backgrounds; this suggests that learning context-agnostic features is more effective than using realistic backgrounds.
The second study measures classification performance in context-agnostic setting the synthetic Picto dataset. By definition, the performance of context-agnosia classifier should not degrade under perturbations of the background. We thus run adaptive attack using PGD adversary which fixes the foreground pixels, and ranges from fixed to unbounded on the background pixels, effectively searching the context space for a background that causes a misclassification on the given object. We also consider two initialization strategies for the PGD adversary: standard random initialization, and initializing to the previous image, inspired by our bias heuristic. We test the same set of strategies as before, plus a classifier trained directly on the GTSRB training set achieving 98% performance on the GTSRB test set (real2sim). Appendix contains samples the generated images, and the results are plotted in Figure
Across all experiments, the models have worse (or very close) performance when using our bias heuristic for initialization. We believe this supports usage of the bias heuristic for context- agnostic learning. Additionally, in the last column of Figure 3b, only our full method maintains passable accuracy, which suggests the gap between models is larger than performance indicates. We also note that real2sim seems to suffer from a "synthetic gap" even at c = 0/255, which not entirely unexpected. However, in both settings, performance degrades very quickly as increases: the effect is most pronounced when the bias heuristic is used to initialize the PGD adversary, though in both cases the accuracy eventually drops O. We emphasize that all of the ex- periments leave the foreground objects completely unperturbed (and easily cumman-dentifiablee our results thus suggest that classifiers trained on natural images can become over-reliant on contextual signals, leading to surprisingly brittle behavior even given unambiguous foregrounds.
GTSRB
CONCLUSION
We introduce the task of context-agnostic learning, theoretical setting for learning models whose predictions are independent of background signals. Leveraging the ability to sample objects and con- texts independently, we propose an approach to context-agnostic learning by minimizing formally defined notion of context bias. Our algorithm has a natural interpretation for training classifiers on vision-based tasks using synthetic data, with the distinct advantage that we do not need to model the background. We evaluate our methods on several real-world domains; our results suggest that our approach succeeds in learning context-agnostic classifiers that generalize to natural images using
8

---

Under review as a conference paper at ICLR 2021
only single synthetic image of each class, while training with natural images can lead to brittleness in the context-agnostic setting. Our performance is competitive with existing methods for learning when data is limited, while using significantly less data. More broadly, the ability to learn from single synthetic examples of each class also affords fine-grained control over the data used to train models, allowing us sidestep issues of data provenance and integrity entirely.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Antreas Antoniou, Harrison Edwards, and Amos J. Storkey. How to train your MAML. CoRR, abs/1810.09502, 2018. wtllhtt:p://axxiv.orgg abw//bsss/11...///// 09502
Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector ma- chines, 2012.
John Blitzer, Ryan McDonald, and Fernando Pereira. Domain adaptation with structural correspon- dence learning. In Proceedings of the 2006 conference on empirical methods in natural language processing, pages 120-128, 2006.
Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain separation networks. In Advances neural information processing systems, pages 343- 351, 2016.
G. Bradski. The OpenCV Library. Dr. Dobb's Journal of Software Tools, 2000.
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. arXiv preprint arXiv:1711.039388 2017.
Li Fe-Fei et al. A bayesian approach to unsupervised one-shot learning of object categories. In Proceedings Ninth IEEE International Conference on Computer Vision, pages 1134-1141. IEEE, 2003.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta- for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 1126-1133 JMLR. org, 2017.
Adrien Gaidon, Qiao Wang, Yohann Cabon, and Fleonora Vig. Virtual worlds as proxy for multi- object tracking analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4340-4349, 2016.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net- works. The Journal of Machine Learning Research, 17(1):2096223 2030, 2016.
Muhammad Ghifary, W. Bastiaan Kleijn, Mengjie Zhang, David Balduzzi, and Wen Li. Deep reconstaction-caasiiiccttiionnn networks for unsupervised domain adaptation, 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances neural infor- mation processing systems, pages 2672-2680, 2014a.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6577, 2014b.
Ankush Gupta, Andrea Vedaldi, and Andrew Zisserman. Synthetic data for text localisation in natural images, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition, 2015.
Judy Hoffman,, Eric Tzeng, Taesung Park, Jun- Yan Zhu, Phillip Isola, Kate Saenko, Alexei A Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. arXiv preprint arXiv: 2017.
Max Jaderberg, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Synthetic data and artificial neural networks for natural scene text recognition, 2014.
Nick Jakobi, Phil Husbands, and Inman Harvey. Noise and the reality gap: The use simulation in evolutionary robotics. In European Conference Artificial Life, pages 704-720. Springer, 1995.
10

---

Under review as a conference paper at ICLR 2021
Matthew Jchnson-RRbeensee Charles Barto, Rounak Mehta, Sharath Nittur Sridhar, Karl Rosaen, and Ram Vasudevan. Driving in the matrix: Can virtual worlds replace human- -enarrtee annota- tions for real world tasks? arXiv preprint 111.0:1610.01983, 2016.
Junsik Kim, Seokju Lee, Tae-Hyun Oh, and In So Kweon. Co-domain embedding using deep quadruplet networks for unseen traffic sign recognition. In Thirly-Second AAAI Conference Artificial Intelligence, 2018.
Junsik Kim, Tae-Hyun Oh, Seokju Lee, Fei Pan, and In So Kweon. Variational prototyping-encode:: One-shot learning with prototypical images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9462-9470, 2019.
Gregory Koch. Siamese neural networks for one-shot image recognition. 2015.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, (10062666:1332-1133 2015.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/nmist//
Aoxue Li, Tiange Luo, Tao Xiang, Weiran Huang, and Liwei Wang. Few-shot learning with global class representations. In Proceedings of the IEEE International Conference on Computer Vision, pages 17244, 2019.
Ming- Yu Liu,, Thomas Breuel, and Jan Kautz. Unsupervised image-to-imant image translation networks. In Advances neural information processing systems, pages 700-708, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 1706::7706066833 2017.
Boris Moiseev, Artem Konev, Alexander Chigorin, and Anton Konushin. Evaluation of traffic sign recognition methods trained on synthetically generated data. In International Conference on Advanced Concepts for Intelligent Vision Systems, pages 576-583, Springer, 2013.
Saeid Motüaan, Quinn Jones, Seyed Iranmanesh, and Gianfranco Doretto. Few-shot adversarial domain adaptation. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus. S. Vish- wanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 6670-6680. Curran Associates, Inc., 2017. WtL.http://paperr ni ps. cc cc/paper/ n adaptation..dd
Tsendsuren Munkhdalai and Hong Yu. Meta networks. In Proceedings of the 34th International Conference Machine Learning-Volume Volume 70, pages 2554 2563. JMLR. org, 2017.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint 1803. 020999 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019. URL http: http:://apppee.eeee neurips cc/paper/ http://ppprs..uuiipsppppp
9015 pytorch an - imperative style high www.-pprfooomneeeeeeeo -eeeaaaneeedee----eeee wwweeee--eeerinne learning library.
pdf
Zhongyi Pei, Zhangjie Cao, Mingsheng Long, and Jianmin Wang. aulti-advvrsarial domain adap-
tation. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Weichao Qiu and Alan Yuille. Unrealev: Connecting computer vision to unreal engine, In European Conference Computer Vision, pages 909-916. Springer, 2016.
11

---

Under review as a conference paper at ICLR 2021
Tiago Ramalho and Marta Garnelo. Adaptive posterior learning: few-shot learning with surprise- based memory module, 2019.
Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. 2016.
Kuniaki Saito, Yoshitaka Ushiku, and Tatsuya Harada. Asymmetric tri-training for unsupervised domain adaptation. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pages
JMLR. org, 2017.
Adam Santoro, Sergey Bartumov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with ing, pages 1842-1850, 2016.
neural networks. In International conference on machine learn-
Alice Seneenauer-Sebbage Louise Heinrich, Mare Schoenauer, Michele Sebag, Lani F. Wu, and Steve Altschuler. Multi-domain adversarial learning, 2019.
Eli Schwartz, Leonid Karlinsky, Joseph Shtok, Sivan Harary, Mattias Marder, Rogerio Feris, Ab- hishek Kumar, Raja Giryes, and Alex M. Bronstein. Delta-encoder: effective sample synthesis method for few-shot object recognition, 2018.
Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Josh Susskind, Wenda Wang, and Russ Webb. Learning from simulated and unsupervised images through adversarial training, 2016.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, pages 4077-4087, 2017.
Stallkamp, M. Schlipsing, J. Salmen, and C. Igel. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural Networks, (0):-, 2012. ISSN 0893-6080. doi: m0..0016jj.ccmmee211222.mm 2.2..022.66666 ott. art article/pp1/30093300011111
/ttp::://wwww..e www.:.sciennedieee.ooooo. http://weee.eee...cccccccceeeeee wwm www.science//////c/coccccecccccc
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Con- ference on Computer Vision and Puttern Recognition, pages 1199-1208, 2018.
Yaniv Taigman, Adam Polyak, and Lior Wolf. Unsupervised cross-domain image generation. arXiv preprint 161 1.02200, 2016.
Josh Tobin, Rachel Fong, Alex Ray. Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSI international conference on intelligent robots and systems (IROS), pages 23-30. IEEE, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033, IEEE, 2012.
Tatiana Tommasi, Novi Patricia, Barbara Caputo, and Tinne Tuytelaars. deeper look at dataset bias. In Domain adaptation in computer vision applications, pages 37-55. Springer, 2017.
Lucas Tabelini Torres, Thiago M. Paixão, Rodrigo F, Berriel, Alberto F. De Souza, Claudine Badue, Nicu Sebe, and Thiago Oliveira-Santos. Effortless deep training for traffic sign detection using templates and arbitrary natural images, 2019.
Jonathan Tremblay, Aayush Prakash, David Acuna, Mark Brophy, Varun Jampani, Cem Anil, Thang To, Eric Cameracci, Shaad Boochoon, and Stan Birchfield. Training deep networks with synthetic data: Bridging the reality gap by domain randomization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 969 9979 2018
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7167-7176, 2017.
12

---

Under review as a conference paper at ICLR 2021
Stéfan van der Walt, Johannes L. Schönberger, Juan Nunez-Iglesias, François Boulogne, Joshua D. Warner, Neil Yager, Emmanuelle Gouillart, Tony Yu, and the scikit-image contributors. scikit- image: image processing in Python. PeerJ, 2xe453, 6 2014. ISSN 2167-8359. doi: 10.7717/peerj. 453. URL htttht/p::///od www.orog...... 10. 7717 wtt.777///ppr...555
Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, pages 3630-3638, 2016.
Sung Whan Yoon, Jun Seo, and Jaekyun Moon. Tapnet: Neural network augmented with task-
adaptive projection for few-shot learning, 2019,
Jian Zhang, Chenglong Zhao, Binghing Ni, Minghao Xu, and Xiaokang Yang. Variational few-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
13

---

Under review as a conference paper at ICLR 2021
A PROOFS
Proof of T'hoormm.. 3.7 By the assumption that a < 1, we have that for all o, the signs of the expected
classification and correct classification match, so that a 2 ò lo* -ōl - Then for all
{ 0²) = Ō
1 |ō|
1 1+ lō
(1 (o))
1
o*0 1 lō f(ō,ō)
1+10
$ ((ō,0)
a
Now to bound the risk, we can write.
R(h) Earo and (((h(o,)))
((h(o,c),00)
C|O
Σ C
h(o,c) 00))
O
Σ (1 0*0")
00|
< 101 8
Σ
1 0+0
2 2-a a (2 0
1
h(),c))))
[C|
((h(o.c),o)
(2 a)|c|10|
(2 2--a)l |BB(hc)]l a]|c
K
2-c
as desired, It also follows that equality holds if and only a â for all
14

---

Under review as a conference paper at ICLR 2021
B
GREEDY BIAS CORRECTION FOR VISUAL TASKS
Algorithm 2: Visual Learning Using Context-Agnostie Synthetic Data
Input: Object space O, context space C, random permutations II, observation function 7 number of rounds R. batch size B, number of classes N, resample probability p. classifier update subroutine Fit, projected gradient descent subroutine PGD, classifier h
Output: Trained classifier h
for r a 1o R do
// initialize empry training batch and random contexts
+ 0:
for to N do
en C;
end
for b + 1 to do
// sample random permutation
π ~ I[N);;
// generate training data
for n + to N do
On); // sample object for class
x <- 7(0, (x));; observe object and random (permuted) context
+ PGD(h, IT;; perform local refinement
X XU { vy)): // add training
Cn t x':// previous sample becomes next context
end
// resample contexts for
to N do p' +- Uni for(0, 1); if p p then ~ C;
end
end end
// perform classifier update h Fit (h, X);
end
15

---

Under review as a conference paper at ICLR 2021
C EXPERIMENTAL SETUP
We used Py Torch 1.5.0 Paszke et al. 2019). OpenCV 4.2.0 (Bradski 2000). and scikit-image 0. 1..22 van der Walt et al. 2014) for all experiments. In setting the number of epochs, we did observe any significant degradation or improvements in performance when training for longer. We use fewer epochs in the case of Omniglot due to computational constraints, as the model is retrained for each test split.
For GTSRB, we use a 5-layer convolutional neural network adapted from the official PyTorch tutorials. To train with Picto, the data augmentation consists of PyTTrrch transforms Rando- mAffine(5, translate=(.15, .15), scale=(0.65,, 1.05), shear=5), SanndomPesspectieeO.... p=1); Col- brJiteer(righhnness:.88 contrast=.8, saturation=.8, hue=:05); OpenCV box blur with random kernel size between and both dimensions (independently sampled, so not necessarily square); and a random exposure adjustment by adjusting all pixels by the same random amount between -30% and 50%. For refinement, we used step sizes of a = 2/ 2/255 with 8 steps and an epsilon of e 4/255 for the foreground only. For the observation function, transformed pictographic sign over the context. We train for 300 epochs using the Adam optimizer (learning rate le-4, weight decay le-4), with 5 examples of each class per batch and 20 batches per epoch. We report results for the model that achieves the best performance on the training set, checking every epochs.
superimpose the segmented foreground of the
For MNIST, we use the two-layer convolutional neural network from the official PyTorch exam- ples for MNIST, with Dropout regularization replaced with pre-activation BatchNorm. To train with Digit, the data augmentation consists of Py PyTorch transforms RandomAffine scalc=(0.755, 1.05). shear=40), candomPerspectiveer nel size between 1 and 6 in both dimensions (independently sampled, so not necessarily square); then set the foreground all pixels with value greater than 0.2. For refinement, we used step sizes of a = 1.6/255 with 8 iterations and no projection (e oo). For the observation function, we blend the object with the context at a 2:1 ratio; this ensures that inputs have well-defined ground truth label. We train for 300 epochs using the Adam optimizer (learning rate le-4, weight decay le-4), with 5 examples of each class per batch and 20 batches per epoch. We report results for the model that achieves the best performance the training set, checking every epochs.
translate=(.15, .15,, p=1); OpenCV box blur with a random ker-
For Omniglot, we use the pre-activation variant of ResNet18 (He et al. 2015). To train with Omni- font, we first preprocess with seikit-learn skeletonize and dilation to standardize stroke widths. Data augmentation consists of PyTorch transforms GandomAAffiee155 translate=(.15. .15), scale=(0.75, 1.1), 2hear=20), RandoomPersecctivee0..5 p=l); OpenCV box blur with random kernel size be- tween I and 3 in both dimensions (independently sampled, so not necessarily square); then resize the images 28 by 28. For refinement, we used step sizes of a 1.6/255 with iterations and no projection (c = oc). For the observation function, we blend the object with the context at 2:1 ratio; this ensures that inputs have well-defined ground truth label. For the n-way classification task, we randomly sample n characters from the Omniglot test set, and use the corresponding characters from the Omnifont dataset as our training set. We then train fresh model for 150 epochs using the Adam optimizer (learning rate lc-4, weight decay le-4), and report performance on the all 20n images in the Omniglot test set, averaged over 20 runs (10 runs for the ablation studies).
16

---

Under review as a conference paper at ICLR 2021
D FULL EXPERIMENTAL RESULTS
We compare model trained using our methods with previous state-of-the-art results from related settings using few-shot learning and domain adaptation GTSRB (Table MNIST (Table and Omniglot (Table 4 When multiple experiments are reported for the same approach, we compare against both the most accurate result as well as the result using the least amount of target data. We distinguish between labelled (L) and unlabelled (UL) data; experiments for which the training data not known are marked (?).
Table GTSRB results.
Training Data
Accuracy (%)
Approach
Method
Source
Target
Source Only Saito et al. (2017)
SynSign
79.2
Baselines
Human Stallkamp et al. 2012)
98.8
Target Only (Ganin et al. (2016)
All L
99.8
VPE (Kim et al. 2019) 8
Picto*
22 classes L
83.8
Few-Shot Learning
MatchNet Vinyalls et al.
(0
22 classes L
53.3
QuadNet Kim et al. [20
8)
22 classes
45.3
DSN Bousmalis al. (016))
SynSign
1280 UL
93.0
ML ISchoenauer-Sebaa al.
2019 8
SynSign 22 classes L
89.1
Domain Adaptation
MADA Pei al. (2018)
)
SynSign
22 classes L
84.8
DANN (Ganin al.
2016))
SynSign
31367 UL
88.7
ATT (Saioo et al.
2017
SynSign
31367 UL
96.2
baseline
Picto
72.0
random-context
Picto
72.1
Context Agnostic
refinement-only
Picto
86.4
cias-correction
Picto
87.3
full
Picto
95.9
Test accuracy on remaining 21 unseon classes.
Kim et al. (2019) use a pictographic dataset similar to Picto.
Reported in Kim et al. (2019).
Reported in Schoenauer- Sebag el 2019)
17

---

Under review as a conference paper at ICLR 2021
Table 3: MNIST results.
Training Data
Accuracy (%)
Approach
Method
Source
Target
Human (etzerr et al. 201
98.0
Baselines
Target Only (Tzeng et al. (211))
A
99.2
FADA (Motian et (017))
SVHN
L class
72.8
+ more data
SVHN
L / class
87.2
SiamNet (Koch 2015)
Omniglot
L / class
70.3
Few-Shot Learning
MatchNet (Vinyals et al. (2016)
Omniglot
L class
72.0
APL (Ramalho and Garnelo 2019)
Omniglot
L class
61.0
+ more data
Omniglot
2
86.0
DSN Bousmalis al. 2016))
SVHN
1000 UL
82.7
DRCN Ghifary al.
(0016)
SVHN
?
81.9
DANN Ganin al.
2016
SVHN
?
73.9
Domain Adaptation
ATT (Sato et al.
201
SVHN
? L 1000 UL
86.0
ADDA (Tzeng al. (2017)
SVHN
60,000 UL
76.0
CyCADA (Hffiman et al.
2017
SVHN
60,000 UL
90.4
baseline
Digit
81.9
random-context
Digit
88.3
Context Agnostic
refinement-only
Digit
89.7
bias-correction
Digit
89.2
full
Digit
90.2
Cuumulative accuracy from adapting over the test set.
Table 4: Omniglot results for one-shot classification.
Accuracy (%)
Approach
Method
Training Data
5 5-way
20-way
Baselines
Human Lake et al. 2015))
95.5
MANN Santoro et al.
2016))
Omniglot
82.2
SiamNet (Koch (2015)
Omniglot
96.78
92.0
MatchNet Vinyall et al.
2016)
Omniglot
98.1
93.8
Few-Shot Learning
PN (Selll et al. (2017)
Omniglot
98.8
96.0
BPL Lake et al. (2055)
Omniglot
96.7
APL Ramalho and Garnelo
2019
Omniglot
97.9
97.2
RN (Sung et al.
(2018)
Omniglot
99.6
97.6
MAML++ (Antoniou et al.
2018
Omniglot
99.5
97.7
TapNet Yoon et (2019)
Omniglot
98.1
GCR (Li et al. (2019))
Omniglot
99.7
99.6
baseline
Omnifont
71.9
+ random-context
Omnifont
69.8
Context Agnostic
refinement-only
Omnifont
90.8
+ bias-correction
Omnifont
80.5
full
Omnifont
95.8
92.2
"hhe exact set up of the one-shot classification task often varies between authors. We believe the broad performance numbers still useful for contextualizing approach, and refer the reader the original works for details.
8 As reported in Vinyals al. 2016
18

---

Under review as a conference paper at ICLR 2021
E
TRAINING AND TEST SET VISUALIZATIONS
E. DATASETS
70
30
70
Figure 4: From top to bottom: samples from the GTSRB test set, Picto dataset, and SynSign dataset.
2
M
P
S
0
4
5
6
9
0
1
2
3 4
5
8 9
3
16
16
33
6
24
Figure 5: From top to bottom: samples from the Omniglot test set, Omnifot dataset, MNIST test set, Digit dataset, and SVHN training set.
19

---

Under review as a conference paper at ICLR 2021
E.2 ABLATION STUDIES
50
20
30
100
STOP
30
Figure 6: Training images from the first ablation study using Picto dataset. From top to bottom: baseline, random-context, refinement-only, bias-correction, full.
80
A
20
30
70
STOP
Figure 7: Test images from the second ablation study using the Picto dataset. Examples of test im- ages generated using PGD adversary initialized randomly (top) and with the bias heuristic (bottom) at E 255/255.
9
7
6
2
3
0
Q
L
4
8
/
4
3
8
7
4
6
9
8
3
6
4
0
2
3
7
4
8
5
6
Figure 8: Training images from the first ablation study for the Digit dataset. From top to bottom: baseline, random-context, refinement-only, bias-correction, full.
20


Reviews:
[{"review_id": "jRpdiZdG87P", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n\n1. This paper studies a theoretical setting for learning models whose predictions are independent of background signals. There are indeed some practical applications of this setting.\n\n2. The proposed approach is very simple, yet effective. This method is able to learn a context-agnostic model by minimizing a formally defined notion of context bias. \n\n3. On the theorical side, the authors try to explain their goal of classification as learning to extract reliable signals.\n\n4. The paper is well written and easy to read. Also, the figures in supplementary materials help understand the paper, especially the context learning part.\n\nWeaknesses/concerns:\n\n1. The proposed method has a strict restriction: The training sets must have a single synthetic image for each object class with no additional information about the target domain. This may limit the training of the model in some hard conditions.\n\n2. The quality of the paper will be upgraded if the authors further investigate more data augmentation or style transfer methods in related work and ablation study. Since the context and object concepts are similar with the style and content in some GANs and transferring works.\n\n3. The experiments seem a little bit weak. The experiments are only conducted on two simple datasets, i.e. GTSRB and MINST.\n\n4. The comparison with existing methods is insufficient. There is no representative baseline comparison. Though this setting is novel, we can conduct task by slighting changing existing methods.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "JZV1BKedr6E", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a context-agnostic learning approach that combines an object area and a context image (s.t. background image) to generate input synthetic image and train the model as context-independent. The proposed method is made more efficient by including this generation process in the training loop, compared to a exhaustive sampling method that randomly selects from a set of combinations of object areas and context images. When applying this method on the task of traffic sign recognition, character recognition, it provides enhanced performance in a setting where model is trained using synthetic data and evaluated on real-world dataset.\n\nMy concern is the presentation.\n\"Context\" and \"context-agnostic\" are properly defined. However, it is very difficult to understand how to achieve \"context-agnostic\" in algorithm 1 and 2. In my understanding, context bias is corrected by forcing a model trained using the opposite (or other) object class in the same context in the next training iteration. If it is correct, how can the bias be corrected just by training with other objects? Does it affect to reduce ||B(h,c)|| in Definition 3.3? Please clarify it.\n\nIn addition, the effectiveness of the proposed method can be verified with such simple tasks, but it is advisable to add a discussion about more complex tasks such as object detection that requires more diverse contexts. This will be more interesting for many.\n\nI can recognize some technical contributions that can appeal to many ICLR researchers.\n\nAs a minor correction,\nIn Section 3.2, the object error o^ is defined but not used anywhere.\n\n----------------------------------------------------------------------------------------------------------\nI have read the revised manuscript and the comments of all reviewers. For concerns that the experiments are not sufficient to validate the proposed method, I am leaning to the author's rebuttals that the datasets have been chosen to meet the heretical assumptions of context-agnostic learning. The given datasets seem to be sufficient to demonstrate the effectiveness of the proposed learning method.\n\nTherefore, I will not change the rating.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "BJ9i5HFgSqx", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper defines the task of context-agnostic learning and proposes an algorithm to solve the problem while assuming the ability to sample objects and contexts independently. They propose to decompose factors contributing to the risk into two, context bias and object error. Based on this interpretation, an algorithm is designed to 'greedily correct bias' while employing adversarial training (or robustness training) for 'local refinement'. The method achieves high accuracy on two synthetic visual tasks, digits and traffic sign classification, when a model is trained using one sample per class from the source domain and tested on an unseen target domain.\n\n+) Theorem 3.1 provides a new view on risk. Risk is decomposed into two factors, context bias and object error. I think this gives new insight to consider the effect of context bias and object modeling on risk separately.\n\n+) The experimental results are impressive because the model is trained using a very limited number of samples (one sample per class from the source domain) but showed high generalization performance on an unseen domain. The proposed method achieves promising performance on two synthetic classification tasks compared to other existing methods for few-shot learning and domain adaptation, which requires more labeled or unlabeled data during training.\n\n-) Their underlying assumption for greedy bias correction is that a classifier learns a strong bias on recent training inputs when taken as contexts. However, if stochastic gradient descent is used for optimization, I think it is unlikely because the model changes continuously. Therefore, it is uncertain how effective this greedy selection strategy can sample contexts with large bias.\n\n-) Relating to the above point, I also have a concern about the experimental validation. In all the experiments, gamma is defined as a function that takes object and context images and outputs their overlap. It is not guaranteed that the proposed heuristic sampling strategy generalizes to other gamma functions.\n\n-) Also, all the experiments are performed for a relatively small number of classes (up to 50), and synthetic images are small iconic images with objects in the center. Although the method shows promising results under this specific setting, it is hard to conclude that the proposed heuristics will generalize other settings, such as when there are more classes, image resolutions are higher, and objects have a larger variation in their appearance. I think evaluation on additional datasets with different characteristics (such as CIFAR-100, Caltech-256, CUB-200) would be necessary.\n\n-) The assumption that one can sample objects and contexts independently may restrict its application.\n\nThough I found this paper proposes an interesting view on risk, I would recommend 'reject' due to concerns stated above.\n\n---\nThanks for the detailed response. I can agree that the observation function gamma in the form of addition can model many noisy signals. However, the argument in the paper that the proposed method works for an arbitrary gamma still lacks experimental validation. So I would like to keep my recommendation. My other concerns have been addressed.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "6Lm59l5MUY", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a theoretical setting for context-agnostic few-shot learning. While I do acknowledge that the background signals should not be involved or impact the predictions, I am concerned why this should have a new setting specific for this case? Can the authors elaborate more on the motivation for this theoretical setting and how this setting can benefit the application in the real world? To me, I appreciate that the synthetic data can provide more supervision for few-shot image classification tasks, while I don't understand why this should set up a new setting here.\n\nIn the experiments, the authors show the comparison to previous methods in Appendix Table 2, 3 , 4, and the proposed Context Agnostic method seems not outperforms previous baselines and sometimes even perform much worse than baselines. For example, 7% lower than GCR on Omniglot. Can the authors explain the  benefits of the proposed model please? Also, I notice the authors mainly use two simple types datasets, traffic signs and hand-written characters, while I am expecting to see the performance on more realistic datasets, such as ImageNet. Can the authors explain why they choose these two types of datasets? Are they easier to separate the context / background ?\n\nAs this paper seems more related to synthetic data, I am curious how synthetic data-based few-shot learning methods perform, such as [a,b]\n\n[a]Eli Schwartz, Leonid Karlinsky, Joseph Shtok, Sivan Harary, Mattias Marder, Abhishek Kumar,\nRogerio Feris, Raja Giryes, and Alex Bronstein. Delta-encoder: an effective sample synthesis\nmethod for few-shot object recognition. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman,\nN. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31,\npp. 2845–2855. 2018.\n\n[b]Jian Zhang, Chenglong Zhao, Bingbing Ni, Minghao Xu, and Xiaokang Yang. Variational few-shot\nlearning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV),\nOctober 2019.\n\nAfter rebuttal:\nThanks very much for the detailed response! I do agree with the AC's comment that *I don't see a drawback the use of only one sample and no info about the target, rather I'd like to know if the proposed method could use more than one sample in order to make the comparison with SoA methods fair and still be competitive or outperforming them.* Also,  I would recommend the authors to have more experimental validation and resubmission. Thus, I keep my score.\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses the task of context-agnostic learning and presents an algorithm to solve the problem while assuming the ability to sample objects and contexts independently. It is reported a theoretical ground proposing to decompose factors contributing to the classification risk in context bias and object error. The method makes use of only one synthetic sample in training, still being able to generalize well.

The paper received contrasting reviews, 2 positive (7 and 6) and 2 below threshold (5 and 5). R2, R3 and R4 raised similar issues, especially regarding the experimental validation, which is the main shortcoming of the work: addressing "simple" datasets only, no comprehensive comparative analysis only in relation to baselines (vanilla SGD) but not in relation to state-of-the-art methods, possibly slightly revised to accomplish the experimental protocol proposed in this work (authors claim that the originality of the work do not allow a proper comparison with SoA method as-is). Indeed, I deem R3's rating (6) a bit overestimated given the provided comments. 
AC does not see an issue the use of only one sample and no info about target, rather, it'd be interesting to know if the proposed method could use more than one sample in order to make the comparison with SoA methods fair, while assessing performance in comparative terms with SoA, to give value to the method also in relation to performance. 

Unfortunately the rebuttal did not lead an increase of the ratings, nor to better comments. 
After the rebuttal, R2 and R4 still remained below threshold; R1 was also not changing idea, remaining positive, and R3 did not react after rebuttal.

Overall, the AC deems this paper containing interesting contributions, but it is not sufficiently ready to be accepted at ICLR mainly because the experiental validation is not showing a fully convincing evaluation of the proposed approach (see above). 

Decision: Reject