Title: GENERALIZING ACROSS DOMAINS VIA CROSS-GRADIENT TRAINING

Abstract: We present CROSSGRAD, a method to use multi-domain training data to learn a classifier that generalizes to new domains. CROSSGRAD does not need an adaptation phase via labeled or unlabeled data, or domain features in the new domain. Most existing domain adaptation methods attempt to erase domain signals using techniques like domain adversarial training. In contrast, CROSSGRAD is free to use domain signals for predicting labels, if it can prevent overfitting on training domains. We conceptualize the task in a Bayesian setting, in which a sampling step is implemented as data augmentation, based on domain-guided perturbations of input instances. CROSSGRAD parallelly trains a label and a domain classifier on examples perturbed by loss gradients of each other's objectives. This enables us to directly perturb inputs, without separating and re-mixing domain signals while making various distributional assumptions. Empirical evaluation on three different applications where this setting is natural establishes that (1) domain-guided perturbation provides consistently better generalization to unseen domains, compared to generic instance perturbation methods, and that (2) data augmentation is a more stable and accurate method than domain adversarial training.

1. INTRODUCTION:

We investigate how to train a classification model using multi-domain training data, so as to generalize to labeling instances from unseen domains. This problem arises in many applications, viz., handwriting recognition, speech recognition, sentiment analysis, and sensor data interpretation. In these applications, domains may be defined by fonts, speakers, writers, etc. Most existing work on handling a target domain not seen at training time requires either labeled or unlabeled data from the target domain at test time. Often, a separate "adaptation" step is then run over the source and target domain instances, only after which target domain instances are labeled. In contrast, we consider the situation where, during training, we have labeled instances from several domains which we can collectively exploit so that the trained system can handle new domains without the adaptation step.

1.1. PROBLEM STATEMENT:

Let D be a space of domains. During training we get labeled data from a proper subset D ⊂ D of these domains. Each labeled example during training is a triple (x, y, d) where x is the input, y ∈ Y is the true class label from a finite set of labels Y and d ∈ D is the domain from which this example is sampled. We must train a classifier to predict the label y for examples sampled from all domains, including the subset D \ D not seen in the training set. Our goal is high accuracy for both in-domain (i.e., in D) and out-of-domain (i.e., in D \ D) test instances.

One challenge in learning a classifier that generalizes to unseen domains is that Pr(y|x) is typically harder to learn than Pr(y|x, d). While Yang & Hospedales (2015) addressed a similar setting, they assumed a specific geometry characterizing the domain, and performed kernel regression in this space. In contrast, in our setting, we wish to avoid any such explicit domain representation, appealing instead to the power of deep networks to discover implicit features.

Lacking any feature-space characterization of the domain, conventional training objectives (given a choice of hypotheses having sufficient capacity) will tend to settle to solutions that overfit on the set of domains seen during training. A popular technique in the domain adaptation literature to generalize to new domains is domain adversarial training (Ganin et al., 2016; Tzeng et al., 2017) . As the name suggests, here the goal is to learn a transformation of input x to a domain-independent representation, with the hope that amputating domain signals will make the system robust to new domains. We show in this paper that such training does not necessarily safeguard against over-fitting of the network as a whole. We also argue that even if such such overfitting could be avoided, we do not necessarily want to wipe out domain signals, if it helps in-domain test instances.

1.2. CONTRIBUTION:

In a marked departure from domain adaptation via amputation of domain signals, we approach the problem using a form of data augmentation based on domain-guided perturbations of input instances. If we could model exactly how domain signals for d manifest in x, we could simply replace these signals with those from suitably sampled other domains d to perform data augmentation. We first conceptualize this in a Bayesian setting: discrete domain d 'causes' continuous multivariate g, which, in combination with y, 'causes' x. Given an instance x, if we can recover g, we can then perturb g to g , thus generating an augmented instance x . Because such perfect domain perturbation is not possible in reality, we first design an (imperfect) domain classifier network to be trained with a suitable loss function. Given an instance x, we use the loss gradient w.r.t. x to perturb x in directions that change the domain classifier loss the most. The training loss for the y-predictor network on original instances is combined with the training loss on the augmented instances. We call this approach cross-gradient training, which is embodied in a system we describe here, called CROSSGRAD. We carefully study the performance of CROSSGRAD on a variety of domain adaptive tasks: character recognition, handwriting recognition and spoken word recognition. We demonstrate performance gains on new domains without any out-of-domain instances available at training time.

2. RELATED WORK:

Domain adaptation has been studied under many different settings: two domains (Ganin et al., 2016; Tzeng et al., 2017) or multiple domains (Mansour et al., 2009; Zhang et al., 2015) , with target domain data that is labeled (III, 2007; III et al., 2010; Saenko et al., 2010) or unlabeled (Gopalan et al., 2011; Gong et al., 2012; Ganin et al., 2016) , paired examples from source and target domain (Kuan-Chuan et al., 2017) , or domain features attached with each domain (Yang & Hospedales, 2016) . Domain adaptation techniques have been applied to numerous tasks in speech, language processing and computer vision (Woodland, 2001; Saon et al., 2013; Jiang & Zhai, 2007; III, 2007; Saenko et al., 2010; Gopalan et al., 2011; Li et al., 2013; Huang & Belongie, 2017; Upchurch et al., 2016) . However, unlike in our setting, these approaches typically assume the availability of some target domain data which is either labeled or unlabeled.

For neural networks a recent popular technique is domain adversarial networks (DANs) (Tzeng et al., 2015; 2017; Ganin et al., 2016) . The main idea of DANs is to learn a representation in the last hidden layer (of a multilayer network) that cannot discriminate among different domains in the input to the first layer. A domain classifier is created with the last layer as input. If the last layer encapsulates no domain information apart from what can be inferred from the label, the accuracy of the domain classifier is low. The DAN approach makes sense when all domains are visible during training. In this paper, our goal is to generalize to unseen domains. Domain generalization is traditionally addressed by learning representations that encompass information from all the training domains. Muandet et al. (2013) learn a kernel-based representation that minimizes domain dissimilarity and retains the functional relationship with the label. Gan et al. (2016) extends Muandet et al. (2013) by exploiting attribute annotations of examples to learn new feature representations for the task of attribute detection. In Ghifary et al. (2015) , features that are shared across several domains are estimated by jointly learning multiple data-reconstruction tasks.

Such representations are shown to be effective for domain generalization, but ignore any additional information that domain features can provide about labels. Domain adversarial networks (DANs) (Ganin et al., 2016) can also be used for domain generalization in order to learn domain independent representations. A limitation of DANs is that they can be misled by a representation layer that over-fits to the set of training domains. In the extreme case, a representation that simply outputs label logits via a last linear layer (making the softmax layer irrelevant) can keep both the adversarial loss and label loss small, and yet not be able to generalize to new test domains. In other words, not being able to infer the domain from the last layer does not imply that the classification is domain-robust.

Since we do not assume any extra information about the test domains, conventional approaches for regularization and generalizability are also relevant. Xu et al. (2014) use exemplar-based SVM classifiers regularized by a low-rank constraint over predictions. Khosla et al. (2012) also deploy SVM based classifier and regularize the domain specific components of the learners. The method most related to us is the adversarial training of (Szegedy et al., 2013; Goodfellow et al., 2014; Miyato et al., 2016) where examples perturbed along the gradient of classifier loss are used to augment the training data. perturbs examples. Instead, our method attempts to model domain variation in a continuous space and perturbs examples along domain loss.

Our Bayesian model to capture the dependence among label, domains, and input is similar to Zhang et al. (2015, Fig. 1d ), but the crucial difference is the way the dependence is modeled and estimated. Our method attempts to model domain variation in a continuous space and project perturbation in that space to the instances.

3. OUR APPROACH:

We assume that input objects are characterized by two uncorrelated or weakly correlated tags: their label and their domain. E.g. for a set of typeset characters, the label could be the corresponding character of the alphabet ('A', 'B' etc) and the domain could be the font used to draw the character. In general, it should be possible to change any one of these, while holding the other fixed. obtained by a complicated, un-observed mixing1 of y and g. In the training sample L, nodes y, d, x are observed but L spans only a proper subset D of the set of all domains D. During inference, only

x is observed and we need to compute the posterior Pr(y|x). As reflected in the network, y is not independent of d given x. 

where ĝ = argmax g Pr(g|x) is the inferred continuous representation of the domain of x.

This assumption is key to our being able to claim generalization to new domains even though most real-life domains are discrete. For example, domains like fonts and speakers are discrete, but their variation can be captured via latent continuous features (e.g. slant, ligature size etc. of fonts; speaking rate, pitch, intensity, etc. for speech). The assumption states that as long as the training domains span the latent continuous features we can generalize to new fonts and speakers.

We next elaborate on how we estimate Pr(y|x, ĝ) and ĝ using the domain labeled data L = {(x, y, d)}. The main challenge in this task is to ensure that the model for Pr(y|x, ĝ) is not overfitted on the inferred g's of the training domains. In many applications, the per-domain Pr(y|x, d) is significantly easier to train. So, an easy local minima is to choose a different g for each training d and generate separate classifiers for each distinct training domain. We must encourage the network to stay away from such easy solutions. We strive for generalization by moving along the continuous space g of domains to sample new training examples from hallucinated domains. Ideally, for each training instance (x i , y i ) from a given domain d i , we wish to generate a new x by transforming its (inferred) domain g i to a random domain sampled from P (g), keeping its label y i unchanged. Under the domain continuity assumption (A1), a model trained with such an ideally augmented dataset is expected to generalize to domains in D \ D.

However, there are many challenges to achieving such ideal augmentation. To avoid changing y i , it is convenient to draw a sample g by perturbing g i . But g i may not be reliably inferred, leading to a distorted sample of g. For example, if the g i obtained from an imperfect extraction conceals label information, then big jumps in the approximate g space could change the label too. We propose a more cautious data augmentation strategy that perturbs the input to make only small moves along the estimated domain features, while changing the label as little as possible. We arrive at our method as follows.

ĝ

x rJ d rJ l J l J d y d C ✓ l G ✓ 1 d S ✓ 2 d ✓ d = (✓ 1 d , ✓ 2 d )

Figure 2 : CROSSGRAD network design.

Domain inference.:

We create a model G(x) to extract domain features g from an input x. We supervise the training of G to predict the domain label

d i as S(G(x i ))

where S is a softmax transformation. We use J d to denote the cross-entropy loss function of this classifier. Specifically, J d (x i , d i ) is the domain loss at the current instance.

Domain perturbation. Given an example (x i , y i , d i ), we seek to sample a new example (x i , y i ) (i.e., with the same label y i ), whose domain is as "far" from d i as possible.

To this end, consider setting

x i = x i + ∇ xi J d (x i , d i ).

Intuitively, this perturbs the input along the direction of greatest domain change2 , for a given budget of ||x ix i ||.

However, this approach presupposes that the direction of domain change in our domain classifier is not highly correlated with the direction of label change. To enforce this in our model, we shall train the domain feature extractor G to avoid domain shifts when the data is perturbed to cause label shifts.

What is the consequent change of the continuous domain features ĝi ? This turns out to be JJ ∇ ĝi J d (x i , d i ), where J is the Jacobian of ĝ w.r.t. x. Geometrically, the JJ term is the (transpose of the) metric tensor matrix accounting for the distortion in mapping from the x-manifold to the ĝ-manifold. While this perturbation is not very intuitive in terms of the direct relation between ĝ and d, we show in the Appendix that the input perturbation ∇ xi J d (x i , d i ) is also the first step of a gradient descent process to induce the "natural" domain perturbation

∆ĝ i = ∇ ĝi J d (x i , d i ).

The above development leads to the network sketched in Figure 2 , and an accompanying training algorithm, CROSSGRAD, shown in Algorithm 1. Here X, Y, D correspond to a minibatch of instances. Our proposed method integrates data augmentation and batch training as an alternating sequence of steps. The domain classifier is simultaneously trained with the perturbations from the label classifier network so as to be robust to label changes. Thus, we construct cross-objectives J l and J d , and update their respective parameter spaces. We found this scheme of simultaneously training both networks to be empirically superior to independent training even though the two classifiers do not share parameters. 

X d := X + l • ∇ X J d (X, D; θ d ) 7: X l := X + d • ∇ X J l (X, Y ; θ l ) 8: θ l ← θ l -η∇ θ l ((1 -α l )J l (X, Y ; θ l ) + α l J l (X d , Y ; θ l )) 9: θ d ← θ d -η∇ θ d ((1 -α d )J d (X, D; θ d ) + α d J d (X l

4. EXPERIMENTS:

In this section, we demonstrate that CROSSGRAD provides effective domain generalization on four different classification tasks under three different model architectures. We provide evidence that our Bayesian characterization of domains as continuous features is responsible for such generalization. We establish that CROSSGRAD's domain guided perturbations provide a more consistent generalization to new domains than label adversarial perturbation (Goodfellow et al., 2014) which we denote by LABELGRAD. Also, we show that DANs, a popular domain adaptation method that suppresses domain signals, provides little improvement over the baseline (Ganin et al., 2016; Tzeng et al., 2017) .

We describe the four different datasets and present a summary in Table 3 .

Character recognition across fonts. We created this dataset from Google Fonts3 . The task is to identify the character across different fonts as the domain. The label set consists of twenty-six letters of the alphabet and ten numbers. The data comprises of 109 fonts which are partitioned as 65% train, 25% test and 10% validation folds. For each font and label, two to eighteen images are generated by randomly rotating the image around the center and adding pixel-level random noise. The neural network is LeNet (LeCun et al., 1998) Handwriting recognition across authors. We used the LipiTk dataset that comprises of handwritten characters from the Devanagari script4 . Each writer is considered as a domain, and the task is to recognize the character. The images are split on writers as 60% train, 25% test and 15% validation. The neural network is the same as for the Fonts dataset above.

MNIST across synthetic domains. This dataset derived from MNIST was introduced by Ghifary et al. (2015) . Here, labels comprise the 10 digits and domains are created by rotating the images in multiples of 15 degrees: 0, 15, 30, 45, 60 and 75. The domains are labeled with the angle by which they are rotated, e.g., M15, M30, M45. We tested on domain M15 while training on the rest. The network is the 2-layer convolutional one used by Motiian et al. (2017) .

Spoken word recognition across users. We used the Google Speech Command Dataset5 that consists of spoken word commands from several speakers in different acoustic settings. Spoken words are labels and speakers are domains. We used 20% of domains each for testing and validation.

The number of training domains was 100 for the experiments in Table 4 . We also report performance with varying numbers of domains later in Table 8 . We use the architecture of Sainath & Parada (2015) 6 .

For all experiments, the set of domains in the training, test, and validation sets were disjoint. We selected hyper-parameters based on accuracy on the validation set as follows. For LABELGRAD the parameter α was chosen from {0.1, 0.25, 0.75, 0.5, 0.9} and for CROSSGRAD we chose α l = α d from the same set of values. We chose ranges so that L ∞ norm of the perturbations are of similar sizes in LABELGRAD and CROSSGRAD. The multiples in the range came from {0.5, 1, 2, 2.5}.

The optimizer for the first three datasets is RMS prop with a learning rate (η) of 0.02 whereas for the last Speech dataset it is SGD with η = 0.001 initially and 0.0001 after 15 iterations. In CROSSGRAD networks, g is incorporated in the label classifier network by concatenating with the output from the last but two hidden layer.

4.1. OVERALL COMPARISON:

In Table 4 we compare CROSSGRAD with domain adversarial networks (DAN), label adversarial perturbation (LABELGRAD), and a baseline that performs no special training. For the MNIST dataset the baseline is CCSA (Motiian et al., 2017) and D-MTAE (Ghifary et al., 2015) . We observe that, for all four datasets, CROSSGRAD provides an accuracy improvement. DAN, which is designed specifically for domain adaptation, is worse than LABELGRAD, which does not exploit domain signal in any way. While the gap between LABELGRAD and CROSSGRAD is not dramatic, it is consistent as supported by this 

4.2. WHY DOES CROSSGRAD WORK?:

We present insights on the working of CROSSGRAD via experiments on the MNIST dataset where the domains corresponding to image rotations are easy to interpret.

In Figure 6a we show PCA projections of the g embeddings for images from three different domains, corresponding to rotations by 30, 45, 60 degrees in green, blue, and yellow respectively. The g embeddings of domain 45 (blue) lies in between the g of domains 30 (green) and 60 (yellow) showing that the domain classifier has successfully extracted continuous representation of the domain even when the input domain labels are categorical. Figure 6b shows the same pattern for domains 0, 15, 30. Here again we see that the embedding of domain 15 (blue) lies in-between that of domain 0 (yellow) and 30 (green).

Next, we show that the g perturbed along gradients of domain loss, does manage to generate images that substitute for the missing domains during training. For example, the embeddings of the domain 45, when perturbed, scatters towards the domain 30 and 60 as can be seen in Figure 6c : note the scatter of perturbed 45 (red) points inside the 30 (green) zone, without any 45 (blue) points. Figure 6d depicts a similar pattern with perturbed domain embedding points (red) scattering towards domains 30 and 0 more than unperturbed domain 15 (blue). For example, between x-axis -1 and 1 dominated by the green domain (domain 30) we see many more red points (perturbed domain 15) than blue points (domain 15). Similarly in the lower right corner of domain 0 shown in yellow. This highlights the mechanism of CROSSGRAD working; that it is able to augment training with samples closer to unobserved domains.

Finally, we observe in Figure 7 that the embeddings are not correlated with labels. For both domains 30 and 45 the colors corresponding to different labels are not clustered. This is a consequence of CROSSGRAD's symmetric training of the domain classifier via label-loss perturbed images.

4.3. WHEN IS DOMAIN GENERALIZATION EFFECTIVE?:

We next present a couple of experiments that provide insight into the settings in which CROSSGRAD is most effective.

First, we show the effect of increasing the number of training domains. Intuitively, we expect CROSSGRAD to be most useful when training domains are scarce and do not directly cover the test domains. We verified this on the speech dataset where the number of available domains is large.

We varied the number of training domains while keeping the test and validation data fixed. Table 8 summarizes our results. Note that CROSSGRAD outperforms the baseline and LABELGRAD most significantly when the number of training domains is small (40). As the training data starts to cover more and more of the possible domain variations, the marginal improvement provided by CROSS-GRAD decreases. In fact, when the models are trained on the full training data (consisting of more than 1000 domains), the baseline achieves an accuracy of 88.3%, and both CROSSGRAD and LA-BELGRAD provide no gains7 beyond that. DAN, like in other datasets, provides unstable gains and is difficult to tune. LABELGRAD shows smaller relative gains than CROSSGRAD but follows the same trend of reducing gains with increasing number of domains.

In general, how CROSSGRAD handles multidimensional, non-linear involvement of g in determining x is difficult to diagnose. To initiate a basic understanding of how data augmentation supplies CROSSGRAD with hallucinated domains, we considered a restricted situation where the discrete domain is secretly a continuous 1-d space, namely, the angle of rotation in MNIST. In this setting, We conducted leave-one-domain-out experiments by picking one domain as the test domain, and providing the others as training domains. In Table 9 we compare the accuracy of different methods. We also compare against the numbers reported by the CCSA method of domain generalization (Motiian et al., 2017) as reported by the authors. 

It becomes immediately obvious from:



5. CONCLUSION:

Domain d and label y interact in complicated ways to influence the observable input x. Most domain adaption strategies implicitly consider the domain signal to be extraneous and seek to remove its effect to train more robust label predictors. We presented CROSSGRAD, which considers them in a more symmetric manner. CROSSGRAD provides a new data augmentation scheme based on the y (respectively, d) predictor using the gradient of the d (respectively, y) predictor over the input space, to generate perturbations. Experiments comparing CROSSGRAD with various recent adversarial paradigms show that CROSSGRAD can make better use of partially correlated y and d, without requiring explicit distributional assumptions about how they affect x. CROSSGRAD is at its best when training domains are scarce and do not directly cover test domains well. Future work includes extending CROSSGRAD to exploit labeled or unlabeled data in the test domain, and integrating the best of LABELGRAD and CROSSGRAD into a single algorithm.

:

The dependence of y on x could also be via continuous hidden variables but our model for domain generalization is agnostic of such structure.

We use ∇x i J d as shorthand for the gradient ∇xJ d evaluated at xi.

https://fonts.google.com/?category=Handwriting&subset=latin

http://lipitk.sourceforge.net/hpl-datasets.htm

https://research.googleblog.com/2017/08/launching-speech-commands-dataset. html

https://www.tensorflow.org/versions/master/tutorials/audio_recognition

The gap in accuracy between the baseline and CROSSGRAD for the case of 1000 domains is not statistically significant according to the MAPSSWE test(Gillick & Cox, 1989).

Figures and Tables:

We use a Bayesian network to model the dependence among the label y, domain d, and input x as shown in Figure1. Variables y ∈ Y and d ∈ D are discrete and g ∈ R q , x ∈ R r lie in continuous multi-dimensional spaces. The domain d induces a set of latent domain features g. The input x is

Figure 1: Bayesian network to model dependence between label y, domain d, and input x.

CROSSGRAD training pseudocode. 1: Input: Labeled data {(x i , y i , d i )} M i=1 , step sizes l , d , learning rate η, data augmentation weights α l , α d , number of training steps n. 2: Output: Label and domain classifier parameters θ l , θ d 3: Initialize θ l , θ d {J l , J d are loss functions for the label and domain classifiers, respectively} 4: for n training steps do 5: Sample a labeled batch (X, Y, D) 6:

, D; θ d )) 10: end for If y and d are completely correlated, CROSSGRAD reduces to traditional adversarial training. If, on the other extreme, they are perfectly uncorrelated, removing domain signal should work well.The interesting and realistic situation is where they are only partially correlated. CROSSGRAD is designed to handle the whole spectrum of correlations.

g of domain 45 lies between g of 60 and 30. g of domain 15 lies between g of 0 and 30.

Adding g of perturbed domain 15 to above. Figure6: Comparing domain embeddings (g) across domains. Each color denotes a domain.

Comparing domain embeddings (g) across labels. Each color denotes a label. Unclustered colors indicates that label information is mostly absent from g.

need to make additional assumptions to ensure that we can generalize to a new d during testing. The assumption we make is that integrated over the training domains the distribution P (g) of the domain features is well-supported in L. More precisely, generalizability of a training set of domains D to the universe of domains D requires that

Table content: <table><tr><td>P (g) =</td><td>d∈D</td><td>P (g|d)P (d) ≈</td><td>d∈D</td><td>P (g|d)</td><td>P (d) P (D)</td><td>(A1)</td></tr><tr><td colspan="7">Under this assumption P (g) can be modeled during training, so that during inference we can infer</td></tr><tr><td>y for a given x by estimating</td><td/><td/><td/><td/><td/><td/></tr><tr><td>Pr(y|x) =</td><td/><td/><td/><td/><td/><td/></tr></table>

modified to have three pooled convolution layers instead of two and ReLU activations instead of tanh.

Table content: <table><tr><td>Dataset</td><td>Label</td><td>Domain</td></tr><tr><td>Font</td><td>36 characters</td><td>109 fonts</td></tr><tr><td colspan="3">Handwriting 111 characters 74 authors</td></tr><tr><td>MNIST</td><td>10 digits</td><td>6 rotations</td></tr><tr><td>Speech</td><td colspan="2">12 commands 1888 speakers</td></tr></table>

Summary of datasets.

Table content: <table/>

table and other experiments that we later describe. Accuracy on four different datasets. The baseline for MNIST is CCSA.Changing model architecture. In order to make sure that these observed trends hold across model architectures, we compare different methods with the model changed to a 2-block ResNet(He et al., 2016) instead ofLeNet (LeCun et al., 1998)  for the Fonts and Handwriting dataset in Table5. For both datasets the ResNet model is significantly better than the LeNet model. But even for the higher capacity ResNet model, CROSSGRAD surpasses the baseline accuracy as well as other methods like LABELGRAD .

Table content: <table><tr><td colspan="5">Method Name Fonts Handwriting MNIST Speech</td></tr><tr><td>Baseline</td><td>68.5</td><td>82.5</td><td>95.6</td><td>72.6</td></tr><tr><td>DAN</td><td>68.9</td><td>83.8</td><td>98.0</td><td>70.4</td></tr><tr><td>LABELGRAD</td><td>71.4</td><td>86.3</td><td>97.8</td><td>72.7</td></tr><tr><td>CROSSGRAD</td><td>72.6</td><td>88.6</td><td>98.6</td><td>73.5</td></tr></table>

Accuracy with varying model architectures.

Table content: <table/>

Accuracy on Google Speech Command Task. For each method we show the relative improvement (positive or negative) over the baseline.

Table content: <table><tr><td colspan="5">Method Name 40 domains 100 domains 200 domains 1000 domains</td></tr><tr><td>Baseline</td><td>62.2</td><td>72.6</td><td>79.1</td><td>88.3</td></tr><tr><td>LABELGRAD</td><td>+1.4</td><td>+0.1</td><td>-0.1</td><td>-0.7</td></tr><tr><td>DAN</td><td>+0.4</td><td>-2.2</td><td>+0.7</td><td>-3.3</td></tr><tr><td>CROSSGRAD</td><td>+2.3</td><td>+0.9</td><td>+0.7</td><td>-0.4</td></tr><tr><td colspan="5">a natural question is, given a set of training domains (angles), which test domains (angles) perform</td></tr><tr><td>well under CROSSGRAD?</td><td/><td/><td/><td/></tr><tr><td>Test domains → CCSA</td><td colspan="3">M0 M15 M30 M45 M60 M75 84.6 95.6 94.6 82.9 94.8 82.1</td><td/></tr><tr><td>D-MTAE</td><td colspan="3">82.5 96.3 93.4 78.6 94.2 80.5</td><td/></tr><tr><td>LABELGRAD</td><td colspan="3">89.7 97.8 98.0 97.1 96.6 92.1</td><td/></tr><tr><td>DAN</td><td colspan="3">86.7 98.0 97.8 97.4 96.9 89.1</td><td/></tr><tr><td>CROSSGRAD</td><td colspan="3">88.3 98.6 98.0 97.7 97.7 91.4</td><td/></tr></table>

Accuracy on rotated MNIST.

Table content: <table/>

Table 9 that CROSSGRAD is beaten in only two cases: M0 and M75, which are the two extreme rotation angles. For angles in the middle, CROSSGRAD is able to interpolate the necessary domain representation g via 'hallucination' from other training domains. Recall from Figures 6c and 6dthat the perturbed g during training covers for the missing test domains. In contrast, when M0 or M75 are in the test set, CROSSGRAD's domain loss gradient does not point in the direction of domains outside the training domains. If and how this insight might generalize to more dimensions or truly categorical domains is left as an open question.

Table content: <table/>

Peer review (iclr2018):
 The authors proposed to perturbed the estimated domain features for data augmentation, which is done by using the gradients of label and domain classification losses. The idea is interesting and new. And the paper is well written. My major conerns are as follows: 1. Section 3 seems a bit too lengthy or redundant to derive the data augmentation by introducing the latent domain features g. In fact, without g, it also makes sense to perturb x as done in lines 6 and 7 in Alg. 1. 2. The assumption in (A1) can only be guaranteed under certain theoretical conditions. The authors should provide more explanations to better convey the assumption to readers. Minors: 1. LabelGrad was not defined when firstly being used in Section 4. 2. Fig. 1 looks redundant.