Title: TOWARDS RESISTING LARGE DATA VARIATIONS VIA INTROSPECTIVE LEARNING

Abstract: Learning deep networks which can resist large variations between training and testing data is essential to build accurate and robust image classifiers. Towards this end, a typical strategy is to apply data augmentation to enlarge the training set. However, standard data augmentation is essentially a brute-force strategy which is inefficient, as it performs all the pre-defined transformations to every training sample. In this paper, we propose a principled approach to train networks with significantly improved resistance to large variations between training and testing data. This is achieved by embedding a learnable transformation module into the introspective networks (Jin et al., 2017; Lazarow et al., 2017; Lee et al., 2018) , which is a convolutional neural network (CNN) classifier empowered with generative capabilities. Our approach alternatively synthesizes pseudo-negative samples with learned transformations and enhances the classifier by retraining it with synthesized samples. Experimental results verify that our approach significantly improves the ability of deep networks to resist large variations between training and testing data and achieves classification accuracy improvements on several benchmark datasets, including MNIST, affNIST, SVHN and CIFAR-10.

1. INTRODUCTION:

Classification problems have rapidly progressed with advancements in convolutional neural networks (CNNs) (LeCun et al., 1989; Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; Szegedy et al., 2015; He et al., 2016; Huang et al., 2017) . CNNs are able to produce promising performance, given sufficient training data. However, when the training data is limited and unable to cover all the data variations in the testing data (e.g., the training set is MNIST, while the testing set is affNIST), the trained networks generalize poorly on the testing data. Consequently, how to learn deep networks which can resist large variations between training and testing data is a significant challenge for building accurate and robust image classifiers.

To address this issue, a typical strategy is to apply data augmentation to enlarging the training set, i.e., applying various transformations, including random translations, rotations and flips as well as Gaussian noise injection, to the existing training data. This strategy is very effective in improving the performance, but it is essentially a brute-force strategy which is inefficient, as it exhaustively performs all these transformations to every training samples. Neither is it theoretically formulated.

Alternatively, we realize that we can synthesize extra training samples with generative models. But, the problem is how to generate synthetic samples which are able to improve the robustness of CNNs to large variations between training and testing data. In this paper, we achieve this by embedding a learnable transformation module into introspective networks (Jin et al., 2017; Lazarow et al., 2017) , a CNN classifier empowered with generative capabilities. We name our approach introspective transformation network (ITN), which performs training by a reclassification-by-synthesis algorithm. It alternatively synthesizes samples with learned transformations and enhances the classifier by retraining it with synthesized samples. We use a min-max formulation to learn our ITN, where the transformation module transforms the synthesized pseudo-negative samples to maximize their variations to the original training samples and the CNN classifier is updated by minimizing the classification loss of the transformed synthesized pseudo-negative samples. The transformation modules are learned jointly with the CNN classifier, which augments training data in an intelligent manner by narrowing down the search space for the variations.

Our approach can work with any models that have generative and discriminative abilities, such as generative adversarial networks (GANs) and introspective networks. In this paper, we choose the introspective networks to generate extra training samples rather than GANs, because introspective networks have several advantages over GANs. Introspective learning framework maintains one single CNN discriminator that itself is also a generator while GANs have separate discriminators and generators. The generative and discriminative models are simultaneously refined over iterations. Additionally, Introspective networks are easier to train than GANs with gradient descent algorithms by avoiding adversarial learning.

The main contribution of the paper is that we propose a principled approach that endows classifiers with the ability to resist larger variations between training and testing data in an intelligent and efficient manner. Experimental results show that our approach achieves better performance than standard data augmentation on both classification and cross-dataset generalization. Furthermore, we also show that our approach has great abilities in resisting different types of variations between training and testing data.

2. RELATED WORK:

In recent years, a significant number of works have emerged focus on resisting large variations between training and testing data. The most widely adopted approach is data augmentation that applies pre-defined transformations to the training data. Nevertheless, this method lacks efficiency and stability since the users have to predict the types of transformations and manually applies them to the training set. Better methods have been proposed by building connections between generative models and discriminative classifiers (Friedman et al., 2001; Liang & Jordan, 2008; Tu et al., 2008; Jebara, 2012; Welling et al., 2003) . This type of methods capture the underlying generation process of the entire dataset. The discrepancy between training and test data is reduced by generating more samples from the data distribution.

GANs (Goodfellow et al., 2014) have led a huge wave in exploring the generative adversarial structures. Combining this structure with deep CNNs can produce models that have stronger generative abilities. In GANs, generators and discriminators are trained simultaneously. Generators try to generate fake images that fool the discriminators, while discriminators try to distinguish the real and fake images. Many variations of GANs have emerged in the past three years, like DCGAN (Radford et al., 2015) , WGAN (Arjovsky et al., 2017) and WGAN-GP (Gulrajani et al., 2017) . These GANs variations show stronger learning ability that enables generating complex images. Techniques have been proposed to improve adversarial learning for image generation (Salimans et al., 2016; Gulrajani et al., 2017; Denton et al., 2015) as well as for training better image generative models (Radford et al., 2015; Isola et al., 2017) .

Introspective networks (Tu, 2007; Lazarow et al., 2017; Jin et al., 2017; Lee et al., 2018) provide an alternative approach to generate samples. Introspective networks are closely related to GANs since they both have generative and discriminative abilities but different in various ways. Introspective networks maintain one single model that is both discriminative and generative at the same time while GANs have distinct generators and discriminators. Introspective networks focus on introspective learning that synthesizes samples from its own classifier. On the other hand, GANs emphasize adversarial learning that guides generators with separate discriminators. The generators in GANs are mappings from the features to the images. However, Introspective networks directly models the underlying statistics of an image with an efficient sampling/inference process.

3. METHOD:

We now describe the details of our approach in this section. We first briefly review the introspective learning framework proposed by (Tu, 2007) . This is followed by our the detailed mathematical explanation of our approach. In particular, we focus on explaining how our model generates unseen examples that complement the training dataset.

3.1. INTROSPECTIVE LEARNING:

We only briefly review introspective learning for binary-class problems, since the same idea can be easily extended to multi-class problems. Let us denote x ∈ R d as a data sample and y ∈ +1, -1 as the corresponding label of x. The goal of introspective learning is to model positive samples by learning the generative model p(x|y = +1). Under Bayes rule, we have

EQUATION

where p(y|x) is a discriminative model. For pedagogical simplicity, we assume p(y = 1) = p(y = -1) and this equation can be further simplified as:

EQUATION

The above equation suggests that a generative model for the positives p(x|y = +1) can be obtained from the discriminative model p(y|x) and a generative model p(x|y = -1) for the negatives. However, to faithfully learn p(x|y = +1), we need to have a representative p(x|y = -1), which is very difficult to obtain. A solution was provided in (Tu, 2007) which learns p(x|y = -1) by using an iterative process starting from an initial reference distribution of the negatives p 0 (x|y = -1), e.g., p 0 (x|y = -1) = U (x), a Gaussian distribution on the entire space R d . This is updated by

EQUATION

where q t (y|x) is a discriminative model learned on a given set of positives and a limited number of pseudo-negatives sampled from p t (x|y = -1) and Z t = qt(y=+1|x) qt(y=-1|x) p t (x|y = -1)dx is the normalizing factor. It has been proven that KL(p(x|y = +1)||p t+1 (x|y = -1)) ≤ KL(p(x|y = +1)||p t (x|y = -1))) (as long as each q t (y|x) makes a better-than-random prediction, the inequality holds) in (Tu, 2007) , where KL(•||•) denotes the Kullback-Leibler divergences, which implies p t (x|y = -1) t=∞ → p(x|y = +1). Therefore, gradually learning p t (x|y = -1) by following this iterative process of Eqn.(3), the samples drawn from x ∼ p t (x|y = -1) become indistinguishable from the given training samples.

3.2. LARGE VARIATIONS RESISTANCE VIA INTROSPECTIVE LEARNING:

Introspective Convolutional Networks (ICN) (Jin et al., 2017) and Wasserstein Introspective Neural Networks (WINN) (Lee et al., 2018) adopt the introspective learning framework and strengthen the classifiers by a reclassification-by-synthesis algorithm. However, both of them fail to capture large data variations between the training and testing data, since most of the generated pseudo-negatives are very similar to the original samples. But in practice, it is very common that the test data contain unseen variations that are not in training data, such as the same objects viewed from different angles and suffered from shape deformation.

To address this issue, we present our approach building upon the introspective learning framework to resist large data variations between training and test data. Arguably, even large training sets cannot fully contains all the possible variations. Our goal is to quickly generate extra training samples with beneficial unseen variations that is not covered by the training data to help classifiers become robust. We assume that we can generates such training samples by applying a transformation function T (• ; σ) parametrized by learnable parameters σ to the original training samples. Let us denote g(• ; ψ) as the function that maps the samples x to the transformation parameters σ, where ψ is the model parameter of the function g. The generated samples still belong to the same category of the original samples, since the transformation function T only changes the high-level geometric properties of the samples. The outline of training procedures of ITN is presented in Algorithm 1. We denote

S + = {(x + i , +1), i = 1...|S + |} as the positive sample set, T t (S + ) = {(x T i , +1), i = 1...|S + |, x T i = T (x + i ; σ t )}

as the transformed positive sample set at t th iteration with transformation parameter σ t and S - t = {(x - i , -1), i = 1...|S -|} as the set of pseudonegatives drawn from p t (x|y = -1). We then will describe the detail of the training procedure.

Discriminative model:

We first demonstrate the approach of building robust classifiers with given σ t . For a binary classification problem, at t th iteration, the discriminative model is represented as where θ t represents the model parameters at iteration t, and f t (x; θ t ) represents the model output at t th iteration. Note that, q t (y|x; θ t ) is trained on S + , T (S + ; σ t ) and pseudo-negatives drawn from p t (x|y = -1). In order to achieve stronger ability in resisting unseen variations, we want the distribution of T (S + ; σ t ) to be approximated by the distribution of pseudo negatives p t (x|y = -1), which can be achieved by minimizing the following Wasserstein distance (Gulrajani et al., 2017) :

qt(y|x; θt) = 1 1 + exp(-y ft(x; θt)) (4)

EQUATION

where ω t is the extra parameter together with f t (•; θ t ) to compute the Wasserstein distance. Each x in the set Xt is computed with the formula x = x T + (1 -)x -, where samples from uniform distribution U (0, 1), x T ∈ T (S + ; σ t ) and x -∈ S - t . The term λ(||∇ xf t (x; θ t )|| 2 -1) 2 is the gradient penalty that stabilizes the training procedure of the Wasserstein loss function.

The goal of the discriminative model is to correctly classify any given x + , x T and x -. Thus, the objective function of learning the discriminative model at iteration t is

EQUATION

The classifiers obtain the strong ability in resisting unseen variations by training on the extra samples while preserving the ability to correctly classify the original samples. We discussed the binary classification case above. When dealing with multi-class classification problems, it is needed to adapt the above reclassification-by-synthesis scheme to the multi-class case. We can directly follow the strategies proposed in (Jin et al., 2017) to extend ITN to deal with multi-class problems by learning a series of one-vs-all classifiers or a single CNN classifier.

Exploring variations. The previous section describes how to learn the robust classifiers when the σ t is given. However, σ t is unknown and there are huge number of possibilities to selecting σ t . Now, the problem becomes how do we learn the σ t in a principled manner and apply it towards building robust classifiers? We solve this issue by forming a min-max problem upon the Eqn.( 6):

EQUATION

Here, we rewrite J(θ) and D(θ, ω) in Eqn.(5) and Eqn.(6) as J(θ, σ) and D(θ, ω, σ), since σ is now an unknown variable. We also subsequently drop the subscript t for notational simplicity. This formulation gives us a unified perspective that encompasses some prior work on building robust classifiers. The inner maximization part aims to find the transformation parameter σ that achieves the high loss values. On the other hand, the goal of the outer minimization is expected to find the the model parameters θ that enables discriminators to correctly classify x T and ω allows the negative distribution to well approximate the distribution of T (S + ; σ t ) . However, direclty solving Eqn. 7 is difficult. Thus, we break this learning process and first find a σ * that satisfies

max σ E (x T ,y)∼T (S + ;σ) [-log(q(y|x T ))] + E x T ∼T (S + ;σ) [f (x T ; θ, ω)] + λE x∼ X [ ∇ xf (x; θ, ω) -1 2 2 ] (8)

where θ and ω are fixed. Then, θ and ω are learned with Eqn.( 6) by keep σ = σ * . Empirically, the first term in the Eqn. 8 dominates over other terms, therefore we can drop the second and third terms to focus on learning more robust classifiers. The purpose of empirical approximation is to find the σ * that make x T hard to classify correctly. Instead of enumerating all possible examples in the data augmentation, Eqn.(8) efficiently and precisely finds a proper σ that increase the robustness of the current classifiers.

We use g(• ; ψ) to learn σ, thus σ = g(x; ψ)+ζ, where ζ is random noise follows the standard normal distribution. The function parameter ψ is learned by Eqn.(8). Notably, following the standard backpropagation procedure, we need to compute the derivative of the transformation function T in each step. In other words, the transformation function T (•; σ) need to be differentiable with respect to the parameter ψ to allow the gradients to flow through the transformation function T when learning by backpropagation.

Generative model In the discriminative models, the updated discriminative model p(y|x) is learned by Eqn.( 6). The updated discriminative model is then used to compute the generative model by the Eqn.(3) in section 3.1. The generative is learned by maximizing the likelihood function p(x). However, directly learning the generative model is cumbersome since we only need samples from the latest generative model.

Let's denote initial reference distribution as p - 0 (x) and p n (x|y = -1) as p - n (x) for simplicity. Following standard introspective learning, we can approximate samples drawn from latest negative distribution by first sampling from p - 0 (x) and iteratively update them to approach desired samples. With p - 0 and Eqn.(3), we have

EQUATION

where Z t indicates the normalizing factor at t th iteration. The random samples x are updated by increasing maximize the log likelihood of p - n (x). Note that maximizing log p - n (x) can be simplified as maximizing n-1 t=1 qt(y=+1|x) qt(y=-1|x) since Z t and p - 0 are fixed in Eqn.( 9). From this observation, we can directly learn a model h t (x) such that

EQUATION

Taking natural logarithm on both side of the equation above, we can get ln h t (x) = f t (x; θ t ). Therefore, log p - n (x) can be rewritten as

EQUATION

where C is the constant computed with normalizing factors Z t . This conversion allows us to maximize log p - n (x) by maximizing n-1 t=1 f t (x; θ t ). By taking the derivative of log p - n (x), the update step ∇x is:

EQUATION

where η ∼ N (0, 1) is the random Gaussian noise and λ is the step size that is annealed in the sampling process. In practice, we update from the samples generated from previous iterations to reduce time and memory complexity. An update threshold is introduced to guarantee the generated negative images are above certain criteria, which ensures the quality of negative samples. We modify the update threshold proposed in (Lee et al., 2018) and keep track of the f t (x; θ t ) in every iteration.

In particular, we build a set D by recording E[f t (x; θ t )], where x ∈ S + in every iteration. We form a normal distribution N (a, b), where a and b represents mean and standard deviation computed from set D. The stop threshold is set to be a random number sampled from this normal distribution. The reason behind this threshold is to make sure the generated negative images are close to the majority of transformed positive images in the feature space.

generated examples from different perspectives. We then further explore the ability of our algorithm in resisting large variations with two challenging classification tasks and show the consistently better performance. Finally, we illustrate the flexibility of our architecture in addressing different types of unseen variations.

Baselines We compare our method against CNNs, DCGAN (Radford et al., 2015) , WGAN-GP (Gulrajani et al., 2017) , ICN (Jin et al., 2017) and WINN (Lee et al., 2018) . For generative models DCGAN and WGAN-GP, we adopt the evaluation metric proposed in (Jin et al., 2017) . The training phase becomes a two-step implementation. We first generate negative samples with the original implementation. Then, the generated negative images are used to augment the original training set. We train a simple CNN that has the identical structure with our method on the augmented training set. All results reported in this section are the average of multiple repetitions.

Experiment Setup All experiments are conducted with a simple CNN architecture (Lee et al., 2018) that contains 4 convolutional layers, each having a 5 × 5 filter size with 64 channels and stride 2 in all layers. Each convolutional layer is followed by a batch normalization layer (Ioffe & Szegedy, 2015) and a swish activation function (Ramachandran et al., 2018) . The last convolutional layer is followed by two consecutive fully connected layers to compute logits and Wasserstein distances.

The training epochs are 200 for both our method and all other baselines. The optimizer used is the Adam optimizer (Kingma & Ba, 2014) with parameters β 1 = 0 and β 2 = 0.9. Our method relies on the transformation function T (•) to convert the original samples to the unseen variations.

In the following experiments, we demonstrate the ability of ITN in resisting large variations with spatial transformers (STs) (Jaderberg et al., 2015) as our transformation function unless specified. Theoretically, STs can represent all affine transformations, which endows more flexible ability in resisting unseen variations. More importantly, STs are fully differentiable, which allows the learning procedure through standard backpropagation.

4.1. CLASSIFICATION:

To demonstrate the effectiveness of ITN, we first evaluate our algorithm on 4 benchmark datasets, MNIST (LeCun et al., 1998) , affNIST (Tieleman, 2013) , SVHN (Netzer et al., 2011) and CIFAR-10 (Krizhevsky & Hinton, 2009) . The MNIST dataset includes 55000, 5000 and 10000 handwritten digits in the training, validation and testing set, respectively. The affNIST dataset is a variant from the MNIST dataset and it is built by applying various affine transformations to the samples in MNIST dataset. To accord with the MNIST dataset and for the purpose of the following experiments, we reduce the size of training, validation and testing set to 55000, 5000 and 10000, respectively. SVHN is a real-world dataset that contains house numbers images from Google Street View and CIFAR-10 contains 60000 images of ten different objects from the natural scenes. The purpose of introducing these two datasets is to further verify the performance of ITN on real-world datasets. The data augmentation we applied in the following experiments is the standard data augmentation that includes affine transformations, such as rotation, translation, scaling and shear. As shown in Table 1 , our method achieves the best performance on all four datasets. The overall improvements can be explained by the fact that our method generates novel and reliable negative images (shown in Figure 1 ) that effectively strengthen the classifiers. The images we generate are different from the previous ones, but can still be recognized as the same class. The boosted performance in value on MNIST dataset is marginal perhaps because the performance on the MNIST dataset is close to saturation. The difference between training and testing split in MNIST dataset is also very small compared to other datasets. Moreover, the amount of improvements increases as the dataset becomes complicated. Based on the observation of the results, we conclude that our method has stronger ability in resisting unseen variations especially when the dataset is complicated. On the other hand, we can clearly observe that our method outperforms the standard data augmentation on all datasets. This result confirms the effectiveness and the advantages of our approach. Additionally, ITN does not contradict with data augmentation since ITN shows even greater performance when integrating with data augmentation techniques. The possible reason for this observation is that the explored space between ITN and augmentation is not overlapped. Therefore, the algorithm achieves greater performance when combining two methods together since more unseen variations are discovered in this case.

Figure 1 : Images generated by our method on MNIST, affNIST, SVHN and CIFAR-10 dataset. In each sector, the top row is the original images and the bottom row is our generated images.

4.2. QUANTITATIVE ANALYSIS:



4.2.1. CROSS DATASET GENERALIZATION:

We have shown the substantial performance improvements of ITN against other baselines on several benchmark datasets. In this section, we want to further explore the ability of our method in resisting large variations. We design a challenging cross dataset classification task between two significantly different datasets (cross dataset generalization). The training set in this experiment is the MNIST dataset while the testing set is the affNIST dataset. The difficulty of this classification tasks is clearly how to overcome such huge data discrepancy between training and testing set since the testing set includes much more variations. Another reason why we pick these two datasets as training and testing set is that they share the same categories, which ensures the challenge is only about resisting large data variations. et al., 2018) . DDTs are similar to STs in a way that both of them can be optimized through standard backpropagation.

As shown in:

We replace the ST modules with the DDT modules and check whether our algorithm can resist such type of transformation. Then, we include both STs and DDTs in our model and verify the performance again. Let MNIST dataset be the training set of the experiments while the testing sets are the MNIST dataset with different types of transformation applied. We introduce two types of testing sets in this section. The first one is the normal testing set with random DDT transformation only.

The second one is similar to the first one but includes both random DDT and affine transformations.

The DDT transformation parameters are drawn from N (0, 0.7 × I d ) as suggest in (Detlefsen et al., 2018) , where I d represents the d dimensional identity matrix. Then the transformed images are randomly placed in a 42 × 42 images. We replicate the same experiment on the CIFAR-10 dataset. We can make some interesting observations from the Table 4 . First, ITN can integrate with flexibly with DDT or DDT + ST to resist the corresponding variations. Second, ITN can resist partial unseen variations out of a mixture of transformations in the testing data. More importantly, the performance of ITN won't degrade when the model has transformation functions that doesn't match the type of variations in the testing data, e.g. ITN(DDT + ST) on testing data with DDT only. This observation allows us to apply multiple transformation functions in ITN without knowing the types of variations in the testing data and still maintain good performance.

5. CONCLUSION:

We proposed a principled and smart approach that endows the classifiers with the ability to resist larger variations between training and testing data. Our method, ITN strengthens the classifiers by generating unseen variations with various learned transformations. Experimental results show consistent performance improvements not only on the classification tasks but also on the other challenging classification tasks, such as cross dataset generalization. Moreover, ITN demonstrates its advantages in both effectiveness and efficiency over data augmentation. Our future work includes applying our approach to large scale datasets and extending it to generate samples with more types of variations.

:

EXPERIMENTSIn this section, we demonstrate the ability of our algorithm in resisting the large variations between training and testing data through a series of experiments. First, we show the outstanding classification performance of ITN on several benchmark datasets. We also analyze the properties of the

Figures and Tables:

Positive sample set S + , initial reference distribution p0(x|y = -1) and transformation function T 2: Output: Parameters θ, ω and ψ 3: Build S - 0 by sampling |S + | pseudo-negatives samples from p0(x|y = -1)

Table content: <table><tr><td colspan="2">4: initialize parameters θ, ω and ψ, set t = 1</td></tr><tr><td colspan="2">5: while not converge do</td></tr><tr><td>6: 7: 8:</td><td>for each x + i ∈ S + and x -i ∈ S -t do Compute transformation parameters σi = g(x + i ; ψ) Choose i ∼ U (0, 1) and compute xi = iT (x + i ; σi) + (1 -i)x -i</td></tr><tr><td>9:</td><td>end for</td></tr><tr><td>10:</td><td>Compute θ, ω by Eqn.(6)</td></tr><tr><td>11:</td><td>Compute ψ by Eqn.(8)</td></tr><tr><td>12:</td><td>Sample pseudo-negatives samples Zt = {z t i , i = 1, ..., |S + |} from p0(x|y = -1)</td></tr><tr><td>13:</td><td>Update all samples in Zt by Eqn.(12)</td></tr><tr><td>14:</td><td>Augment pseudo-negatives sample set S -t = S -t-1 ∪ {(z t i , -1), i = 1, ..., |S + |} and t = t + 1</td></tr><tr><td colspan="2">15: end while</td></tr></table>

testing errors of the classification experiments described in Section 4.2.1, where w/DA and w/o DA indicates whether data augmentation is applied.

Table content: <table/>

ITN has clear improvements over CNN, WGAN-GP and WINN. The amount of improvement is much larger than on the regular training and testing splits shown in Section 4.1. More importantly, our performance in this challenging task is still better than CNN with data augmentation. This encouraging result further verifies the efficiency and effectiveness of ITN compared with data augmentation. It's not surprising that data augmentation improves the performance by a significant margin since the space of unseen variations is huge. Data augmentation increases the classification performance by enumerating a large number of unseen samples, however, this bruteforce searching inevitably lacks efficiency and precision. 4.2.2 RESISTING ABILITY UNDER DATA PAUCITY Another way to evaluate the ability of resisting variations is to reduce the amount of training samples. Intuitively, the discrepancy between the training and testing sets increases when the number of samples in the training set shrinks. The purpose of this experiment is to demonstrate the potential of ITN in resisting unseen variations from a different perspective. We design the experiments where the training set is the MNIST dataset with only 0.1%, 1%, 10% and 25% of the whole training set while the testing set is the whole MNIST testing set. Each sample is randomly selected from the pool while keeps the number of samples per class same. Similarly, we repeat the same experiments on the CIFAR-10 dataset to further verify the results on a more complicated dataset. As shown in Table3, our method has better results on all tasks. This result is consistent with Section 4.2.1 and Section 4.1, which undoubtedly illustrate the strong ability of ITN in resisting unseen variations in the testing set. The constant superior performance over data augmentation also proves the efficiency of ITN.

Table content: <table><tr><td colspan="5">Under review as a conference paper at ICLR 2019</td><td/><td/><td/><td/></tr><tr><td/><td/><td colspan="2">w/o DA</td><td/><td/><td>w/ DA</td><td/><td/></tr><tr><td>Method</td><td>CNN</td><td>WGAN-GP</td><td>WINN</td><td>ITN</td><td>CNN</td><td>WGAN-GP</td><td>WINN</td><td>ITN</td></tr><tr><td colspan="2">0.1%(M) 36.50%</td><td>29.43%</td><td colspan="3">27.18% 16.47 % 18.07%</td><td>15.35%</td><td colspan="2">14.46% 12.67%</td></tr><tr><td>1%(M)</td><td>7.66%</td><td>6.86%</td><td>5.10%</td><td>3.48 %</td><td>4.48%</td><td>3.98%</td><td>3.66%</td><td>2.82%</td></tr><tr><td>10%(M)</td><td>2.02%</td><td>1.63%</td><td>1.49%</td><td>0.98 %</td><td>1.24%</td><td>1.18%</td><td>1.00%</td><td>0.92%</td></tr><tr><td>25%(M)</td><td>1.29%</td><td>1.13%</td><td>1.00%</td><td>0.78 %</td><td>0.83%</td><td>0.79%</td><td>0.77%</td><td>0.66%</td></tr><tr><td>0.1%(C)</td><td>81.99%</td><td>80.92%</td><td colspan="3">78.24% 72.50 % 79.04%</td><td>78.75%</td><td colspan="2">76.97% 70.43%</td></tr><tr><td>1%(C)</td><td>72.31%</td><td>71.34%</td><td colspan="3">69.79% 61.48 % 65.23%</td><td>64.84%</td><td colspan="2">63.26% 58.07%</td></tr><tr><td>10%(C)</td><td>59.02%</td><td>57.37%</td><td colspan="3">56.02% 45.06 % 47.75%</td><td>46.86%</td><td colspan="2">46.04% 42.62%</td></tr><tr><td>25%(C)</td><td>51.35%</td><td>49.01%</td><td colspan="3">48.43% 34.56 % 36.50%</td><td>35.46%</td><td colspan="2">34.29% 30.60%</td></tr></table>

testing errors of the classification tasks described in Section 4.2.2, where M and C represents the experiments conducted on the MNIST dataset and CIFAR-10 dataset, respectively.4.3 BEYOND SPATIAL TRANSFORMEREven though we utilize STs to demonstrate our ability in resisting data variations, our method actually has the ability to generalize to other types of transformations. Our algorithm can take other types of differentiable transformation functions and strengthen the discriminators in a similar manner. Moreover, our algorithm can utilize multiple types of transformations at the same time and provide even stronger ability in resisting variations. To verify this, we introduce another recently proposed work, Deep Diffeomorphic Transformer (DDT) Networks (Detlefsen

Table content: <table/>

testing errors of cross dataset classification experiments, where CNN (w/ DA) represents the CNNs with data augmentation.

Table content: <table/>

Peer review (iclr2019):
 This paper suggests the use of learned transformation networks, embedded within introspective networks to improve classification performance with synthesized examples. The authors cite a number of related works, and give a good introduction to both introspective learning, as well as the particular usage for large variation resistance. This discussion forms the majority of the paper, and while the explanation seems clear it would be nice to have a stronger dedicated section on exact relations to both GAN and potentially VAE mathematically. Any kind of discussion which could tie the thorough derivation to some of the contemporaries in generative modeling would help the accessibility of the paper. My primary concerns are from the experimental sections of the paper. The setup of the problem seems such that any strong baseline could be directly tested, since the final CNN is ultimately trained in a two stage setup on the aggregated dataset (as per subsection Baselines). Here it is also worth mentioning DAgger https://ri.cmu.edu/pub_files/2010/5/Ross-AIStats10-paper.pdf / https://www.cs.cmu.edu/~sross1/publications/Ross-AIStats11-NoRegret.pdf which is used in another context but in a similar way for improving imitation and online learning. Many of the baseline methods for table 1 seem far from what I would consider "strong" baselines. Given that the core proposal of the paper is improving classifiers, the importance of having high quality baselines cannot be overstated. Particularly, baseline CNN numbers for MNIST, SVHN, and CIFAR-10 are far from what has been seen in simple papers such as Wide ResNet https://arxiv.org/abs/1605.07146, ResNet https://arxiv.org/abs/1512.03385, Universum Perscription (which bears some resemblence to this work in high level concept) https://arxiv.org/abs/1511.03719, or even older work such as Maxout Networks http://proceedings.mlr.press/v28/goodfellow13.pdf . Particularly, these papers show examples of simple CNNs which outscore the best values reported in this table, on the same datasets. Running the same setup but with these improved classifiers as baselines would make a much stronger support for the core hypothesis that ITN can be used to improve strong classifiers. Table 2 seems to me an improper comparison. Methods such as zero-shot or meta-learning type approaches seem much more appropriate for testing cross-generalization improvement. In some sense, none of the tested methods besides ITN should even be able to cross-generalize well, so the fact that ITN does better here is not surprising to me. While this is a benefit of ITN as stated, seeing a comparison to methods deisgned for cross-generalization as well would make the results of Table 2 much stronger. Table 3 also seems have improper comparisons, in that there are a large number of works using semi-supervised generative models (Improved Techniques for Training GANS, which is already cited, SS-VAE https://arxiv.org/abs/1406.5298, Temporal Ensembling https://arxiv.org/abs/1610.02242, VAT https://ieeexplore.ieee.org/abstract/document/8417973/, Ladder Networks http://papers.nips.cc/paper/5947-semi-supervised-learning-with-ladder-networks, Auxiliary Deep Generative Models https://arxiv.org/abs/1602.05473, Manifold Tangent Classifier https://papers.nips.cc/paper/4409-the-manifold-tangent-classifier) for improved classification in low data domains. Adopting the same settings and comparing directly to these methods would greatly strengthen this section as well, as simple classifiers (as shown in many of these previous papers) are not generally great baselines for semi-supervised modeling. In addition, there should be a direct case where becoming robust to these kinds of transformations fails. For example, if my classification task is the rotation/position of a repositioned MNIST digit, becoming robust to these types of transformations may be harmful. An experiment or discussion about when robustness to large data variation might be harmful would be a good inclusion as well. As a more general comment, this method seems applicable outside of image domains, and it would be interesting to see it applied in other settings though it is likely outside the scope of this particular paper. Formatting of the paper (specifically spacing between sections and subsections) seems a bit off in general. If the authors are applying /vspace tricks to shrink spaces in the format, I would recommend taking a closer look at how and where, to make spacing more consistent over the whole document. Comparing to past ICLR papers (best papers, or high rated from past conferences) to see how they approach formatting could improve the visual appeal of this paper. Overall, this paper has a lot in its favor. The experimental section is thorough, if not as strong as I would like. The derivation of the method and motivation is clear, and there are a lot of avenues explored. However, as it currently stands the experimental sections should be stronger to really prove out the core claim "Our method, ITN strengthens the classifiers by generating unseen variations with various learned transformations." compared to other methods using generative and semi-supervised methods in a similar vein. In addition, the clarity and approachability of the paper could be improved by drawing a relation to parallel related work such as GAN or VAE in more detail.