Title: IMPROVING INVARIANCE AND EQUIVARIANCE PROP-ERTIES OF CONVOLUTIONAL NEURAL NETWORKS

Abstract: Convolutional Neural Networks (CNNs) learn highly discriminative representations from data, but how robust and structured are these representations? How does the data shape the internal network representation? We shed light on these questions by empirically measuring the invariance and equivariance properties of a large number of CNNs trained with various types of input transformations. We find that CNNs learn invariance wrt all 9 tested transformation types and that invariance extends to transformations outside the training range. We also measure the distance between CNN representations and show that similar input transformations lead to more similar internal representations. Transforms can be grouped by the way they affect the learned representation. Additionally, we also propose a loss function that aims to improve CNN equivariance.

1. INTRODUCTION:

The overwhelming success of Convolutional Neural Networks (CNNs) is generally attributed to their ability to learn task-specific representations from large quantities of data. This has led to many state of the art results in areas such as image classification (He et al., 2015) , semantic segmentation (Liu et al., 2015) , and game-playing Go agents (Silver et al., 2016) . However, our current understanding of CNN representations is limited, though there is a growing body of literature on visualization techniques for interpreting internal representations (Mahendran & Vedaldi, 2016; Dosovitskiy & Brox, 2015; Nguyen et al., 2016) .

In this work, we aim to understand the effect that the training data has on three key properties of representations: invariance, equivariance, and equivalence (Lenc & Vedaldi, 2015) . Invariance, with respect to a particular transformation (e.g. rotation by θ = 5), is achieved when the feature vectors for input images at a particular layer do not change when the inputs are transformed. This can be seen as a measure of the robustness of the representation. Equivariance is a generalization of invariance and allows the representation to change in predictable ways in response to input transformations. Equivariance is a rough measure of the structure of the representation space because we can reason about input space transformations in the more abstract representation space. Likewise, measuring equivalence among network representations trained on either the same or different data distributions gives insight into how the training data shapes the representation.

Data augmentation has been considered essential for top CNN performance since the seminal work of Krizhevsky et al. (2012) . When input images are stochastically perturbed, less overfitting is observed because all inputs are unique. Since then, others have experimented with various data augmentation strategies with great success (Howard, 2013; Wu et al., 2015; He et al., 2014) . While others have noted that data augmentation leads to greater representation invariance (Lenc & Vedaldi, 2015; Wu et al., 2015; Peng et al., 2014) , we measure the amount of invariance achieved in response to 9 types of transforms at various magnitudes. We additionally measure the equivariance and pairwise representation distance of the resulting CNNs.

In this work we quantify the invariance and equivariance properties of 70 CNNs on two datasets trained using different input transformations. We find the representation at the penultimate layer (fc7) is structured wrt almost all input transformations when no augmentation is used. Applying data augmentation effectively collapses the structure of the representation space, leading to input transformation invariance for all 9 transforms, including magnitudes of transforms that were not observed during training.

We measure the pairwise distances of the CNN representations by measuring the error of learned mappings between CNN representations. There is a strong bias towards similarity among CNNs trained with the same type of transformation, and we can group transformations based on their mutual average distances. Similar types of transformations (e.g. Color Jitter, Gaussian Noise) yield more similar representations compared to other dissimilar transform pairs. We also propose a way to increase the equivariance of a CNN by finetuning on a novel loss function that simultaneously minimizes classification error for both transformed and untransformed inputs. This leads to an increase in equivariance while improving or slightly decreasing performance on the untransformed images (dataset dependent), though this trade-off can be controlled by weighting the terms of the loss function.

2 RELATED WORKS Lenc & Vedaldi (2015) studied image representations and gave definitions for representation invariance, equivariance, and equivalence. They measured these properties wrt a limited number of transforms for the convolutional layers of the popular AlexNet architecture. We provide extensions to the definitions of these properties to measure the relative degree to which representations possess these properties. We also consider a wider variety of transformations and use the fc7 layer of AlexNet because it is the most invariant layer.

Convergent learning is the idea that identical networks (differing in random initialization) converge to the same representation (i.e.. representation equivalence). In Li et al. (2015) , the authors show that there are many corresponding convolutional filters among CNNs for one-to-one, one-to-many, and many-to-many relationships. In complementary fashion, we measure representation distances between last fully connected layers in CNNs trained over different transformations of their inputs.

In Krizhevsky et al. (2012) , applying random crops, horizontal flips and color jittering during training reduced the top-1 error by over 1%. Howard (2013) applied additional brightness, contrast, crop, and scale transformations at both training and test time to reduce validation top-1 error from 40.7% to 37.0%. Wu et al. (2015) adds rotation and photo filter effects such as lens distortion, vignetting, and color casting for further improvement. Similarly, we explore a series of 10 types of transformations applied at training time; however, we focus on how they affect the learned representation instead of optimizing classifier performance.

In Peng et al. (2014) , synthetic images rendered from 3D CAD models are used to explore invariance in object detection networks for factors such as pose, object texture, and foreground/background color. Training a CNN on images that vary in these aspect results in greater invariance than training with less data variety. 3D CAD models are also used in Aubry & Russell (2015) to show that upper layers of CNNs disentangle and (locally) linearize independent object factors such as object viewpoint, style, color, and scale.

ManiTest (Fawzi & Frossard, 2015) defines classifier invariance as the average magnitude of the minimal transform that causes predictions to change. In contrast, we examine internal CNN representations and measure both representation distance and classifier performance.

3. INVARIANCE, EQUIVARIANCE, AND EQUIVALENCE:

In this section, we provide definitions and intuition on the three properties of interest, borrowing basic definitions and notation from Lenc & Vedaldi (2015) . Though we focus on CNN image representations, the following applies to any kind of data representation.

3.1. EQUIVARIANCE AND INVARIANCE:

If the domain of interest (e.g. natural images) is X ⊂ R nxn , a function φ : R nxn → R m assigns a feature vector to every input image and thus defines an image representation. We will use X to denote the input space and Z the output space (i.e. φ : X → Z). φ may be equivariant to some transformations, but not others, so we say that φ is equivariant wrt g

: X → X iff ∃ M g : Z → Z, ∀x ∈ X : φ(gx) ≈ M g φ(x)

(1) This means that transformation by g in X corresponds to a transformation by M g (if it exists) in Z. Thus if M g exists, it informs us of the structure of the abstract space Z in terms of the concrete space X. If M g is the identity transform, then φ maps x and gx to the same point in Z, which leads to invariance wrt g (i.e. φ(gx) ≈ φ(x)). This is related to the idea of whether CNNs collapse (invariance) or linearize (equivariance) view manifolds of 3D objects (Bakry et al., 2015; Aubry & Russell, 2015) .

While the previous definitions suggest equivariance and invariance are all-or-nothing properties, we use the following definition to precisely define the ≈ used in Eq 1. We measure the equivariance of φ wrt a transform g as

Equivaraince φ (g; L) = min Mg 1 N N i=1 L(φ(x i ), M g φ(gx i )) (2)

where L is a suitable loss function such as classification error or L2 distance (Section 3.3). Invariance is similarly measured by fixing M g as the identity function.

In practice, it is difficult to maximize M g over the space of all functions, so we restrict M g to be a simple parametric function (e.g. linear) and learn the parameters from data. Thus our results can be considered a lower bound on the actual equivariance of the representation. Eq. 2 also uses the paradigm that M g attempts to undo in Z the transformation g performed in X. This is also done in Lenc & Vedaldi (2015) and allows us to compute classification based metrics by applying the same classifier to both φ(x) and M g φ(gx).

3.2. EQUIVALENCE AND REPRESENTATION DISTANCE:

In Lenc & Vedaldi (2015) , φ 1 is equivalent to φ 2 if φ 1 has the same information in the sense that

∃ E 1 : Z 1 → Z 2 , ∀x ∈ X : φ 2 (x) ≈ E 1 φ 1 (x)

(3) However, this relation is only symmetric (as the name equivalence implies) iff

E 1 is invertible. If E 1 is invertible, then E -1 1 satisfies φ 1 (x) ≈ E -1 1 φ 2 (x).

In the other direction, if the relation is symmetric, then ∃E 2 such that φ 1 (x) ≈ E 2 φ 2 (x) and such an E 2 is an inverse of E 1 .

In light of this, we propose renaming Eq 3 to be the sub-representation property with the the terminology that φ 2 is a sub-representation of φ 1 . We then define φ 1 and φ 2 to be equivalent iff an invertible E 1 exists. One example of sub-representation are two reprensetations characterized by (1) the RGB pixels of an image and (2) the corresponding grayscale intensities. The grayscale is a sub-representation of the RGB because we can recover the grayscale from the RGB, but not vice versa.

As with equivariance, we are interested in measuring the extent that two representations are distinct. Thus we define the distance between two representations as

EQUATION

As with M g , finding the optimal E 1 , E 2 is difficult, so we learn linear models from data. Choices for the loss function, L, are discussed in Section 3.3.

3.3. DISTANCE METRICS:

The two distance metric we employ are normalized euclidean distance in the representation space and classification error.

L L2 (r i , r i ) = ||r i -r i || 2 ||r i || 2 , L err (r i , r i ) = 1 -δ(y i , ŷi ), ŷi := argmax j C(r i ) j

where r i = φ(x i ), r i = M g (gx i ), C(r i ) is the predicted distribution of class labels, y i is the ground truth label of r i , and δ is the Kronecker delta function. For L L2 , we normalize each dimension of r to have similar magnitude, similar to the normalization of Li et al. (2015) . The classifier, C, is composed of the remaining layers of the CNN after φ. Table 1 : Each CNN is trained with one of these types of transformations over parameter ranges that differ in magnitude. Each sampled input to the CNN is stochastically perturbed by sampling transform parameters from the range and applying the transformation to the image.

4. IMPROVING EQUIVARIANCE:

In this section, we propose a loss function for training a network to increase its invariance and equivariance. CNNs for classification tasks are typically trained with a cross entropy loss, i.e.

L d (X, Y ) = 1 N N i=1 H(y i , C(φ(x i ))) (5)

where X is the training images, Y is the training labels, and H computes cross entropy.

L d can be augmented by a term that biases the representation φ to be equivariant to some set of transforms G = {g 1 , g 2 , ..., g J }. We propose the loss function, L e , which is minimized when the transformed images are both correctly classified and spatially close to the untransformed image in Z.

L e (X, Y ) = 1 N J N i=1 J j=1 H(y i , C(x ij )) + λ 1 ||x ij -φ(x i )|| 2 2 + λ 2 ||φ(g j x i ) -φ(x i )|| 2 2 (6)

where

x ij = M gj φ(g j x i ).

The first term of the sum is the classification loss of the transformed images, while the second and third terms respectively enforce equivariance and invariance. In our experiments, we set λ 1 = 50 4096 , λ 2 = 25 4096 , though the results do not appear highly sensitive to this particular setting. Each equivariance mapping M gj constitutes a new portion of the network that attempts to undo the transformation g j in the Z. As such, the parameters of M gj must be learned along with the parameters of the network.

Combining Eq. 5 and 6, we arrive at our combined loss:

EQUATION

We found that λ 3 = 0.3 provides good compromise between performance on the transformed and untransformed data, though λ 3 can be tuned for individual performance requirements.

5. EXPERIMENTAL SETUP:

Here we detail the experiments we conduct including the transformations, networks, and datasets.

For the first set of experiments, we simply measure the invariance and equivariance of the AlexNet1 architecture trained under various data augmentation schemes. We measure the representation of layer fc7 (denoted φ f c7 ) after applying the ReLU non-linearity and dropout2 .

5.1. TRANSFORMATIONS:

We examine a set of 10 transformation types. See Table 1 for brief descriptions or Appendix A for full details. We train 4 CNNs for each type of transform (only 2 for blurring, 1 for baseline) for a total of 35 networks per dataset. Each of the 4 CNN within a transform type is trained with different magnitudes of transformations. Training with data augmentation is performed by stochastically transforming each network input with a transform taken from a specified range.

After training, we measure invariance and equivariance wrt fixed transforms for each type. For example, the 4 CNNs trained on different ranges of rotations (θ -20, 20] ) all have their equivariance measured wrt rotations of θ = ±{2.5, 5, 10, 15, 20, 25, 30, 40}. As a baseline, we trained a model with no data augmentation and measured the invariance and equivariance of that network wrt all transforms.

∈ [-5, 5], θ ∈ [-10, 10], θ ∈ [-15, 15], θ ∈ [

5.2. EQUIVARIANCE MAPPING:

To measure equivariance wrt g j , we learn M gj (Eq. 2) and measure the classification accuracy over the transformed images. Due to data augmentation during training, the network may already be invariant wrt g j , so we model M gj as a linear residual mapping to bias it towards an identity mapping (He et al., 2015) . That is M gj (r) = σ(r + R(r)), where R is to be estimated from data and σ(x) = max(x, 0) is the ReLU non-linearity. We experimented with R as a linear mapping and as a neural network with a single hidden layer. In nearly all cases, the linear mapping outperformed the neural network, so we present results only for the linear mappings. Therefore, we have

EQUATION

where (W j , b j ) are parameters to be estimated from data.

Each M gj is trained using fully online SGD for 10 epochs with a static learning rate of 0.001, using an L2 weight decay penalty of 10 -5 . Momentum with β = 0.9 is used. For learning M gj , only (W j , b j ) are updated. The rest of the network parameters are treated as fixed. The loss to be minimized is ||M gj (φ f c7 (g j x)) -φ f c7 (x)|| 2 2 .

5.3. REPRESENTATION DISTANCE MAPPING:

Here we are interested in learning E 1 (or equivalently E 2 ) from Eq. 4 for each ordered pair (φ i , φ j ) in a set of of network representations {φ i }. We consider linear mappings with E 1 (r) = σ(W r + b) and estimate (W, b) from data. Hyper-parameters for learning are the same as in Section 5.2. The loss to be minimized for each ordered pair is

||φ i (x) -E 1 φ j (x)|| 2 2 .

5.4. FINETUNING EQUIVARIANCE:

For this experiment, we take the CNNs trained with the most extreme parameter ranges for each transform type and finetune the representation using Eq. 7 as the loss function. For each CNN, we selected 4-6 transformations within the training range to be the G in Eq. 6.

We finetune for 150,000 weight updates with a mini-batch size of 10 for RVL-CDIP and 100 for ILSVRC. The learning rate is 0.0005 and decays by a factor of 10 every 60,000 updates. Momentum of β = 0.9 was used with no weight-decay regularization.

5.5. DATASETS:

We use two large datasets with distinct properties. 

6. RESULTS:



6.1. MEASURING INVARIANCE AND EQUIVARIANCE:

Figure 1 shows the invariance and equivariance measurements for each type of transform on ILSVRC. Additional results for both ILSVRC and RVL-CDIP can be found in Appendix B.1. For each transform type, we compare the CNN trained with the most extreme transforms to the baseline model trained with no data augmentation. The transformations we measured range from no transform to approximately double the transformation magnitude seen during training.

In many instances (e.g. Blur, Noise), the baseline model's performance deteriorates rapidly even for mild transformations. The large difference between the baseline invariance (black line) and equivariance (red line) indicate that Z is structured wrt most transforms (Color is one exception).

For CNNs trained with data augmentation, the equivariance mapping does not improve performance for transforms that are observed during training. This suggests that Z may not be (linearly) struc- tured wrt those particular transforms possibly because the structure has been collapsed due to the training object. Some transforms (e.g. Noise, Blur, Shear), at large magnitudes, do show a gap between the equivariance and invariance lines, suggesting that Z is structured wrt these transforms.

Equivariance performance does drop off outside the training range, but not near as sharply as the baseline model (e.g Figs. 7a, 7c, 7f), so CNN representations generalize somewhat to unseen transforms.

Figure 2 shows how various magnitudes of training augmentation affect the invariance/equivariance properties of the network for select transforms. Other transforms behaved similarly. Larger training ranges yields greater robustness for a wider range of transforms.

6.2. REPRESENTATION DISTANCE:

In this second experiment, we measured the pairwise representation distances of 37 CNNs (34 w/ transforms and 3 baseline networks). Using equation 4 (L = L L2 ), we computed a pairwise distance matrix for the 37 network representations. We attempted a T-SNE embedding (Maaten & Hinton, 2008) visualization, but the distances seem intrinsically difficult to embed in 2D (see Appendix B.3).

Comments on the asymmetry of the distances measured can be found in Appendix B.3.

There is, however, a strong bias towards networks of the same training transformation to be nearer to each other compared to random chance. We performed a K-NN analysis by counting the percentage of same-transform pairs that are K nearest neighbors (Figure 3a ). Approximately 80% of sametransform pairs are within 5 neighbors of each other for RVL-CDIP (10 neighbors for ILSVRC). This indicates that networks trained with the same transform end up with more similar representations, though the strength of the transformation and network initialization also play a role.

We also visualize the average distance between CNNs grouped by transformation type (Figs 3c and 3b ). Patterns emerge in both datasets. In RVL-CDIP, Crop is the most unique transform because the CNNs learn features as a different static image scale. On the other hand, Crop is not as unique for ILSVRC because objects appear at multiple sizes, so the features learned are not targeted at a single size. Blur, especially for ILSVRC, seems to give unique representations because it changes the local textures from which the representation is derived in a bottom-up fashion. Refining the representation in a top-down manner may be a further avenue for improving CNN representation robustness.

Rotation, Shear, and Perspective transforms are mutually similar for both datasets. This is likely because all three move local pixels in a rigid fashion. Another mutually similar group is Baseline, Color Jitter, and Gaussian Noise, which all operate on pixels independently. Elastic Deformations are somewhat similar to Shear, but not to other transforms. Table 2 : Equivariance measurements for several CNNs on RVL-CDIP before and after finetuning using Eq. 7. The first row of each transform type shows performance over untransformed images. In most cases finetuning improves equivariance on the finetuning transforms and on the untransformed images.

6.3. IMPROVED EQUIVARIANCE:

For RVL-CDIP, finetuning using Eq. 7 improves both equivariance and performance on untransformed images by ∼0.5% (Table 2 ). This may be because the original network was trained with a wide range of transform parameters, so the representation caters to no particular set of transform parameters. By introducing M gj during finetuning, the representation can individualize to the untransformed image, while still avoiding the overfit that occurs without data augmentation. Results for ILSVRC were mixed, with finetuning improving equivariance for 3 of 9 transformations (see Table 4 in Appendix B).

7. CONCLUSION:

This work gives the results from a large scale empirical study into the invariance and equivariance properties of CNNs trained under different input transformations. We quantify these properties for 70 CNNs across 10 types of transforms for 2 large datasets. CNNs are able to learn invariance to all transforms tried, and this invariance/equivariance extends somewhat to transforms outside the training range. We show evidence that the CNN linearizes its representation wrt Rotation, Perspective, and Shear transforms. In general, the baseline CNN showed very little invariance to even moderate transformations. 

APPENDIX A TRANSFORM DETAILS:

For space reasons, exact details and parameterizations of the transforms used to train the CNNs were omitted from the main text, but are included here. Table 1 gives a brief explanation of the 9 transforms, but each transform will be explained in detail hereafter. Pixels that move into the image dimensions from outside the image (e.g. pixels rotated in) are set to be intensity 0 (see Figure 6b ).

A.1 COLOR JITTER:

Color Jitter adds a random value to each color channel (RGB or grayscale) of the input image.

The random value drawn is separate for each channel, but the same value is applied to each spatial location, essentially making each color either brighter or darker. If the addition of the random value causes a pixel to go outside the normal [0, 255] range, it is truncated back to the range.

During CNN training, the random values are drawn from a Gaussian Distribution parameterized by a mean µ and standard deviation σ. Four CNNs were trained, with µ = 0 and σ ∈ {5, 10, 15, 20}.

When measuring invariance/equivariance, we apply a deterministic color jitter transformation that is the same for all input images. We simply adjust the brightness of the image (all color channels tied) by a fixed amount in ±{2, 5, 10, 15, 20, 25, 30, 45}.

A.2 CROP:

During training with Crop transformations, we first resize the input images to a larger size (such as 240x240, 256x256, 288x288, or 320x320), and then take a 227x227 crop from the larger imgae.

The larger the original image, the more detail the CNN gets to see, but also a smaller percentage of the original image is captured in the input window. CNNs trained with other transformations have their input images resized to 227x227.

For measuring invariance/equivariance, the baseline transform for the Crop CNNs is the center crop, and the transforms are crops at other locations. For example, for invariance to upper left corner crops, we measured the difference in CNN activations between the network applied to the center crop and the network applied to the upper left crop. We measured 25 spatial locations arranged in a 5x5 grid. 

A.3 ELASTIC DEFORMATIONS:

Elastic Deformations are a way of locally distorting images by applying a smoothed random displacement field (Simard et al., 2003) . Image transformations can be characterized by a backward mapping of pixel locations in the output image to locations in the input image. For elastic deformations, first an random displacement field is sampled that would map output pixels to random input locations in their local neighborhood. The size of the local neighborhood is controlled by an α parameter. This displacement field is then smoothed using Gaussian filtering with a specified σ. This causes local regions of the displacement field to point in the same direction.

For training CNNs, we used a fixed σ and sampled α for each input image. The four CNNs we trained with Elastic Deformations can be described as

(α ∈ [0, 5], σ = 2), (α ∈ [0, 10], σ = 2), (α ∈ [0, 5], σ = 3), (α ∈ [0, 10], σ = 3).

When measuring equivariance/invariance (in contrast to training) we apply the same exact transform to every image. For elastic deformations, this means we applied the same displacement field to each image. We did this for a number of displacement fields of various parameter settings. As shown in Figure 7c These settings included every combination of α ∈ {5, 10, 15} and σ ∈ {2, 2.5, 3, 3.5}.

A.4 GAUSSIAN BLUR:

We used the standard Gaussian Blur transform which replaces each pixel by a weighted average of the neighboring pixels, where weight magnitudes are Gaussian wrt spatial distance. The shape of the Gaussian weighting function is controlled by σ, which is measured in pixels. We trained two CNNs with Gaussian Blur, where for each input image, we sampled a sigma from a uniform distribution (i.e. σ ∈ [0, 1.5] and σ ∈ [0, 3]).

For measuring equivariance/invariance, we used σ ∈ [0.5, 1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5]. We measured the invariance/equivariance of Gaussian Noise transforms defined by σ ∈ [2.5, 5, 7.5, 10, 12.5, 15, 20, 25, 30, 40] .

A.5 GAUSSIAN NOISE:



Gaussian:



A.6 MIRROR:

Mirroring refers to reflecting the input image over the horizontal or vertical image axes. We trained 3 CNNs with mirroring. The first performed horizontal mirroring with probability 0.5. The second performed vertical mirroring with probability 0.5. The third performed both types of mirroring with independent probability 0.5 for a total of 4 combinations of flips.

We measures the equivariance/invariance of all 4 combinations of flips.

A.7 PERSPECTIVE:

Perspective transforms are a class of transforms with the constraint that straight lines in the input image remain straight in the output image. One way to parameterize (8 parameters) a perspective transform is to specify the output coordinates of the input unit square, so that the unit square is mapped to an arbitrary quadrilateral.

When training a CNN, for each image we first sampled σ from a uniform distribution. Then we sampled the displacement of the coordinates of the unit square from a Gaussian Distribution with mean µ = 0 and standard deviation σ. Thus the output coordinates for the upper left corner of the unit square are x = N (0, σ), y = N (0, σ). For the lower right corner, they would be x = 1 + N (0, σ), y = 1 + N (0, σ). Then the image is warped according to the perspective transform defined by the output coordinates of the unit square.

The ranges used for training are sigma = [0, 0.001], [0, 0.002], [0, 0.003], [0, .004]. For measuring invariance/equivariance, we sampled 10 perspective transforms from a range of equally spaced σ values in increasing magnitude along the x-axis of Figure 7f . Examples of these transforms are given in Figure 6 .

A.8 ROTATION:

A Rotation transform is specified by an angle θ and is always performed about the center of the image. During training, we sample θ from a uniform distribution for each image. We trained four CNNs with rotations with θ drawn from the following ranges: [-5, 5], [-10, 10] , [-15, 15] , [-20, 20] . For measuring equivariance/invariance, we used θ ∈ ±{2.5, 5, 10, 15, 20, 25, 30, 45}.

A.9 SHEAR:

A Shear transform warps a square image into a parallelogram. We parameterize it by specifying a shear angle θ and an orientation that is either horizontal or vertical. While the original square image has corners that are 90 deg, the parallelogram that results has corners with angles of 90 + θ and 90 -θ. During training, we sample θ from a uniform distribution for each image. We trained four CNNs with shear with θ drawn from the following ranges: [-5, 5], [-10, 10] , [-15, 15] , [-20, 20] . Each shear has equal probability of horizontal or vertical orientation. For equivariance/invariance, we measured only horizontal shears with θ ∈ ±{5, 10, 15, 20, 25, 30}.

B ADDITIONAL RESULTS:



B.1 INVARIANCE AND EQUIVARIANCE:

In this section, we include additional results from the experiments in Section 6.1 that did not fit in the main text. Figure 1 presents invariance and equivariance measurements for the ILSVRC dataset (similar to Figure 7 ).

We observe several interesting differences to the results on RVL-CDIP. The baseline model for ILSVRC is significantly more invariant to vertical mirrors than horizontal mirrors, though the RVL-CDIP baseline performs equally bad on both types of mirrors. Brightness increases affect ILSVRC trained CNNs more, though this is likely because natural images are more susceptible to image saturation because they occupy the full spectrum of pixel intensities, while document images tend to be bi-modal. The RVL-CDIP CNN trained on Gaussian Noise was able to learn invariance, while the corresponding ILSVRC CNN did not. However, both the baseline and noise CNN for ILSVRC can correct for the noise with an equivariant mapping.

Figures 8 and 9 show the invariance/equivariance for all 70 CNNs trained with data augmentation.

In general, we see an overwhelming trend that training with greater variety of inputs transforms results in greater invariance and equivariance. The exception seems to be that the magnitude of color jittering does not highly affect the CNN robustness. Early CNN layers can easily become invariant to overall brightness changes by extracting information based on pixel differences, and prior work has shown that later layers encode information mostly by which neurons are non-zero, rather than the magnitude of the neuron activations (Agrawal et al., 2014) .

For Crop transforms (Figure 8c ), we examined 25 evenly spaced crops arranged in a 5x5 grid. The x-axis of the figure shows these crops ordered in scan-line order. The periodic nature of the graph shows that crops nearer the center of the image yield higher performance. There is virtually no difference between the invariance and equivariance measurements for the Crop CNNs, showing that the high level CNN features trained for classification are not predictive for image extrapolation. This is because the equivariance mapping for Crop transforms uses the CNN activations for some un-centered crop to predict the CNN activations over the center crop. We also report the accuracy of each CNN in Table 3 . While all transforms help improve invariance, we see that some input transformations help performance on the untransformed images, while other hurt performance. In particular, Crop and Shear transforms improve performance the most, while Elastic Deformations and Gaussian Blur seem to hurt performance most.

B.2 CROSS DATASET INVARIANCE/EQUIVARIANCE MEASUREMENTS:

In this section, we describe and give results for a new experiment. While all previous experiments measure invariance/equivariance on the same dataset that the CNN was trained on (though with different splits), here we use a new dataset to measure the invariance/equivariance of CNNs trained on RVL-CDIP. The new dataset, ANDOC, is also composed of grayscale document images, though while RVL-CDIP is composed of scanned office documents, ANDOC is composed of digitized Figure 10 : Cross dataset measurements of invariance/equivariance reveal that these properties do extend to data domains beside the one used to train the CNN, though the invariance is slightly weaker. These CNNs were trained on RVL-CDIP and measured using data from a dataset of historical documents. Graphs show 1 -L L2 , so higher is better.

historical documents, which differ significantly from modern office documents. As the RVL-CDIP CNNs were not trained to classify ANDOC documents, we resort to measuring the L L2 difference in the feature representations. In general, the measured invariance/equivariance is lower for ANDOC than for RVL-CDIP (see Figure 10 for examples). This makes sense because the CNNs learn to encode discriminative information about RVL-CDIP documents. However, the invariance and equivariance properties are not lost when switching to a new domain of data. In fact, for Mirror, Rotation, and Shear transforms, the invariance is higher, though this is likely due to less information about the input image being encoded for the new domain.

B.3 REPRESENTATION DISTANCE:

In this section, we include additional results from the experiments in Section 6.2 that did not fit in the main text. Figure 11 shows an example T-SNE embedding of the CNN representations. There appears to be very little cluster structure and the T-SNE embeddings appear to be sensitive to the random initialization of the embedding vectors. The results from multiple runs of the T-SNE Figure 12 : Absolute Differences in the one-way distances (Eq. 9) for all pairs of CNNs using L L2 distance. Row/Col labels are abbreviations for the transforms used to train the CNN. For context, a typical L L2 representation distance (Eq. 4) is roughly 0.50 for RVL-CDIP and 1.00 for ILSVRC. This means that one-way distance differences of approximately 10% and 5% are observed for RVL-CDIP and ILSVRC respectively.

embedding in general do not agree on either which points are outliers or on nearest neighbor pairs. This is likely because representation distances do not seem to follow the triangle inequality axiom.

While Eq. 4 yields a symmetric function by averaging two one-way distances, we find that for some pairs of CNNs, there is significant difference between the magnitudes of those one-way distances.

In other words,

EQUATION

is large. Figure 12 plots a heat map of Eq. 9 applied pairwise to each CNN for each dataset. Optimization difficulties are one possible cause of large values in Figure 12 . However, a more likely explanation is that some CNNs encode unique information about the input that cannot be predicted from the representations learned from other CNNs.

B.4 IMPROVED EQUIVARIANCE:

Table 4 shows the results of finetuning using Eq. 7 for CNNs trained with different types of data augmentation. Overall, finetuning improves 3 of 9 transform types. We believe that performance does not improve for some transforms because the dataset naturally contains examples of those transforms due to different object poses. This is not the case for RVL-CDIP because the document images are always scanned in the same manner. 

:

We used the reference CaffeNet architecture, which differs slightly fromKrizhevsky et al. (2012)

We follow the test time convention of halving neuron activations instead of zeroing out half the activations.

Figures and Tables:

Figure 1: Invariance and equivariance accuracy measurements for ILSVRC. Each plot compares a CNN trained with a particular transformation (blue line for equivariance, green for invariance) to the baseline model (red for equivariance, black for invariance) trained with no transformations. The x-axis ranges over different parameters for each type of transformation. Each point represents the error of the M g learned on that particular parameterization of the transformation.

Figure 3: (a) shows how often transforms of the same type are K-nearest neighbors. It reveals a strong bias towards networks trained with the same transformation having more similar representations. Heatmaps (b) and (c) show average distances between representations induced by the transforms. Lighter squares indicate greater distance, while darker squares indicate the transforms are closer together. Diagonal entries are distance 0, but were set to white for visualization purposes.

Figure 4: Example instances of the RVL-CDIP dataset.

Figure 5: Example Elastic Deformations transforms for which we measured invariance and equivariance in Figure 7c.

Noise is similar to Color Jitter except that we sample different random values for each spatial location. The strength of the noise is controlled by σ, which is measured in pixel intensity ([0, 255] scale). When training the four CNNs, σ is sampled from a uniform distribution for each input image. The four ranges used were [0, 5], [0, 10], [0, 15], [0, 20].

Figure 6: Example Perspective transforms for which we measured invariance and equivariance in Figure 7f. (a) is the original untransformed images, while (b-d) are ordered in increasing magnitude of transformation.

Figure7: Invariance and equivariance accuracy measurements for RVL-CDIP (similar to Figure1). Each plot compares a CNN trained with a particular transformation (blue line for equivariance, green for invariance) to the baseline model (red for equivariance, black for invariance) trained with no transformations.

Figure 8: Equivariance and Invariance measurements of CNNs for some transforms. The first two columns are for RVL-CDIP. The last two columns are for ILSVRC.

Figure 9: Equivariance and Invariance measurements of of CNNs for some transforms. The first two columns are for RVL-CDIP. The last two columns are for ILSVRC.

Figure 11: T-SNE visualization of the pairwise distance matrix for (a) RVL-CDIP and (b) ILSVRC. Best viewed electronically with zoom. Symbols and colors indicate the training transform of the network. The size of the markers indicate the relative magnitude of the training transformation.

Accuracy of all CNNs over untransformed images (center crops for Crop transforms).

Table content: <table/>

Equivariance measurements for several CNNs on ILSVRC before and after finetuning using Eq. 7. The first row of each transform type shows performance over untransformed images. Results are more mixed compared to RVL-CDIP.

Table content: <table><tr><td>Transform</td><td colspan="2">ILSVRC ILSVRC</td><td>Transform</td><td colspan="2">ILSVRC ILSVRC</td></tr><tr><td/><td colspan="2">Original Finetune</td><td/><td colspan="2">Original Finetune</td></tr><tr><td>Color Jitter</td><td>50.26</td><td>50.67</td><td>Crop center from 256x256</td><td>54.58</td><td>54.18</td></tr><tr><td>brightness +10</td><td>50.25</td><td>50.62</td><td>upper left corner</td><td>52.90</td><td>52.68</td></tr><tr><td>brightness -15</td><td>49.95</td><td>50.63</td><td>bottom right corner</td><td>53.13</td><td>52.66</td></tr><tr><td>Elastic Deformations</td><td>48.37</td><td>50.46</td><td>Gaussian Noise</td><td>49.46</td><td>49.88</td></tr><tr><td>σ = 2.5, α = 5</td><td>49.55</td><td>49.82</td><td>σ = 8</td><td>49.26</td><td>49.52</td></tr><tr><td>σ = 3, α = 10</td><td>49.44</td><td>49.64</td><td>σ = 16</td><td>48.79</td><td>48.93</td></tr><tr><td>Gaussian Blur</td><td>47.49</td><td>46.59</td><td>Mirror</td><td>53.91</td><td>53.63</td></tr><tr><td>σ = 1</td><td>47.12</td><td>46.03</td><td>Horz.</td><td>54.02</td><td>51.53</td></tr><tr><td>σ = 2</td><td>45.94</td><td>44.60</td><td>Horz. + Vert.</td><td>53.70</td><td>51.49</td></tr><tr><td>Perspective</td><td>52.59</td><td>52.44</td><td>Rotation</td><td>52.44</td><td>53.70</td></tr><tr><td/><td>53.31</td><td>51.99</td><td>θ = -10</td><td>53.88</td><td>53.11</td></tr><tr><td/><td>53.01</td><td>52.27</td><td>θ = 15</td><td>52.66</td><td>51.76</td></tr><tr><td>Shear</td><td>54.97</td><td>54.84</td><td/><td/><td/></tr><tr><td>Horz, θ = -10</td><td>54.15</td><td>52.62</td><td/><td/><td/></tr><tr><td>Vert, θ = -15</td><td>51.85</td><td>50.77</td><td/><td/><td/></tr></table>

Peer review (iclr2017):
This work presents an empirical study of the influence of different types of data augmentation on the performance of CNNs. It also proposes to incorporate additional loss functions to encourage approximate invariance or equivariance, and shows there are some benefits. The paper reads well and the objectives are clear. The study of invariances in CNNs is a very important topic, and advances in this area are greatly appreciated. The paper splits itself in two very different parts -- the empirical study of equivariances in existing CNNs, and the proposal of equivariance objectives. However, taken separately each of these two parts could be better executed. On the empirical study, its breath is relatively limited, and it*s hard to draw any far-reaching conclusions from it: - Only one network is studied; at least one other architecture would have made for better generalization. - Only one layer (fc7) is studied; this presents issues as the top layer is the most invariant. At least one convolutional layer (possibly more) should have been considered. - The reliance on the scanned text dataset does not help; however the ImageNet results are definitely very encouraging. It is nice to see how performance degrades with the degree of transformations, and the authors do interpret the results, but it would be better to see more analysis. There is only a limited set of conclusions that can be drawn from evaluating networks with jittered data. If the authors could propose some other interesting ways to assess the invariance and equivariance, they would potentially draw more insightful conclusions from it. On the proposed loss function, only a very quick treatment of it is given (Section 4, half a page). It does not differ too much from known invariance/equivariance objectives studied in the literature previously, e.g. Decoste and Scholkopf, *Training Invariant Support Vector Machines*, Machine Learning, 2002. I*m not sure that dividing the paper into these two different contributions is the best approach; they both feel a bit incomplete, and a full treatment of only one of them would make for an overall better paper.