Generalization to Out-of-Distribution transformations
Authors: Shanka Subhra Mondal
Zack Dulberg
Jonathan Cohen
Venue: ICLR 2022 Submitted (2022)
Keywords: Out of distribution
generalization
convolution
polar transformation

Abstract: Humans understand a set of canonical geometric transformations (such as translation, rotation and scaling) that support generalization by being untethered to any
specific object. We explored inductive biases that allowed artificial neural networks to learn these transformations in pixel space in a way that could generalize
out-of-distribution (OOD). Unsurprisingly, we found that convolution and high
training diversity were important contributing factors to OOD generalization of
translation to untrained shapes, sizes, time-points and locations, however these
weren’t sufficient for rotation and scaling. To remedy this we show that two more
principle components are needed 1) iterative training where outputs are fed back
as inputs 2) applying convolutions after conversion to log-polar space. We propose POLARAE which exploits all four components and outperforms standard
convolutional autoencoders and variational autoencoders trained iteratively with
high diversity wrt OOD generalization to larger shapes in larger grids and new
locations.

Under review as a conference paper at ICLR 2022
GENERALIZATION TO OUT-OF-DISTTIBUTIOO TRANS- FORMATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Humans understand of canonical geometric transformations (such as transla- tion, rotation and scaling) that support generalization by being untethered to any specific object. We explored inductive biases that allowed artificial neural net- works to learn these transformations in pixel space in way that could generalize -ittt--diittitttionn (OOD). Unsurprisingly, we found that convolution and high training diversity were important contributing factors to OOD generalization of translation to untrained shapes, sizes, time-points and locations, however these weren't t sufficient for rotation and scaling. To remedy this we show that two more principle components are needed 1) iterative training where outputs are fed back as inputs 2) applying convolutions after conversion to log-polar space. We propose POLARAE, an autoencoder operating in log-polar space which exploits all the four components and outperforms standard convolutional autoencoders and varia- tional autoencoders trained iteratively with high diversity wrt OOD generalization of rotation and scaling transformations.
INTRODUCTION
Humans have a unique ability to generalize beyond the scope of prior experience (Chollet 2019 Lake et al., 2017, Mareus 2001), while artificial agents struggle apply knowledge to distributions outside the convex hull of their training data (Santoro et al., 2018 Lake & BBaoon 2018). One way humans seem to achieve such generalization is by learning a set of primitive abstract structures, like the 1D ordinal scale (Summerfield et al., 2020) and grid-like representations (Hafting et al. 2005). These structures can also thought as symmetry functions: transformations that are some way invariant to the specific value of their arguments. As a concrete example, we can imagine moving any object around in space, regardless of its shape, color or size. fundamental question is how can such symmetry functions be learned?
We hypothesize that during development, humans learn a set of canonical transformations e.g. the translation, rotation and change in size of objects that are grounded in the sensorimotor system [Barsalou] 2008), and learned as a consequence of predicting the sensory results of primitive actions Battaglia al. 2013). The abilities translate, rotate or scale arbitrary objects then become our first abstract affordances (Gibson 2014). Indeed, infants that spend more time playing with blocks better abstract mental rotation tasks (Schwarzer al. 2013).
Our aim is to model part of this process by presenting artificial neural networks (LeCun et al. 1995) with 2-dimensional shapes, and training them predict the effect of translation, rotation or scaling of that shape in pixel space. This is analogous to predicting tactile or visual signals resulting from simple movement or saccade (Rao & Ballard 1999, Wolpert al., 1995). We do not explicitly model motor actions, but rather transform the image offline, and feed the result back to the model. We then test the extent to which such predictions generalize OOD along dimensions such as shape, size. location and time. Evidence of OOD generalization would suggest that the network has learned a symmetry function.
We assume that, in order to learn symmetry functions, must introduce principled inductive bi- ases. The first convolution. Second, to constrain the network to leam a primitive function that apply to any shape, we assume it requires exposure to sufficiently diverse set of examples. We therefore operationalize and vary 'diversity" as the number of distinct shapes present the training set. As the translational invariance built into convolution is well aligned with the task of translation

---

Under review as a conference paper at ICLR 2022
unsurprisingly, fully convolutional autoencoder with high training diversity acheives per- fect OOD generalization for translation to unseen shapes in larger grids and new locations (Figure D. However the two components weren't t sufficient for OOD generalization of rotation (Figure 2 and scaling. To remedy this we introduce two more principle components. First is the effect of "iteration' during training (output fed back input iteratively), based the idea that sequential applications of the same transformation should maintain the identity of an object (i.e. object per- manence (Piaget 2006)). Secondly, motivated by the strong OOD generalization of translation as depicted in Figure D and the fact that translation in log- log-polar space is equivalent to rotation and scaling in cartesian space (Figure of (Esteves et al., 2017] Tootell et al., 1982)). we hypothesize that performing convolutions in log polar space would help in the OOD generalization of rotation and scaling.
AE
Figure 1: Translation OOD (larger, hollow, shapes with 50 vertices 256x256 grid (trained 64x64 grid). Ground truth (GT) and autoencoder (AE) predictions displayed.
1:10
E:20
1:1
To:00
1::00
[::31
(::4)
T-X
+
*
(a) Size (Larger Grid)
(b) Location
Figure 2: Comparison of OOD rotation between AE trained with high diversity and GT across multiple times.
We summarize the following contributions:
We build flexible data generator that can produce and transform large variety of simple but structured images irregular polygons with variable size, location, and complexity.
We show that iteration has the ability to conserve the shape far past the time horizon seen during training and is necessary for better OOD generalization of rotation and scaling trans- formations along with convolution and high diversity.
We find an interesting tradcoff between diversity and iteration where each could partially make up for less of the other to produce better OOD generalization capabilities.
We novelly propose POLARAE, fully convolutional autoencoder build using concepts from polar transformer networks (Esteves et al. 2017) which exploits all the four compo- nents demonstrating better OOD generalization performance than standard autoencoder and a variational autoencoder in cartesian space.
RELATED WORK
PSYCHOLOGY
Humans perform mental transformations OOD (i.e. unseen shapes). For example, humans can mentally translate or rotate shapes at a steady rate (Shepard & Metzler, [97T) or scale abstract distances (Trope & Liberman, 2010). There also known mechanisms that might make these
2

---

Under review as a conference paper at ICLR 2022
transformations general. For example, there is known impact of diversity the generalization of properties in psychology. namely, the diversity effect (Osherson et al. 1990). Iteration is also a plausible mechanism in humans- psychological data for mental rotation suggest that we transform objects iteratively, since larger angles of rotation elicit longer reaction times (Shepard & Metzler 1971]. There is also evidence that discrete temporal context updating by recurrent thalamocortical loops serves predictive learning in the brain O'Reilly et al., (0144). Basically, iterated operations achieve generality since they are time-invariant, Finally, humans have built-in architectural trans- forms, such as the transformation of retinal images into a pollaol coordinate system (Maiello et al., 2020
2.2 MACHINE LEARNING
There have been several works that demonstrate that high diversity is necessary for OOD general- ization. Xu et al. (2020) provided theoretical guarantees that ReLU MLP networks extrapolate well with linear functions and sufficient diversity in their training set Sufficient diversity of input is also required for systematic generalization of neural network based reinforcement learning agents (Hill et al. 2019). Madan et al. (202T) showed that data diversity improves OOD category-viewpoie classification but didn't t focus the generative aspect of it using CNNs. There have also been works collecting large scale controlled synthetic datasets (Borji et al. 2016 Gross et al.] 2010 Qiu et al, 2017] to facilitate learning invariance different types of transtormations in deep neural networks. Following a similar motivation, we want CNNs to be able to learn such transformations and to be able to generate them on OOD examples. The idea of iterative training has also been explored in this domain. Using an iterative training technique similar to ours generative adversarial networks learned 3D rotations (but not scaling) (Galama & Mensink 2019). However there were limited assessment of how different amounts of iteration during training impacted extrapolation in time. Generally, predicting past the number of iterations seen during training has required building in conservation law of some sort (Cranmer et al. [2200] Greydanus et al.] (019). Kumar et al. (2020) showed that gradual self training at each iteration helps domain adaptation. In a different context, Kuchaiev & Ginsburg (2017) proposed iterative output re-feeding to train deep autoencoders for improved collaborative filterin.. Recently, the idea that iteration can achieve OOD generalization has been put forth in the context of problem solving (Schwarzschild et al. 2021). Kim et al. (2020) explored the effectiveness of log-polar space in achieving rotation invariance but was limited to classification and didn't t study other types of transformations like scaling.
METHOD
3.1 DATA GENERATOR
All training stimuli were shapes contained within a 64x64 pixel grid space. We constructed irregular N-sided polygonal shapes by first sampling N angular distances between 0 and 2nr, and then sam- pling a radial distance from a centroid ( [Feenter Bcenter) at each of these angles uniformly between 0 and a scale parameter r. This produced set vertices; pixels within the convexity of the vertices were set to 1, and pixels outside were set to O. There was also the option to produce 'hollow' shapes such that an interior cut-out of the shape was set to O, in order to produce a test-set of shapes with different distribution from the training set. This data generator was therefore capable of producing a combinatorically large set of possible shapes. Our procedure for shape generation most simi- lar to Attneave forms (Attneave & Arnouli, 1956) and also bears relation to the method of Fourier descriptors (Zhang & Lu| 2005), but was selected due to its computational speed and interpretable manipulations of shape parameters. Each shape used as the input neural network, and was transformed in one of the following ways to generate the target for training: for translation, shift pixels to the right; for rotation, rotate 27 radians clockwise; for scaling, increase radial length of the vertices by 0.1 (Figure B). These transformations were hard-coded but meant to represent a set of innate primitive actions.
3.2 MODEL ARCHITECTURE
We first describe the three baseline models followed by the proposed model:
3

---

Under review as a conference paper at ICLR 2022
pixels
pixel
Input
Target
Inout
n25 ad
Input
Target
Translation
Rotation
Scaling
Figure 3: Transformations of shapes produced from the data generator.
The first model was fully convolutional autoencoder. The encoder consisted of 3 convolutional layers, the first with 16 3x3 kernels. (stride=2, padding=1), the second with 32 3x3 kernels (stride=2, padding=1), 1), and the last with 64 7x7 kernels (stride=1). The decoder had 3 layers (padding=1, output padding=1) that inverted these operations with transposed convolution layers that were mirror images of the encoder layers. This produced an output with the same dimensions as the input image. All layers were followed by rectified linear (ReLU) activations. Being fully convolutional, could accept any input grid size. All weights were initialized using Xavier uniform initialization. We use AE to denote this model.
The second model The encoder and decoder architectures were similar to AE. We use VAE to denote this model.
fully convolutional variational autoencoder (Kingma & Welling, 2013).
The third model was VAE (Higgins et al. 2016). We used 4 since that learnt good disentangled representation of the data generative factors in the original work.
Polar Transform
Inverse Transform
Figure 4: POLARAE architecture: The input image is passed through polar transform module which consists of polar origin predictor followed by conversion to log-polar space. The obtained polar representation passed through a fully convolutional autoencoder and finally the output is passed through an inverse transform module to convert the image back to cartesian coordinate space.
The proposed model was a combination of polar transformer networks and the fully convolutional autoencoder described above. More specifically, given input image the polar origin predictor computed single channel feature map using fully convolutional network and the centroid of the heatmap was used as the polar origin. The polar transformer module converted the image to log polar coordinates using the predicted polar origin. The transformed image was then fed to AE. Since the rotation of the input image resulted in vertical shift in log polar space wrapping at boundary, we used wrap around padding on the vertical dimension and zero padding in the horizontal dimension before applying convolutions for the first two layers of the encoder. An inverse polar transformer module image space. The inverse polar transformer module used the sample points of the original input image used to compute the log-polar transform and mean approximation since single point in the original image contributed to multiple points in the log-polar space to convert the image back to cartesian coordinates. Since this is effectively an autoencoder operating the log-polar space use POLARAE to denote this model. Figure Aegureesshone shows the architecture.
used to convert the output of the AE from log polar space to the original
3.3 TRAINING AND TESTING
All networks were trained by providing randomly generated shapes with N E 20], and back- propagating the mean squared enror (MSE) loss between the output and the appropriately trans- formed target shapes. Weight updates were performed with the ADAM optimizer, using a learning rate of 2.5e-4, weight decay of le-5, and batch size (BS) of 32. Networks were all trained for Nsteps=100000, We trained separate networks for translation, rotation, and scaling, but the training procedure and the variation in diversity and iteration described next common to all networks.

---

Under review as a conference paper at ICLR 2022
For rotation, inputs were centered filled shapes with [10,10], Feenter = 32, yeenter 32, and xe radians, where & is the numbers of iterations. Note networks till that. These set of networks
targets were input shapes rotated clockwise by 25 k = 50 would amount to full rotation and we evaluate were tested for OOD generalization to hollow shapes and larger shapes in larger grids. The set of networks which were tested for OOD generalization of rotation new locations were trained on filled shapes in the upper left region with r E [5, 7]. Eeenter, Moenter E [14,18]. Finally the set of networks which were tested on larger shapes on the same grid size were trained on centered filled shapes with E [5, 7]. For scaling, inputs were centered filled shapes with E [5, 7]. and targets were input shapes with the radial distance of the vertices increased by 0.1k. These set of networks were tested for OOD generalization to hollow shapes and larger shapes in larger grids. To test OOD generalization at new locations, networks were trained with r E [5, 7]: Teenter, yeenter E [22,26]. To generate the targets for this case the radial distances were increased by 0.08k.
To vary diversity (D), we trained a separate set of networks for each of 100, 1000, 10000 and in items in the training set (in. involved randomly generating new images on the fly every step, to approximate an infinite diversity of shapes). To vary iteration, we introduced a training variant (Algorithm[Tin in Appendix) that treated the network as iterated function. based on the principle that (n(-1) If (n))). where f is a rotation or a scaling function approximated by model M, and ne is shape after transformations. For each input, the final output was generated by k applications of M (Line #5-7 in Algorithm D The final output was compared with the target image (initial ko image transformed times, for example, 25 clockwise rotations). The loss and the gradients were calculated on the final output and then used to perform a single weight update (Line 18-10). For each training step, the integer was sampled uniformly between I and 1, (Line #3). To assess the effect of varying the amount of iteration we trained seperate set of models with I = 1,2,9 9. These values were mainly chosen to study the effect of no iteration (Le 1). just one step of iterative training (4+ = 2) and sufficient number of steps (L, = 9) which is still quite less than the number of iterations used during testing as mentioned later. While this procedure resembles recurrent neural networks like the LSTM (Hochreiter & Schmidhuber [197), it differs in that it accepts only single input (whereas recurrent networks usually accept sequences) and it only propagates the output, rather than hidden network state, so it is memory-less.
At test time, the networks were presented with new images which it hadn't 't seen before, and then repeatedly applied (outputs passed back inputs) to assess the correspondence between each suc- cessive transformation and the correct shape at that time point. We used the intersection over union (IOU) metric to assess test performance at time and time 50 (temporal extrapolation since it re- flects transformations far past the horizon used during training). We used a selection of test datasets to assess OOD generalization along different dimensions. For the dimension of Shape, we tested on shapes that had a hollow center since training shapes were all filled. For Location, we tested in the lower right region with Feenter, Vcenter E [46,50] for rotation and Feenters Veenter E [38,42] for scaling transformation. For Size (Larger grid), we doubled the grid-size from 64x64 to 128x122 and doubled the radius of the shapes so they were much larger than shapes used for training. For Size (Same grid), the radius of the shapes were increased from [5.7] to 10 on 64x64 grid. Each test data set contained 500 shapes.
4 RESULTS
We provide mean IOU scores along with standard deviation across five runs for each instance.
4.1 ROTATION
To demonstrate the effect of iteration we first focused on only extrapolation in time; since networks were trained on small number (1-9) of iterations, repeatedly transforming stimuli much longer than that at test time (up to 50 times) would reflect extrapolation. As well, due to the combinatorial size of the shape space, I.I.D. generalization rather than memorization was still necessary to succeed at this task. As seen from Figure 5(a) the AE trained with (D in It 9) was able to conserve the shape even till t = 50 and also achieved highest IOU (Figure 5(b).. This indicates that iteration tightens the error tolerance of each network output to achieve greater stability in time. There is also an interesting tradeoff between diversity and iteration shown in Figure ((6] apparenty for t = 50.
For lower diversity and higher iteration (D = 100, T2 9), the mean IOU score is very similar
5

---

Under review as a conference paper at ICLR 2022
(slightly better) the case with higher diversity, lower iteration (D inf 1). These iterative networks were also more stable in time than networks trained to rotate large angles in a single pass (Figure (6. iterative networks rotate at smaller increments while maintaining greater accuracy over time. However even with high diversity and iteration (D = in = 9), AE failed to rotate larger shapes in larger grids (Figure ((a) at locations (Figure ((b) suggesting that only the three components aren't I sufficient.
Rotation (64x64)
300
0339
6.03
0.01
300
0.00
5002
0817
6001
0.24
C4
1000
0.00
0.00
600
0213
0.002
0.000
0.003
0.04
0.64
0.023
(a) Iteration shape
conserves
(b) Divensity-Itcration Iteration tradcoff
Figure 5: Effect of diversity, iteration IID rotation quality for AE.
0.08
0.08
0.07
007
0.0
It. 9
0.06
0.06
0.05
005
0.55
0.03
0.0
0.03
0.01
001
081
0
10
30
10
20
40
10
20
50
Time Step
Time
Time Step
Figure 6: Comparison of iteratively trained rotation networks (1,= 8, 9) with networks that were trained rotate a specified angular distance in a single forward pass. For example, if the network I. experienced between 1 and 9 iterative passes during training, then the comparison network 9n
was trained to rotate an object by minimal rotations (9MR or 25 radians) in a single pass. For higher values of iterations/MR, the iteratively trained networks have improved performance time, as indicated by MSE, despite rotating shapes using many more steps.
1:28
7:12
7-28
1:38
1-40
*
7F
(a) (Larger Grid)
(b) Location
Figure 7: Comparison of OOD rotation quality between ground truth (GT), AE across time.
From Table I we that POLARAE achieves about 28% and 44% improvement in mean IOU score for OOD generalization to larger shapes in larger grid and location respectively compared to the second best performing model VAE. Figure ((a) 8(b) demostrates that POLARAE can rotate larger shapes in larger grids and new locations respectively, whereas VAE distorts the shapes, or cannot detect it and not able rotate in time.

---

Under review as a conference paper at ICLR 2022
P020
P52
P-22
2MQ
'OLANC
YOMMAC
(a) Size (Larger Grid)
(b) Location
Figure 8: Comparison of OOD rotation quality between ground truth (GT), VAE, POLARAB across
multiple times.
Figure shows the effect of diversity and iteration for POLARAE OOD generalization larger shapes in larger grids (9()) and unseen locations ( As indicated by the colour gradient for the case of 50, best performance was obtained by maximizing both diversity and iteration during training, and as pointed out carlier lesser diversity can be compensated by using higher iteration.
For OOD generalization hollow shapes and larger shapes in the same grid there were no major qualitative differences in the rotation over time (Figure 17 in Appendix).
Time:
Time 50
Time =1
Thme
Traning set diversiy (2)
0.00
0167
11.
123
03
01163
0116
5
1.55
0107
3.2
83
Ponne SXC dive sity
2.90
$.00
008
12.
5.27
048
000Z
600
0.90
2
30300
011.5
8113
0002
10000
6.2
0.6
0.3
k7-
0115
811
C415
0.0
-02
657
038
037
E14
02
42
Number of Iterations (1) suring training
Number of toeratons during training
Size (Larger Grid)
(b) Location
Figure 9: Effect of diversity iteration for POLARAE OOD rotation
1.2
KNT
9
9
9
9
Ω
V
8
8
RARAL
A
X
X
KOLMAR
(a) MNIST
(b) Shape Structure
Figure 10: Comparison of OOD rotation quality between AE, POLARAE across time.
We also experimented with two more 2D datasets, MNIST (LeCun et al. 1998) and Shape Structure (Carlier al.) 2016). For MNIST we trained the networks iteratively to rotate digits 0-5 on 64x64 grid and tested on digits 6-9 on a larger 128x122 18 grid. As we can see from Figure I0(a) POLARAE was able to succesfully rotate novel shapes presented in larger grid with time whereas AE failed. For the Shape Structure dataset we just evaluated the performance of the networks on novel shapes with time. (Figure TO(b). (Ob)
7

---

Under review as a conference paper at ICLR 2022
Table IOU scores of trained models tested OOD for rotation at time-points and 50.
Model
Shape
Size (Same grid) t = 50
Size (Larger grid)
Location
t= 1 t = 50
f ==
t t=
t == 50
t t= 1
t t= 50
AE
0.86 0.86-0007 0.57 0.04
0.89 0.00 0.62 0.00 0.68 0668.0.07 0.35 10.07
0.28 1..0 0.00
VAE
0.72=0.00 0.38=0.00 0.76
00.03 0.71 071100000 0.38
0.01 0.00
8-VAE
0.59=0.00
0.24 0.65
0.28
0.69
0.36+0.01
0.08=00.02 0.00
POLARAE
0.76#0.01 0.61
0.88#0.00
0.66
0.77=0.00 0.66#40.66
0.63=+0.01 0.44
4.2 SCALING
For scaling transformation too, the best performing AE at 50 was trained with (D in., 9), and the diversity-iteratin followed a similar pattern as in Fig. 5(b) However even with high diversity and iteration, AE failed scale larger shapes in larger grids (Figure 13 in Appendix) or at new locations (Figure 13(b) in Appendix).
7
7
7
7
7
*
*
*
*
4
*
*
9
7
7
7
*
*
*
$
1
9
9
9
9
9
www
9
9
9
9
9
9
(a)
Size (Larger Grid)
(b) Location
(c)
MNIST
Figure 11: Comparison of OOD scaling quality between ground truth (GT), VAE, AE, POLARAE
across multiple times.
From Table 2 we can see that POLARAE achieves 27% and 29% improvement in mean IOU score for OOD generalization to larger shapes in larger grid and location respectively compared to the second best performing model VAE. Figure TTa] [Tbb) demostrates that POLARAE can scale larger shapes in larger grids and new locations respectively, whereas VAE distorts the shapes without actually scaling it with time. POLARAE can also scale unseen MNIST digits presented on a larger grid in time unlike AE (Figure (Ic)).
Figure 14 in Appendix shows the trade-off between diversity and iteration of POLARAE. As indi- cated by the colour gradient for the case of 50, best performance here too was obtained for high diveristy( D = in and high iteration( l = 9). Here too for lower diversity and higher iteration (D 1000,
9), the mean IOU score very similar to the case with higher diversity, lower iteration (D = inf, T 1).
Table 2: IOU scores of trained models tested OOD for scaling at time time-points I and 50.
Model
Shape
Size (Same grid)
Size (Larger grid)
Location
t= i 50
50
50
t=
50
AE VAE 8-VAE POLARAE
0.87=0.01 0.44=0.01 0.01
0.75
0.97#0.01
0.42
0.79=t0.05
0.23=
0.67=0.00 0.22 0.85 0.00
0.41 0.03
0.00
0.49
0.62 +.280
0.55=0.01 0.00=
0.02
0.28
#0006
0.55
0.250
0.30 0009
0.81 0.81 10.00 0.46 0.00
0.89
1.000
0,71 10.01
10.00
0.76 1..01
0.82
10.01
0.57 1.003
4.3 EXTENSION TO 3D SHAPES
We extended the models to perform 3D object rotation from voxel occupancy grids. We used Mod- elNet40 (Wu et al. 2015), where objects were rotated around direction. We iteratively trained to
8

---

Under review as a conference paper at ICLR 2022
POLARAE
AE
POLARAE
MNT
AE
POLARAE
Figure 12: Comparison of OOD rotation quality of 3D shapes across time.
rotate 30 object categories on 30x30x3 voxel grids and tested the heldout 10 categories presented larger 50x50x50 grids. We used techniques from Esteves et al. (2017) to extend POLARAE to 3D, predicting axis and using it as origin to compute cylindrical coordinates (channel wise polar transformation). For all channels, the origin was the same and each channel was a 2D slice of the 3D voxel grid. Similarly the inverse transform module extended 3D convert the output back to voxel grid. For AE we used a fully 3D convolutional autoencoder. As we can see from Figure I2 POLARAE is able rotate unseen 3D objects with time barring minor distortions whereas AE completely distorts it and is unable to rotate.
5 DISCUSSION & FUTURE DIRECTIONS
We tested the ability of standard autoencoders, variational autoencoders in cartesian space and au- toencoders log-polar space extrapolate learned transformations in pixel space. Unsurprisingly, convolution and high diversity were sufficient for OOD generalization of translation. However, those weren't sufficient for OOD generalization rotation and scaling. Iterative training helped preserve during training, but was still not sufficient rotate larger
the shape far past the time-horizon shapes in larger grids or at new locations with time.
Performing standard convolutions in log-polar space combined with high diversity and high num- ber of iterations during training resulted in stable transformations for larger shapes in larger grids and new locations far past the time horizon seen during training. Here, we propose that conserva- tion of shape emerges from the symmetry implied by training the networks as iterated functions; as Noether's theorem states, for any symmetric action, there a corresponding conservation law (Noether 1971)
We also found interesting interaction between diversity and iteration, where each could partially make up for less of the other to produce OOD generalization capacities. Based on all these results, we suggest that humans may use these strategies synergistically during development order to learn canonical transformations. Intuitively, child may optimally learn to predict the sensory effects of these transformations by both transforming many different objects as well as repeatedly transforming the same object. In this way, they are able to abstract canonical transformations away from individual object instances, while maintaining a sense of object permanence.
Finally, it appeared that transforming inputs into appropriate coordinate space using POLARAE significantly improved OOD generalization to rotation, scaling. We believe that the polar coordinate space better able leverage the inductive bias of translational convolution. We hope that these mechanisms inspired from humans can be incorporated into future neural network models to improve generalization to downstream tasks like classification, localization.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Fred Attneave and Malcolm D Arnoult. The quantitative study of shape and pattern perception. Psychological bulletin, 53(6):452, 1956.
Lawrence W. Barsalou. Grounded cognition. Annual Review of Psychology, 59(1)=117-645, jan
2008. ISSN 00664308. doi: 10.1::00.111/0///111111 4001 9900069.59900000000999939999999 URL http:///ww
annualreviews org/ /tsgg//oi// 1146/ http://nnnuuevvveeeeeee
59. 199.10306.... 093639
Peter W. Battaglia, Jessica B. Hamrick, and Joshua B. Tenenbaum. Simulation as an engine of physical scene understanding. Proceedings of the National Academy of Sciences of the United States of America, 110/45)::8332-183333 nov 2013. ISSN 00278424. doi: 10.1073/
maa.1206572110. URL. htt.t:::t/............tt //ww. www. www.. http::/wwwww.parrroooooo/ooaa/// http:///ccooooeeee//////////11/0
7/ www. /ww..pns.. org/ wecg/content//in 01tp//111//5/111211111 absttratt
Ali Borji, Saeed Izadi, and Laurent Itti. ilab-20m: A large-scale controlled object dataset to inves- tigate deep learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2221-2230, 2016.
Axel Carlier, Kathryn Leonard, Stefanie Hahmann, Geraldine Morin, and Misha Collins. The 2d shape structure dataset: user annotated open access database. Computers & Graphics, 58: 23-30, 2016.
François Chollet. On the Measure of Intelligence. arXiv 1911.01547v2. 2019.
Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. Lagrangian neural networks. arXiv preprint arXiv:2003.0400 2020.
Carlos Esteves, Christine Allen-Blan Blanchelance,tt,, Xiaowei Zhou, and Kostas Daniilidis. Polar transformer networks. arXiv preprint arXiv: (1t.::77000088, 2017.
Ysbrand Galama and Thomas Mensink. IterAANs: Iterative GANs to learn and control 3D object transformation. Computer Vision and Image Understanding, 189, 2019. ISSN 1090235X. doi: 10. 1016/j.cviu.2019, 102803. TtR.htttss://
wrg 101 6/ // w.w....vv....................... 2019. 102803
James J. Gibson. The theory of affordances (1979). In The People, Place, and Space
Reader, pp. 56 60. 2014. ISBN 9781315816852.
doi: 10432449781315881682 URL http:p/w.......oooooooooaaaaaaaa 2o6p@00100Wx22eer/B0000Wt3
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. In Advances in Neural Information Processing Systems, pp. 15379-15389, 2019.
Ralph Gross, lain Matthews, Jeffrey Cohn, Takeo Kanade, and Simon Baker. Multi-pie. Image and vision computing, 28(5))807-88) 2010.
Torkel Hafting, Marianne Fyhn, StURLa Molden, May-Britt Moser, and Edvard Moser. Mi- crostructure of a spatial map in the entorhinal cortex. Nature, 436(7052):801-886 2005.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick,
Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. 2016.
Felix Hill, Andrew Lampinen, Rosalia Schneider. Stephen Clark, Matthew Botvinick, James L Me- Clelland, and Adam Santoro. Emergent systematic generalization in a situated agent. arXiv preprint /rXiv: 005/0, 2019.
Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Neural Computation,
(8):1735-1788 nov 1997. ISSN 08997667. doi: 10. 12.1/e2//cco...m 99..9.......... 1735. URL https:
www. mwtmpesttssss..m journals org/ http:/ooxx//oo wbtpaa...................... 199.
Jinpyo Kim, Wooekun Jung, Hyungmo Kim, and Jaejin Lee. Cyenn: a rotation invariant enn using polar mapping and cylindrical convolution layers. arXiv preprint arXiv:2007.105300 2020.
10

---

Under review as a conference paper at ICLR 2022
Diederik P Kingma and Max Welling. arXiv:1312.6114 2013.
Auto-encoding variational
bayes.
arXiv preprint
Oleksii Kuchaiev and Boris Ginsburg. Training deep autoencoders for collaborative filtering. arXiv
preprint arXiv: 01715, 2017.
Ananya Kumar, Tengyu Ma, and Percy Liang. Understanding self-training for gradual domain adaptation. In International Conference on Machine Learning, pp. 5468-5479. PMLR, 2020.
Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequenee recurrent networks. In International Conference on Machine Learning, pp. 2873-2882. PMLR, 2018.
Brenden M Lake, Tomer D Ullman, Joshua Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
Yann LeCun, Yoshua Bengio, et al. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, (16)(10):1995, 1995.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFE, 86(11):227--3222 1998.
Spandan Madan, Timothy Henry, Jamell Dozier, Helen Ho. Nishchal Bhandari, Tomotake Sasaki, Frédo Durand, Hanspeter Pfister, and Xavier Boix. When and how do cnns generalize to out-of- distribution category-viewpoint combinations. arXiv preprint arXiv:2007.08032 2021.
Guido Maiello, Manuela Chessa, Peter Bex, and Fabio Solari. Near-optimal combination of dis-
parity across pog-polar scaled visual field. PLoS computational biology, 16(4):e1007699, 2020.
Gary F Marcus. The Algebraic Mind. 2001. doi: 10.755|/mitpress11187.0...00 URL http:
/oggnt mit Baclu/boo//aae wwtp:///weeebbraaaaaaaaaaae mimi
Emmy Noether. Invariant Variation Problems. Transport Theory and Statistical Physics,
1(3):186-201 207, mar 1971. ISSN 15322424.
doi: 10. 10.10800041447718214446
URL
http::/arxe wwwpexiwi.orraaaaom org abs http:///s.ppp...////333.////////
0041 004114571082311
Randall C O'Reill,y, Dean Wyatte, and John Rohrlich. Learning through time the thalamocortical loops. arXiv preprint arXiv: 2014.
Daniel N. Osherson, Ormond Wilkie, Edward E. Smith, Alejandro López, and Eldar Shafir. Category-based induction. Psychological Review, 97(2):185-20), 1990. ISSN 0033295X. doi:
URL /tt////corrrr//// 00.1037700033-295X.972222 / 18953-881188 001
Jean Piaget. The construction of reality in the child. Basic Books, 2006. doi: 10. 168-000.
Weichao Qiu, Fangwei Zhong, Yi Zhang, Siyuan Qiao, Zihao Xiao, Tae Soo Kim, and Yizhou Wang. Unrealev: Virtual worlds for computer vision. Proceedings of the 25th ACM international conference on Multimedia, pp. 1221-1224, 2017.
Rajesh PN Rao and Dana H Ballard. Predictive coding the visual cortex: functional interpreta- tion of some extra-classical receptive-fiele field effects. Nature neuroscience, 2(1):79 87, 1999.
Adam Santoro, Felix Hill, David Barrett, Ari Morcos, and Timothy Lillicrap. Measuring abstract reasoning in neural networks. In International Conference Machine Learning, pp. 4477-4486, 2018.
Gudrun Schwarzer, Claudia Freitag, and Nina Schum. How crawling and manual ob- ject exploration are related to the mental rotation abilities of 9-month-old infants. Frontiers in Psychology. 4, 2013. ISSN 16641078. doi: 10.3389/fpsyg.2013. 00097. URL http/////aattaaa//aSSSSS/STaaaaa
nobi. mlm. nih hrt.::/........aooooooatataaaaaa

---

Under review as a conference paper at ICLR 2022
Avi Schwarzschild, Eitan Borgnia, Arjun Gupta. Furong Huang, Uzi Vishkin, Micah Goldblum, and Tom Goldstein. Can you learn an algorithm? generalizing from easy to hard problems with recurrent networks. arXiv preprint (tte::2000045337, 2021.
Roger N. Shepard and Jacqueline Metzler. Mental rotation of three-dimensional objects. Science, 171(3972)701-703, 1971. ISSN 00368075. doi: 10.1126/science.1 17..397..0....
Christopher Summerficld, Fabrice Luyckx, and Hannah Sheahan. Structure learning and the poste- nior parietal cortex. Progress in Neurobiology, 184:101717, 2020.
Roger Tootell, Martin Silverman, Eugene Switkes, and Russell L De Valois. Deoxyglucose analysis of retinotopic organization in primate striate cortex. Science, 218((575):9022999) 904-9044 1982.
Yaacov Trope and Nira Liberman. Construal-level theory of psychological distance. Psychological review, 117(2):440, 2010.
Daniel M Wolpert, Zoubin Ghahramani, and Michael Jordan. An internal model for sensorimotor integration. Science, 269(5232) 1995.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1912-1920, 2015.
Keyulu Xu, Jingling Li, Mozhi Zhang, Simon S Du, Ken-ichi Kawarabayashi, and Stefanie Jegelka. How neural networks extrapolate: From feedforward to graph neural networks. arXiv preprint arXiv:2009.11844 2020.
Dengsheng Zhang and Guojun Lu. Study and evaluation of different Fourier methods for image retrieval. Image and Vision Computing, 23(1):33-49, jan 2005. ISSN 02628856. doi: 1016016/ limavis.2004.09.00 URL htttts://teeeeaeeeeeette monash http////oooae////////ooooaaaaaaa
htgp::/mw.....t.ta....ooooaoaaae motoorttooorrmmeeeteoe--o--o wew.oorrrmmaeeeeeeee retri
A APPENDIX
Size (Larger Grid)
(b) Location
Figure 13: Comparison of OOD scaling quality between ground truth (GT), AE across time.
12

---

Under review as a conference paper at ICLR 2022
10
10
Time 1
Time 50
Time
Time 50
Training set diversiity (D)
0.009
0.066
0.004
0.044
0.022
0.022
-00
0.078
0.088
0.047
0.033
0.025
0.037
008
Training set diversityy
1000
0.002
0.
0.001
9:9
0.000
0.064
0:4
0.051
0.037
-06
1000
0.6
0.034
072
0.017
0.007
0.36
0.018
0:11
0.017
0.000
06
10000
0.86
0.85
0.5 g
0.64
070
075
-04
10000
071
072
0.00 g
0.38
0:46
0.50 g
00
0.000
070
0.014
0.012
02
074
0.037
0.007
0.015
0.009
0.026
02
0.0
Number of Iterations (b) during training
Number of Iterations (b) during training
(a) Size (Larger Grid)
(b) Location
Figure 14: Effect diversity vs iteration for POLARAE on OOD scaling
10
1.
Time 1
Time 50
Time 1
Time 50
Training set diversity (D)
100
0.002
0.033
0.004
0.011
0.018
0.005
-.8
Training set diversity (D)
100
0.024
0.015
9.027
0.033
0.027
0.017
0.8
1000
0.000
0.00
0.001
0.001
0.029
0:47
0.005
0.026
0.6
1000
071
0.015
076
0.008
078
0.044
0.038
0.022
0.007
06
10000
0.60
0.80
0
0.56
0.50
0.6
-04
10000
0.80
0.80
0.00
0.11
0.
0.43
04
0000
0.001
0.019
0.010
-02
0.001
0.024
0.023
0.003
002
Number of Iterations (l)) during training
Number of Iterations (k) during training
(a) Size (Same grid)
(b) Shape
Figure 15: Effect of diversity vs iteration for POLARAE on OOD scaling
100
100
Time
Time 50
Time
Time 50
Training set diversity (D)
100
0.006
0.005
0.005
0.034
0.015
0.015
-08
Training set diversity (D)
100
0.003
0.007
0.017
0.032
0.023
0.015
08
1000
0.60
0.003
0.54
0.077
0.87
0.002
844
0.025
048
0.001
0.64
0.05
-06
1000
0.6
0.018
072
0.008
075
9011
0.32
0.035
041
0.044
0.5
0.03
06
10000
0.66
0.7
0.88
0.53
0.60
0.65
-04
10000
021
074
076
8.37
042
0.50
+
04
0.006
0.033
0.001
0.028
0.003
0.054
-02
0.005
0.025
0.010
0.033
0.045
0.019
00
Number of Iterations (I+) during training
Number of Iterations (h) during training
(a) Size (Same grid)
(b) Shape
Figure 16: Effect of diversity vs iteration for POLARAE on OOD rotation
13

---

Under review as a conference paper at ICLR 2022
AT
Figure 17: Comparison of OOD rotation on hollow shapes.
8
PORRE
FOLARSE
Figure 18: Comparison of OOD scaling hollow shapes.
Algorithm 1: Iterative training with D) inf
Initialize: Model M with parameter 0;
for step s=/ to Nsteps do
A + r-nddom integer drawn uniformly from [1; input, target + ceata_generator(:: Tcenter, Voenter N, k, BS); for iteration iter=] k do output + (input)
input output
loss (SEL
loss. bassbackkwrrd);
10
Update parameter 0:
14

---

Under review as a conference paper at ICLR 2022
0.12
It.
0.10
It.
MSE
0.08
It
0.06
It
9
0.02
0.00
20
40
60
80
100
Time step
0.014
0.013
0.012
0.011
It.
MSE
0.010
0.009
0.008
0.007
0.006
2
3
Time step
Figure 19: Top: Mean squared error (MSE) on I.I.. test set at each time step for 9 trained rotation networks. The label 'It. = N' indicates that during training, the number of iterations of the newtork on given batch was sampled uniformly between and N. Bottom: the same plot blown up to visualize the first 6 time-steps. Of note, networks trained with higher iterations actually have worse MSE at the first time step, but achieve much better MSE the long run. Confidence intervals represent standard error between 3 identically trained networks.
15


Reviews:
[{"review_id": "uDJyk-E02I-", "reviewer": "Reviewer_JcBj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The paper proposes interesting new ideas (iteration, log-polar convolutions) to approach a long standing, seminal problem in computer vision. \n\n2. Experiment design is solid, and design decisions take care to ensure a high degree of scientific rigor. \n\n3. Paper is very well written. It is easy to follow and understand. \n\n4. The approach used here---empirical, controlled analysis of generalization is a great tool to better understand and improve generalization performance of computer vision models.\n\nWeaknesses:\n\n1. All experiments are conducted on a very simple toy dataset, despite several more complex datasets existing: The paper suggests the use of this dataset generation pipeline because it provides them complete control over the training and test distributions, interpretable manipulations and computational speed. However, rotational invariance is a long standing problem, and several datasets exist to study it. Especially in 2D. For instance the popular MNIST-Rotation dataset. In fact, if we limit the problem to 2D rotations, it is possible to take almost any image dataset and create 2D variations of it with great computational speed. In fact, there are several datasets that exist to study 3D rotation invariance now, which all allow generating new rotations (albeit at a greater computational cost). In is unclear how well these findings extend to more complex data, which significantly reduces the utility.\n\n2. Use of boundary points by the network: As mentioned in Sec 3.2, the inverse transform module uses the sample points of the original input image. This presents a problem, as implementing this approach on any other dataset would first require segmenting the objects to obtain the boundary control points. Thus, POLARAE can only be used if segmentation masks exist, which limits its applicability. This is also tricky because for some experiments (hollow OO\u0010D samples) as this information is the same for in-distribution and OOD samples. If the model relies heavily on control points, it would explain why hollow shapes are not a problem for the model. \n\n3. Overlapping train/test distributions, not truly OOD: Sec 3.3 mentions that fir scaling, input images had r in range (5,7), while test images were created by scaling vertices in increments of 0.1. Polygons for which r was towards the lower end of this spectrum would still be within the training distribution range of (5,7) for multiple iterations of scaling. Thus, the test target is not guaranteed to be OOD. To ensure it, a range of 7+delta would be necessary. Same is true for rotation. A rotated target sample could by itself also be generated by the training procedure itself with a non-zero probability. Thus, the the train and test distributions are not disjoint, and thus overlapping. A similar logic holds true for location as well. Thus the only purely OOD case is of the hollow shapes, but that suffers from its own problem as mentioned above.\n\n4. Baselines: It is unclear what the paper means by \"understanding a transformation\", which makes baselines tricky. Generating 2D rotations, translations and scalings of an image by itself is trivial: we don't need a CNN for that. The reason behind training a CNN capable of these transformations might be that such models would then have an understanding of this transformation just as humans have, and then we can hope that the representations learned by these models can generalize on downstream tasks. However, this has not been explored, or sufficiently discussed in the paper currently. In a nutshell - why are we training models to rotate/translate? What kind of OOD situations should these models generalize and be able to rotate/translate, and why is that a necessary requirement?\n\n5. Not placed in the context of existing work: Several ideas discussed in the paper have been explored in recent literature which have not been mentioned/discussed. For instance, the idea of novel shape-viewpoint and the role of data diversity on generalization to OOD has been discussed at length in [1]. Similarly, generalization to OOD viewpoints has been discussed in several generative works on novel view synthesis [2,3,4]. Also, the interplay between convolutions and translation invariance has been studied in [5,6]. Placing this work in the context of existing literature in empirical analysis of invariances is essential to make its contributions clear.\n\nI am happy to revise my review if I have misunderstood any details, or there is an error in the review. Please feel free to point them out!\n\nReferences\n\n1. Madan, S., Henry, T., Dozier, J., Ho, H., Bhandari, N., Sasaki, T., Durand, F., Pfister, H. and Boix, X., 2020. When and how do CNNs generalize to out-of-distribution category-viewpoint combinations? arXiv preprint arXiv:2007.08032.\n\n2. Borji, A., Izadi, S. and Itti, L., 2016. ilab-20m: A large-scale controlled object dataset to investigate deep learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2221-2230).\n\n3. Gross, R., Matthews, I., Cohn, J., Kanade, T. and Baker, S., 2010. Multi-pie. Image and vision computing, 28(5), pp.807-813.\n\n4. Qiu, W., Zhong, F., Zhang, Y., Qiao, S., Xiao, Z., Kim, T.S. and Wang, Y., 2017, October. Unrealcv: Virtual worlds for computer vision. In Proceedings of the 25th ACM international conference on Multimedia (pp. 1221-1224).\n\n5. Zhang, R., 2019, May. Making convolutional networks shift-invariant again. In International conference on machine learning (pp. 7324-7334). PMLR.\n\n6. Chaman, A. and Dokmanic, I., 2021. Truly shift-invariant convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3773-3783).\n", "summary_of_the_paper": "At its heart, the paper explores which inductive biases (and image representations) enable CNNs to generalize better across 2D transformations including rotations, translations and scale variations. The paper starts by proposing a methodology for generating a controlled dataset of binary masks consisting of random polygons. All training and testing is conducted on this dataset. After showing that convolutions and high data diversity enable better generalization across these transformations, the paper proposes two additional ways to improve performance. Firstly, their approach transforms input images into log-polar space, as object rotations in the original space is equivalent to translations in this space, and thus convolutions on log-polar space are equivariant to rotations in the original pixel space. Secondly, the paper proposes an iterative approach to model transformations, where output mask generated by their Encoder-Decoder model is fed back into the network as an input. The motivation behind this iterative approach is that these transformations form a symmetric group, and so the output after k iterations is still a member of the set which can be reached via a single transformation. The motivation is that such training would force the network to preserve the shape being transformed across iterations. The paper shows that networks trained with these inductive biases and training methodologies do indeed generalize better across these 2D transformations.", "summary_of_the_review": "While the paper proposes some interesting ideas (iteration, log-polar convolutions) and attempts to understand generalization across 2D transformations in a rigorous manner, it is currently not mature enough for publication. \u0010I would advise (1) extending the experiments on some more realistic datasets listed below, (2) paying greater attention to the training/testing distributions used, and (3) placing the work in the context of existing works above all to make this work more solid.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LXCu9MJBRaZ", "reviewer": "Reviewer_31wD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n- This authors use a unique setup to test the generalization capabilities of models. In particular, their data generation strategy is flexible and can produce a large variety of images with different rotations, scales, and translations.\n\n- The authors test the ability of the models to extrapolate beyond the transformations present in training, and identify 4 ways to improve generalization ability (iterative training, diversity, convolutions, log-polar coordinates)\n\n-The authors show that some strategies are effective for translation, but more is needed to address rotation and scaling.\n\n-The paper is well written and interesting to read.\n\nWeaknesses:\n\n-Though the data generation setup and test environment is interesting and unique, it's not clear how effective the techniques will be on more complicated realistic datasets. We already know that diversity and convolutions are useful tools. There's also prior work showing iterative training is useful for addressing domain adaptation https://arxiv.org/abs/2002.11361. So, the novel recommendation is to try log-polar coordinates. It would be interesting to test if log-polar coordinates can help OOD generalization on ImageNet. ", "summary_of_the_paper": "The authors trained autoencoders and variational autoencoders in cartesian space, as well as autoencoders in log-polar space, with generated data representing a range of interesting transformations. They then tested the ability of the models to extrapolate beyond the learned transformations in pixel space. The authors find that iterative training, data diversity, convolutions, and transformation to log-polar space all improve the generalization performance.\n", "summary_of_the_review": "The paper provides a unique way to test generalization abilities of models and offers some novel recommendations and insights.\n\nMy score reflects my view that the data generation process and test environment that the authors propose are important and novel contributions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9rmROiSECJa", "reviewer": "Reviewer_sNFP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The evaluation is technically sound. \n2. Experiments shows that operating in log-polar space does have in o.o.d generalization for affine transformations.\n\nCons:\n1. Only a very simple synthetic dataset is used for evaluation. I suggest authors to use at least 2 or 3 datasets (such as the Shape dataset) to prove that their method's effectiveness is generalisable to other datasets.\n2. Baselines are very limited. Only vanilla CNN and VAE are compared against. There are many more VAE variations that can be compared. For example, AIR [1] model factorises transformations and content by designing an explicit affine-transformation module, while NeRF-VAE [2] uses neural radiant fields.\n3. The paper has limited novelty. The proposed model is a VAE in log-polar space. The properties of polar space have already been explored in various papers mentioned by the authors. Recently its effectiveness in achieving rotation invariance has also been explored[3].\n4. Many design choices seem arbitrary without proper explanations. For example, why is I_t chosen to be 1,2,9? \n\nMinor:\n1. Some figures (figure6) are of very low resolutions.\n2. The method can be described more in maths, less in text.\n3. Algorithm 1 should be in the main text, not in Appendix. Or if the authors want to put it in Appendix, mention it (say 'Algorithm 1 in Appendix...')\n\nRef:\n1. Eslami, S. M., et al. \"Attend, infer, repeat: Fast scene understanding with generative models.\" Advances in Neural Information Processing Systems 29 (2016): 3225-3233\n2. Kosiorek, Adam R., et al. \"Nerf-vae: A geometry aware 3d scene generative model.\" arXiv preprint arXiv:2104.00587 (2021).\n3. Kim, Jinpyo, et al. \"CyCNN: a rotation invariant CNN using polar mapping and cylindrical convolution layers.\" arXiv preprint arXiv:2007.10588 (2020)", "summary_of_the_paper": "The paper proposed POLARAE, an auto-encoder that operates in log-polar space. The paper demonstrated the model, together with iterative learning, is better than simple baselines in o.o.d generalization for a set of affine transformations.", "summary_of_the_review": "A technically sound paper that has limited novelty. More baselines should be compared against on more datasets.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4slog3W5jbn", "reviewer": "Reviewer_3Ri2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The key contribution of this paper is to propose to utilize convolution, high training diversity, iterative training, and log-polar space for improving the generalization ability of AE models for OOD settings. The authors demonstrate its usefulness on a toy task.  \n\nThough the authors start the paper with psychology literature and discussions, they fail to really establish connections between their proposed methods and the literature. \n\nFor example, the authors write in the paper \"...we hypothesize that performing convolutions in log polar space would help in the OOD generalization of rotation and scaling\", but they never provide any, not even distantly related, support from psychology. O the other hand, there is no mathematical justification, similar to the paper [https://arxiv.org/abs/1709.0188](https://arxiv.org/abs/1709.01889](https://arxiv.org/abs/1709.01889),)\n\nAs for experiments, the authors propose a new data generator to create lots of simple but structured images. In my opinion, this dataset is too simple, and cannot justify the claim of the authors. I don't know why the authors do not use the 3D dataset used in [https://arxiv.org/abs/1709.0188](https://arxiv.org/abs/1709.01889](https://arxiv.org/abs/1709.01889),), which seems more difficult than the one used in this work. \n\nThe writing of this paper seems sloppy and needs to be improved significantly. For example in the abstract, the name of the proposed method, POLARAE, is introduced but its meaning has never been properly explained. \n\nMany figures in the paper need to be improved: \n\n- It's difficult to read the numbers in Figure 4\n- Figure 6 and Figure 17 are very blurry and should use PDF formats.", "summary_of_the_paper": "Though the authors start the paper with psychology literature and discussions, they fail to really establish connections between their proposed methods and the literature.", "summary_of_the_review": "Though it is useful to investigate the OOD transformation on toy tasks first, it is important to scale it up to more realistic settings later. The observations in this paper are not likely to generalize to realistic settings that we really care about. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies different inductive biases that would improve OOD generalization (and in particular under translation, rotation and scaling) for image tasks. The study is focused on a toy dataset which allows authors to have more control over the data generation process and the transformations. Authors further show that iterative training using an auto-encoder and presenting data in log-polar space helps with rotation and scaling transformations on their toy dataset.


Strong Points:
- The paper is well written and easy to follow.
- The data generation process and the resulting toy dataset are novel and interesting.
- The experiment design and evaluations are solid.

Weak Points:
- No natural image datasets: While using a toy dataset has several benefits it does not grant that the conclusions would generalize to realistic settings. Reviewers have suggested several realistic datasets and I encourage authors to evaluate their findings on some of these datasets.
- Limited Baselines: As reviewers have pointed, comparison with baselines can be improved by including stronger baselines as well as more clear discussion about other techniques such as augmenting the data with the transformations.
- Related Work: A proper discussion of related work to set the context and highlight the contributions of this paper is missing. In particular, reviewers have pointed to prior work on the benefits of presenting the image in log-polar space.

Unfortunately, authors did not engage with reviewers during the discussion period. Given the prior work and lack of any natural image dataset, I think the novelty and significance of this work is limited. Therefore, I recommend rejecting the paper. However, I encourage authors to improve the paper by addressing the points raised by the reviewers.
Decision: Reject