Visual hyperacuity with moving sensor and recurrent neural computations
Authors: Alexander Rivkind
Or Ram
Eldad Assa
Michael Kreiserman
Ehud Ahissar
Venue: ICLR 2022 Poster (2022)
Keywords: visual system
convolutional neural networks
recurrent neural networks
active vision
active sensing
ocular drift

Abstract: Dynamical phenomena, such as recurrent neuronal activity  and perpetual motion of the eye, are typically overlooked in models of bottom-up visual perception. Recent experiments suggest that tiny inter-saccadic eye motion ("fixational drift") enhances visual  acuity beyond the limit imposed by the density of retinal photoreceptors. Here we hypothesize that such an enhancement is enabled by recurrent neuronal computations in early visual areas. Specifically, we explore a setting involving a low-resolution dynamical sensor that moves with respect to a static scene, with drift-like tiny steps. This setting mimics a dynamical eye viewing objects in perceptually-challenging conditions. The dynamical sensory input is classified by a convolutional neural network with recurrent connectivity added to its lower layers, in analogy to recurrent connectivity in early visual areas.  Applying our system to CIFAR-10 and CIFAR-100 datasets down-sampled via 8x8 sensor, we found that (i) classification accuracy, which is drastically reduced by this down-sampling, is mostly restored to its 32x32 baseline level when using a moving sensor and recurrent connectivity, (ii) in this setting, neurons in the early layers exhibit a wide repertoire of selectivity patterns, spanning the spatiotemporal selectivity space, with neurons preferring different combinations of spatial and temporal patterning, and (iii) curved sensor's trajectories improve  visual acuity compared to straight trajectories, echoing recent experimental findings involving eye-tracking in challenging conditions. Our work sheds light on the possible role of recurrent connectivity in early vision as well as the roles of fixational drift and temporal-frequency selective cells in the visual system. It also proposes a solution for artificial image recognition in settings with limited resolution and multiple time samples, such as in edge AI applications.

Published as a conference paper at ICLR 2022
VISUAL HYPERACUITY WITH MOVING SENSOR AND RECURRENT NEURAL COMPUTATIONS
Alexander Rivkind", Or Ram', Eldad Assa, Michael Kreiserman & Ehud Ahissar Dept. of Brain Sciences, Weizmann Institute, Rehovot, Israel falexander. .ngirriiike rivkind, or ram, eidad.asse mi chael krei serman, ehud. ah achissrr @wei zmann. ac. i1
ABSTRACT
Dynamical phenomena, such as recurrent neuronal activity and perpetual motion of the eye, are typically overlooked in models of bottom-up visual perception. Re- cent experiments suggest that tiny inter-saccadic eye motion ("fixational drift") enhances visual acuity beyond the limit imposed by the density of retinal photore- ceptors. Here we hypothesize that such an enhancement is enabled by recurrent neuronal computations in early visual areas. Specifically, we explore a setting involving a low-resolution dynamical sensor that moves with respect to static scene, with drift-like tiny steps. This setting mimics a dynamical eye, viewing objects in perceptually challlenng conditions. The dynamical sensory input is classified by a convolutional neural network with recurrent connectivity added to its lower layers, in analogy to recurrent connectivity in early visual areas. Apply- ing our system to CiFAR-10 and CiFAR- 100 datasets down-sampled via 8x8 sen- sor, we found that (i) classification accuracy, which drastically reduced by this down-sampling, is mostly restored to its 32x32 baseline level when using mov- ing sensor and recurrent connectivity, (ii) in this setting, neurons the early lay- ers exhibit wide repertoire of selectivity patterns, spanning the spatio-temporal selectivity space, with neurons preferring different combinations of spatial and temporal patterning, and (iii) curved sensor's trajectories improve visual acuity compared to straight trajectories. echoing recent experimental findings involving eye-tracking in challenging conditions. Our work sheds light on the possible role recurrent connectivity in early vision as well as the roles of fixational drift and -requullfrequuncc selective cells in the visual system. It also proposes solu- tion for artificial image recognition in settings with limited resolution and multiple time samples, such in edge Al applications.
INTRODUCTION
Biological vision is known to be dynamical process. Two factors contributing to these dynamics are eye motion and recurrent neuronal connections in the brain. Our eyes move constantly with movements that, kinematically, can be divided into saccades quick gaze shifts, and drifts small scanming movements between saccades (often referred to as "fixational drift") (Rucci et al. 2018) These dynamical aspects of vision are reflected only partially in contemporary computer vision systems. Some works addressed large scale shifts in visual attention resembling saceades (Mnih et al. 2014). Others explored properties and benefits of recurrent top down connections (Nayebi
et al. 2018), reminiscent of top-down processing in biological vision (Hochstein & Ahissar 2002))
Notably, the dynamics of low-level visual processes, occurring early in the bottom-up visual hier- archy and sensitive to the fixational drift (Snodderly et al., 2001 Ölveczky et al. 2003 Malevich
al. 2020; Hohl & Lisberger 201T). remains largely overlooked in models of vision as well as in bio-inspired computer vision systems.
In fact, since the seminal studies by Hubel Wiesel (1962). selectivity in primary visual cortex has been traditionally described in terms of static spatial filters (e.g., simple and complex spatial fields or Gabors of varying frequency and orientation). convolutional neural networks (CNNs)
"Equal contribution

---

Published as a conference paper at ICLR 2022
(Krizhessky et al. 2012). which have dominated computer vision over the last decade, features resembling the spatial filters deduced from biological studies emerge spontaneously over the course of training (Zeiler & Fergus 2014 Lindsey et al.] 2019). In some cases, remarkable correlations were found between spatial neural representations in CNNs and those identified in the biological brain (Yamins & DiCarlo 2016).
On the other hand, temporal dynamics, and sensitivity to temporal features, characterize visual neu- rons throughout the visual system, from retinal receptors and ganglion cells thalamic and cortical
neurons (Berry et al., 1997; Chichilnisky 2001] Lee et al. 1981 Levick et al.
1972 Reinagel &
Reid
2000; [Shimaoka et al. 2018). Existing evidence suggests that both eve motion (Snodderly
et al.
2001 Ahissar & Arieli 2001 Ölveczky et al. 2003 Malevich et al. 2020 Gruber et al.,
2021
Hohl & Lisherger| 2011 I) and recurrent neuronal connectivity (Bejjanki et al. [01 Samonds
et al.
2013) contribute to this temporal dynamics. Furthermore, it was found that recurrent connec-
tions
improve correlates of artificial neural networks to neural activity in visual cortical areas (Kar)
ct al.
2019 Kubilius et al., 2019 Kietzmann et al. 2019).
One niche where spatio-temporal computation is probably necessary is the perception of tiny ob- jects. It is well known that the acuity of biological vision is not limited by the spatial resolution of retinal photoreceptors ("visual hyperacuity",, |Westheimer (2009); Barlow (1979)). Vernier acuity, for example, is dramatically higher than might be expected from pure spatial acuity derived from the photoreceptor density in the retinal mosaic (Westheimer 2009). Whether hyperacuity is obtained via spatial, temporal, or spatio-temporal mechanisms is not yet known (Rucci et al.. 2018). In any case, it is evident that the visual processing allowing hyperacuity, or perception of any tiny stimulus, should cope with the fixational drift; if it doesn't, 't, the drift, whose amplitude is at least two orders of magnitude larger than the smallest perceivable spatial offsets, would impair acuity (Ahissar & Arieli 2001 Rucci et al.) 2018; Ratnam et al.) 2017). The same drift motion could potentially improve acu- ity if spatio-temporal computations are employed. Such computations can be based on the emphasis of high-frequency spatial details (Rucci et al. 2007), temporal coding of spatial offsets (Ahissar & AArilli [201]] 2012), Bayesian inference (Anderson et al., 2020). or on any other derivative of the interactions between ocular motion and the external image. Furthermore, is reasonable to attribute such apatio-temporal computations to early visual areas, which are known to exhibit faster dynamics and shorter integration windows compared to regions upstream in the visual processing chain (Gau- thier al. 2012). Indeed, it had been shown that the recurrent neuronal circuitry in early visual areas could enable countering the blurring from retinal motion (Burak et al. 2010).
Using the information available from over-sampling low-resolution images has extensive history in computer vision as part of the field of super-resolution (Milanfar 2017). Multi-image super- resolution (MISR (Farsiu al.) 2004), distinguished from single-image super-resolution (Glas-) ner et al. 2009: Dong et al. 2015). aims to reconstruct high-resolution images from a set of low- resolution ones (Arelin al. 2020 Ge et al. 2018 Bhat al. 202T Li et al. 2017). An adjacent
field of research, low-resolution object recognition, investigates algorithms to maximize the perfor- mance on a given task (Xi et al. 2220 Ge al. (0118). Both fields use low-resolution images as input but differ in the goal of the training and evaluation.
In this paper, we introduce a classifier that exploits spatio-temporal computations in early layers to perceive tiny images. More specifically, we trained a convolutional neural network with recurrent connectivity introduced to early layers. The network receives a sequence of low-resolution images generated via sensor motion mimicking ocular drift. We used high-resolution images to obtain set of features that were then used to facilitate learning in a teacher-student framework (Hinton) let al. 2013). The outcome is dynamical classifier that suffers from only small drop in accuracy when tasked with a significant decrease in spatial resolution, a decrease that substantially impairs the accuracy of comparable static feed-forward classifier.
Using a novel generative model, we found that our dynamical classifier developed features that were primarily sensitive spatial changes, others that were primarily sensitive temporal changes, and a majority that exhibited sensitivity to mixed spatio-temporal patterns.
Finally, when examining the correlations between patterns of motion and accuracy of classification, we observed that curved trajectories are favorable for recognition, consistent with recent findings of the curvature of fixational drift trajectories in humans. (Intoy & Rucei 2020 Gruber & Abissar 2020).
2

---

Published as a conference paper at ICLR 2022
2 RESULTS
2.1 TASK AND MODELS
To create a synthetic setting reminiscent of ocular drift, we used images from popular CiFAR datasets (Krizhevsky et al. 2009), embedded in a large (200x200 pixel) scene padded by zeros. Sensor position was defined in units of pixels on the scene and its motion was modeled by stochas- tic process that is discussed below, The sensor's frames were obtained by cropping 32x32 pixels window from the scene around the sensor position. Resolution was then reduced to 8x8 using a standard OpenCV (Bradski 2000) function (that does not include anti-aliasing filter) with bi-cubic interpolation (Fig. [A].
A ResNet50 (He et al. able as a part of Keras package (Chollet 2015), was used as a model of reference. The model was fine-tuned to one of the CiFAR datasets, reaching accuracy of 96.83 and 82.94 percent for CiFAR- 10 and CiFAR 100, respectively (Table Standard resolution (32x32)). In order to feed the 32x32 pix CiFAR images the network, images were upsampled by factor of 7 to (32x7)x(32x7) pix (i.e. 224x224 pix which are the dimensions of ResNet50 input).
2016) network pre-trained on ImageNet (Deng et al. 2009), which is avail-
In order verify the generality of conclusions, we tested another more compact variant of reference CNN with 3M parameters. This smaller network that we refer to as Small-net (Table S7 receives 32x32 pix images input, therefore, eliminating the need to upsample the CiFAR images before feeding them into the network. This smaller network also simplified the analysis of internal representations as explained below.
2.1.1 TRAINING
We applied feature learning paradigm (Hinton al. 2015). while using our reference network as teachers for the dynamical recurrent classifier (DRC) student.
Typical CNNs perform series of spatial pooling operations. Max pooling layers in the reference CNNs effectively reduce spatial resolution while preserving relevant information about the underly- ing scene. To develop our DRC, we exploited this spatial pooling line-up. We thus took instances of trained CNNs and replaced their bottom layers with recurrent convolutional networks (Fig, ( Specifically. we used stack of ConvGRU layers (Ballas et al.) 2015) Van Valen al.) 2016) with- out spatial poolings to replace the original network all the way from the input to the point where the CNN's spatial resolution reduced by the desired factor (Table SS3 In our case, the reso- lution was decreased by factor of 4, therefore the appropriate resolution was achieved after the second max- max-pooling layer. At this layer's output, the resolution of the ResNet50-basee teacher is (8x7)x(8x7) (the 'x7' factor is due to the upsampling of the original 32x32 images by factor of 7 to (32x7)x(32x));; while the Small-net-based teacher resolution is 8x8. We refer to the bottom recurrent part of the DRC as the reference (teacher) network architecture (either ResNet50 or Small-net). We refer to this reused part of the DRC as DRC- back-end (DRC-BE) (Fig. B
(DRC-FE). For the rest of the processing stack we reuse
We trained the DRC in two steps first, the DRC-FE was trained to reproduce features of the teacher network Here we used mean-squared loss between the teacher network and the DRC-FE (other optimization goals, such as mean absolute loss and cosine similarity, resulted in very similar per- formance and are not shown). Positional data were concatenated with the images time series; see Appendix B.4 for further details. Next, the DRC-BE was fine-tuned using cross-entropy loss.
Our model was mostly implemented in the Keras package (Chollet 2013), with the convolutional GRU layer adapted from the project of (Van Valen et al. 2016).
2.2
PERFORMANCE
2.2.1
BASELINE
In order to evaluate the performance improvement which can be attributed to the unique architecture of the DRC, we considered a few baseline solutions.
'The code can be found at http://///ggiiaaa..ooo///ra//aa/ 1 aasi fier

---

Published as a conference paper at ICLR 2022
A.
Zero padding
Time-series from device in device's resolution
Full resolution
ANN
Class
Time
32 pix
B.
Teacher (CNN)
Nt spatial-pooling layer
Full resolution input
Low resolution feature map
Class
Time-series resolution frames
Teach
DRC-BE
Temporal pooling
Low resolution feature map
Time
Feed-forwrard
convolutional connectivity
Recurrent convolutional connectivity
Positional info
(optional)
DRC-FE
Student (DRC)
Figure 1: Description of the the task and the system A. A time-series of low-resolution images, simulating sequence of frames generated by a sensor moving a static scene, fed the network one-by-one following the order of acquisition, along with optional position information: the network integrates the information from the whole sequence of images and outputs a class. sample tra- jectory composed of T' 3 steps (orange - red blue) is illustrated on the full resolution image together with the corresponding 3 generated frames. B. Network architecture and training proce- dure: Teacher is feed-forward convolutional neural network, e.g., ResNet50 Small-net. Student multilayer recurrent convolutional neural network. At the first phase of training. Student's trained to reproduce teacher's features. These features extracted
bottom layers (DRC-FE) from the teacher at the layer where the teacher's spatial resolution corresponds to the student's 's in- put resolution. The top shared layers (DRC-BE) are then fine-tuned improve the accuracy of the low-resolution task. See main text for more details.
4

---

Published as a conference paper at ICLR 2022
Table 1: Test-set accuracy [%] of various baseline solutions based on ResNet50 compared to
ResNet50-DRC in various configurations. Results are presented as mean = std.
CiFAR-10
CiFAR-100
Standard resolution (32x32)
96.83 + 0.09
82.94 + 0.23
Low resolution (8x8) baseline:
Naive training
78.88 = 0.54
54.41 0.21
Naive training teacher
78.54 + 0.09
54.75 0.14
Averaged prediction, steps
83.86 + 0.47
60.24 0.22
Averaged prediction, 10 steps
83.87 + 0.23
60.22 0.25
Averaged prediction, steps teacher ResNet+GRU, 5 steps, w/o position input ResNet+GRU, steps, with position input
84.04
61.56 0.23
83.52 + 0.22
59.32 0.20
83.94 0.12
59.61 = 0.59
ResNettconnGRU. 5 steps, w/o position input
83.73 = 0.10
58.62 = 0.16
ResNet+cony GRU, steps. with position input GAN-based Xi et al. 2020
83.84 = 0.09
58.05 = 0.24
88.1
Low resolution (8x8) DRC:
DRC steps, not moving DRC steps, w/o position input DRC steps, w/o position input
75.87 + 0.29
52.02 0.07
87.83 + 0.38
67.23 0.30
90.16 0.59
70.38 0.67
DRC steps, with position input DRC steps, deeper, with position input DRC 10 steps, with position input
92.26 + 0.19
74.23
93.45 + 0.15
76.24
94.77+00009
78.75+0.55
The accuracy of the ResNet50 reference (teacher) network when applied single low resolution image, was chosen as a simplistic baseline. The performance of this network, shown in Table ('Naive training"), demonstrates large degradation of accuracy in both datasets (See also Table S4] for network's 's architecture). To facilitate fair comparison, we also trained such naive classifier with feature learning as done in DRC. The results here did not change significantly.
As more advanced baseline, we considered an averaged prediction (AP) of a feed-forward model over the T sampled frames. Namely, the estimated probability P& of class given by ár T Et= pL. where Pi are predictions of the above naive baseline. The situation here is similar to test time data augmentation (Perez& & Wang 2017) with sensor motion being the augmenter. Notably, the AP saturated with the number of time steps while our full system kept improving, as described below (Table []. Here application of teacher slightly improves the accuracy, and in the case of CiFAR-100, the improvement is significant (1.34% on average).
1
Next, we evaluated models where a CRnVGRU (resp. GRU) is connected before (resp. the top of) the last global average pooling layer of ResNet; we denote it as Resnet+convGRU (resp. ResNet+GRU) (Table S At their best, these models achieved accuracy lower by approximately 4% and 7% for CiFAR-10 10 and CiFAR-100 datasets respectively. compared to 5-step DRC w/o posi- tional information. The fact that these models and the AP achieve approximately equal performance indicates that trainable recurrent connectivity in top layers has little benefit over simplistic integra- tion. This is in contrast the DRC, where recurrent connectivity implemented the low layers. This result is not surprising since convolutional networks tend to develop invariance to small shifts (Zeiler Fergus 2014), such as those that DRC relies on, albeit with notable caveats (Azulay &
Weiss 2018).
Finally, we refer to a recent work (Xi et al. 2020) that uses a generative adversarial network to en- hance feature representation in CiFAR- 10 task with 8x8 resolution. This solution performs slightly better than DRC without positional information and five timesteps but underperforms versus the same DRC setting with ten steps. Furthermore, results for CiFAR-100 available in this work.
5

---

Published as a conference paper at ICLR 2022
Table 2:
Accuracy [%] of Small-net variants with 10 time-steps. Each column corresponds single realization. The version, marked by is used for further representation analysis.
depth position input
CiFAR-10
Standard resolution (32x32)
88.6
Low resolution (8x8) DRC
3
84.6
3
yes
85.2
6
no
85.8
6
yes
87.1
2.2.2
OUR MODEL
Table [ Summarizes the performance of the DRC on both datasets. The simplest version, with five time-steps and with no positional encoding outperforms the baseline solutions, including the GRU- based one. can be clearly seen that adding time steps or increasing the network's depth leveraged to higher accuracy in both datasets. The version with the ten time steps achieves accuracy which is just 2-4% inferior to the full resolution setting.
To check whether DRC advantage vs. baselines relies solely on the recurrent computation in the bottom layers, we tested without motion. The corresponding result in Table indicates that this nof the case.
Table 2 reports four representative examples of Small-nets trained on CiFAR-10. Here we see that same trends of performance hold for shallower and more compact network architecture and for teacher trained from scratch, without transfer learning.
2.3 REPRESENTATION
To better understand how our dynamical network extracts high-level features from low-resolution images, we analyzed the activation sensitivity of each of the 64 neurons of the final layer of the DRC- FE ("feature-neurowf neurons"),, We started by using activation maximization with gradient ascent over the input pattern space (Zeiler & Fergus 2014) and obtained the maximally-activatiee for the teacher network (Fig. (S7). Unfortunately, applying this tool to the spatio-temporal features learned by the student network failed to converge systematically (Fig. SS4). We thus developed a deep generator network (DGN) (Table [S9 and Appendix A), partially inspired by (Nguyen et al., 2016), that proved capable of repeatably producing AMs with spatio-tempone patterns, while remaining consistent with the results obtained using gradient ascent in purely spatial setting (Fig. S7).
patterns (AMs)
For each feature-neuron, we found specific series of images that maximized its activity, by allow- ing the generator to devise unconstrained ppatio-tempora patterns. As previously seen in (Zeiler & Fergus 2014), we found that Gabor-like images maximized the activation of many Ceature-neurone of the teacher network, reminiscent of the sensitivity of neurons in the early visual system to simi- lar stimuli (Carandini al., (2005). As expected, we also found that the Gabor-like patterns in (dynamic) students' feature-ncuron were often reminiscent of those of the (static) teacher. Impor- tantly, however, the features presented in our student network exhibited dynamics with high spatial and temporal variability, resembling visual receptive fields for drifting Gabors (Figures 2A, S8 S9)
To isolate the contribution of spatial and temporal variations to the AM of each feature-neuron we explored two settings of constrained maximization (see Fig. 2 for visualization of both con- straints). Specifically, we tasked the generator with creating either purely spatial input patterns, where all frames must be identical (see middle rows in examples at Fig. [ or purely temporal patterns in which differences between frames were allowed, but all the pixels of each individual
frame were identical (c.g.. bottom rows in Fig. B We found features that were more sensitive to temporal dynamics along with others that were more sensitive to spatial dynamics, with the majority of features exhibiting mixed spatio-temporal sensitivity patterns (Fig. 2
Interestingly, many features exhibited apatio-temporal AMs that were substantially higher than the corresponding purely spatial and purely temporal AMs, suggesting specific coding benefits for spatio-temporal fields in our dynamic network. This finding illustrates the importance of studying spatio-temporal receptive fields in the visual system (DeAngelis et al. 1993] Rust al. 2005).

---

Published as a conference paper at ICLR 2022
A.
B.
time
ST
T
30
Spatial Activation
90
10.5
00
0.
1.0
1.5
25
Temporal Activation
Figure 2: Spatial and temporal receptive fields in the top recurrent layer A. Examples of six pairs of teacher (high-resolution, left) and student (low-resolution, right) feature AM. Note the visual resemblance between the teacher's and student's s features and the dynamical nature of the student's features. The arrow illustrates time flow. B. Central plot: the values along the X- (and resp. the Y-)axis represent the activation values in constrained maximization setting when limiting the DGN to purely temporal (resp. spatial) changes. The size and the color of the dots represent the intensity of activity in the full unconstrained (spatio-temporal) maximization. Call-outs depict predominantly temporal (bottom-right dot), predominantly spatial (top-left dot), and mixed spatio- temporal (center-left dot) selectivity. Rows correspond to spatio-temporal (ST), purely spatial (S), and purely temporal (T) maximization. Columns correspond to timesteps where eight timesteps out of ten shown for visual clarity.
2.4 SENSOR'S 'S TRAJECTORY AND ITS EFFECT ON PERFORMANCE
While ocular drift is considered a diffusive, stochastic process, recent evidence suggests that its high-level properties are controlled by the brain stimulus mesk-dependent manner. particular, Gruber & Ahissar (2020): Intoy & Rucci (2020) demonstrated that ocular drifts in human subjects exhibit more curved paths when viewing more informative regions. We thus examined the effect of sensor trajectory recognition accuracy. We devised family of stochastic diffusive trajectories with controllable curvature properties. To do that, we assumed that the random distribution of angles between successive intervals is governed by von Mises distribution (Appendix B for details).
Testing our DRC with varying к (the parameter which controls the trajectory's 's mean curvature; Appendix B for details), found that the recognition performance improved with curvature, providing a possible functional interpretation for the experimental findings of (Gruber Abissar, 2020 Intoy & Rucci 2020). 3 gradual improvement in accuracy CiFAR- 100 dataset

---

Published as a conference paper at ICLR 2022
decreases. Representative trajectories for each tested value are shown in the top panel with their corresponding accuracy presented in the bottom panel.
To distinguish between the properties of the actual trajectory with the generated sequenee of points as opposed to the collection of sample points, we performed a shuffling experiment. Fig. B shows that randomly shuffling points the trajectory mostly affects trajectories with low curvature. For trajectories with negative kappa, shuffled trajectory visually resembles non-shuffled ones and, there- fore, it comes with surprise that shuffling has little no effect. On the other hand, for trajectories with low curvature shuffling does have significant effect. This implies that while the same infor- mation is available in both cases, the decoding is compromised the case of a shuffled path.
To leverage the advantage of the trajectory's 's curvature further, we devised another family of trajecto- ries for which curvature was explicitly enforced. We refer these trajectories as "spirals" (Appendix for details). As can be seen in Fig. the 'spiral' trajectories resulted in the best performance. The 'spiral' ensemble of trajectories is the one that is reported TablesTande
We used the curvature-ings index defined in Gruber & Ahissar 2020) (See also Appendix B in order
to compare our synthetic trajectories with real drift trajectories acquired an experiment (Gruber & Ahissar 2020). We found that by using values of -1.0 and 0.0 the generated distributions have similar characteristics (see details in Appendix [B) to the ones observed in the experiment with the relevant condition- recognizing small 2d shapes. The empirical distributions in (Gruber & Ahissar 2020) were obtained by dividing the data to periods where relevant information is projected on the retina, and periods where such stimulus is not available to the retina (compare Fig. S1O
with Fig. 3, "Natural-small' curvature-index distributions in Gruber & Ahissar (2020). These similarities demonstrate that (a) the range of were using is biologically relevant, and (b) our model can be used for exploring the mechanistic details underlying the biological control preferring such curvatures. as they demonstrate their advantage in recognition.
To conclude, we find that a curved motion of the sensor is beneficial for our DRC setting, offer- ing potential functional interpretation similar kinematies observed in human vision (Gruber & Ahissar 2020 Intoy & Rucci 2020).
K
20
K
K
2
==
K =
-1
K
-2
-5
spiral
105
105
100
100
100
100
100
100
100
100
90
100
100
90
100
95
100
95
100
95
100
95
100
95
100
75%
Accuracy
70%
65%
60%
== 20
K 5
K
==0
R=-1
K=
==
spiral
Figure 3: Sensor's trajectory and its impact performance. Top: representative examples of trajectories are shown for 5-time steps long trajectories with gradually increasing curvature, which corresponds to decreasing (Equation Spiral trajectories governed by Equation are also shown. See Appendix B for details. Bottom: accuracy for DRC performing time steps on CiFAR-100 is plotted for each setting for original (bluc) and randomly shuffled (orange) trajectories.
DISCUSSION
We introduced a dynamical recurrent classifier (DRC), a system that recruits tiny sensor motions to compensate for low spatial resolution with temporal over-sampling. This setting is novel, although
8

---

Published as a conference paper at ICLR 2022
some of its components had been used before (Kanazawa et al., 2021 Arefin et al. 2020). We introduced recurrent dynamics to the low layers of the network (the DRC-FE in Fig. LB), followed by time averaging and by feed forward convolutional layers. This architecture is reminiscent of the biological brain: the dynamics in early visual areas is faster and the integration windows are shorter than in higher areas. Therefore, the assumption of static (or slowly varying) representation in high areas is reasonable. Furthermore, high visual areas (c.g. V4 and ITT exhibit invariance to variety of stimulus distortions (Cadieu et al.) 2007] Rust & DiCarlo 2010), which is absent in the low areas (VI, V2). This fact is echoed by our training method which allows the student and the teacher to have different low level representations but implies similarity of representations from the point along processing hierarchy, upstream of which, a common architecture is used (DRC-BE in Fig. B
Moving the sensor over the image may be considered as yet another variant of test-time data aug- mentation. However, we find that the recurrent computation provides an extra benefit compared to the averaged prediction baseline [ corresponding to such an augmentation.
Importantly, the task of recognition from series of low-resolution frames differs from the related task of MISR (Farsiu et al. 2004; Arefin et al., 2020). While the latter task requires high-resolution reconstruction of the input scene, the former one does not. The DRC does not need to learn all the particularities required to reconstruct a high-resolution image; instead, it focuses on extracting the necessary features for the given task. Future work may compare the performance of the DRC with standard classifiers that use task driven super-resolution (Haris et al. 2018) as a preprocessing cameras, UAVs and self-driving cars with the need of performing real-time image recognition tasks, on a stream of visual input images captured from moving vision sensor of limited quality, e.g. (Desai al. 2015
step. These solutions applicable in variety of settings such as body
Merenda et al. 2020 HHward et al. 2017 Jiang et al. 2018). Table T Suggessts that in a setting with large number of samples and limited resolution DRC may be competitive e.g. vs. Xi Xi al. (2020).
Recently, Anderson et al. (2000) performed an approximate Bayesian inference to decode features that could account for the improvement in acuity observed in the experiments of [Ratnam et al. (017). Our approach relies on the assumption that the primitives are shaped by the stimuli in full resolution - i.e. in the regime of convenience rather than handcrafted, and are then adapted to the more challenging regime of low resolution. Furthermore the inference is performed by a trained neuronal agent as opposed to the idealistic Bayesian estimate in (Anderson et al., 2020).
A teacher network assisted our DRC in developing its latent neuronal representation. In biological terms, this would be analogous to hyperacuity being based on representations developed using reg- ular acuity. Consistent with this analogy are the findings that (i) the development of hyperacuity in humans follows the development of regular acuity (tested using Snellen tables) (Skoczenski & Nor- cia 2002 Wang et al. 2009) and (ii) recognizing the smallest Snellen optoypes. which improves with age (Wang et al. 2009). likely requires hyperacuity (Ratnam et al.] [017] Intoy & Rucei] 2020
Interestingly, there is evidence that the fixational drift contributes to the perception of the
small Snellen optotypes (Ratnam et al. 2017 Intoy & Rucci 2020).
The trajectory along which samples are taken affects the recognition accuracy, echoing experimental findings (Gruber& & Ahissar (2020) Intoy & Rucci 2020). Notably, the sensor trajectories in this work were generated independently of the underlying scene. This is a possibly sub-optimal situation, and future work may focus on closed-loop interaction between sensor trajectory and the perceived scene (Ahissar & Assa, 2016: Gruber al., [202T] Rucei et al.] 2018 Intoy & Rucei (020). This could also shed light on the ongoing effort to identify controllable ingredients in the ocular drift motion
(Ratnam et al., 2017).
The results of this work can be used when constructing specific hypotheses about the ways in which the visual system copes with tiny images. Specifically, our work suggests that the ocular drift plays major role in such conditions, supporting earlier hypotheses and empirical findings (Marshall &
Talbot 1942; Steiimaan & Levinson [990, Ahissar & Aricli (200,; [Sooddelly et al. [001]] Rucci
et al. 2007 Ratnam et al.) [017] Anderson 20000 (rruber & Ahissar 2020;; Gruber et al.)
2021 Intoy& & Rucci 2020) and that the processing of the drift-derived apatio-temport information
requires recurrent processing in retinal, sub-cortical or cortical visual networks (Burak al. 2010
Our results also support the inclusion of eye position signals in such processing (Burak et al.
2010
the accuracy required from such signals likely dictates that they should be derived from retinal
signals (e.g.. Ahissar et al. (2015)).

---

Published as a conference paper at ICLR 2022
4 ACKNOWLEDGEMENTS
We would like to thank Michal Irani for a valuable discussion at the early stage of this work and thank Liron Gruber for proofreading the paper. This project has received funding from the European Research Council under the European Union Horizon 2020 Research and Innovation Programme (Grant Agreement No. 786949). E.A. holds the Helen Diller Family Professorial Chair of Neurobi- ology.
REFERENCES
Ehud Ahissar and Amos Arieli. Figuring space by time. Neuron, ((2)):85-201,, 2001.
Ehud Ahissar and Amos Arieli. Seeing via miniature eye movements: dynamic hypothesis for vision. Frontiers in computational neuroscience, 6:89, 2012.
Ehud Ahissar and Eldad Assa. Perception as a closed-loop convergence process. Elife, 5:e12830, 2016.
Ehud Ahissar, Shira Ozana, and Amos Arieli. 1-d vision: Encoding of eye movements by simple receptive fields. Perception, 44(8-9):986- 9994999996-99 2015.
Alexander G Anderson, Kavitha Ratnam, Austin Roorda, and Bruno A Olshausen. High-acuity vision from retinal image motion. Journal of vision, 20(7):34-34, 2020.
Md Rifat Arefin, Vincent Michalski, Pierre-Lue St-Charles, Alfredo Kalaitzis, Sookyung Kim, Samira E Kahou, and Yoshua Bengio. Multi-image super-resolution for remote sensing using deep recurrent networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 106-207, 2020.
Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize so poorly small image transformations? arXiv preprint arXiv: 12177, 2018.
Nicolas Ballas, Li Yao, Chris Pal, and Aaron Courville. Delving deeper into convolutional networks for learning video representations. arXiv preprint 1511 1,06422, 2015.
H. B. Barlow. Reconstructing the visual image in space and time. Nature, 27965710):889-199 May 1979. ISSN 0028-0836, 1476-4687.
doi: 10.1038/279189dl URL http:// www nature.
com/ att.aaarcieee/t1eeaaeee
Vikranth R Bejjanki, Jeffrey M Beck, Zhong-Lin Lu, and Alexandre Pouget. Perceptual learning
as improved probabilistic inference in early sensory areas. Nature neuroscience, 14(5):642-644
2011.
Michael Berry, David K Warland, and Markus Meister. The structure and precision of retinal spike trains. Proceedings of the National Academy of Sciences, 94(10):5411-511 1997.
Goutam Bhat, Martin Danelljan, Lue Van Gool, and Radu Timofte. Deep burst super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9209-92111 2021.
G. Bradski. The OpenCV Library. Dr. Dobb's Journal of Software Tools, 2000.
Yoram Burak, Uri Rokni, Markus Meister, and Haim Sompolinsky. Bayesian model of dynamic image stabilization in the visual system. Proceedings of the National Academy of Sciences, 107 (45):19525-19530, 2010.
Charles Cadieu, Minjoon Kouh, Anitha Pasupathy, Charles E Connor, Maximilian Riesenhuber, and Tomaso Poggio. A model of v4 shape selectivity and invariance. Journal of neurophysiology. 98 (3):1733-1750, 2007.
Matteo Carandini, Jonathan B Demb, Valerio Mante, David J Tolhurst, Yang Dan, Bruno A Ol- shausen, Jack Gallant, and Nicole Rust. Do we know what the early visual system does? Journal of Neuroscience, 1077 2005.
10

---

Published as a conference paper at ICLR 2022
EJ Chichilnisky. A simple white noise analysis of neuronal light responses. Network: computation in neural systems, 12(2):199, 2001.
François Chollet. keras.
hww.r/ubbb.coomcccooeetkeeererae 2015.
Gregory C DeAngelis, Izumi Ohzawa, and RD Freeman. Spatiotemporal organization of simple-cell receptive fields in the cat's striate cortex. ii. linearity of temporal and spatial summation. Journal of Neurophyssolloy. 69(4):1118-1135, 1993.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255, Icve, 2009.
Soham Jayesh Desai, Mohammed Shoaib, and Arijit Raychowdhury. An ultra-low power, " "always- on" camera front-end for posture detection in body worm cameras using restricted boltzman ma- chines. IEEE transactions on multi-scale computing systems, 1 1(4):187-19, 2015.
Chao Dong. Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. IEEE transactions pattern analysis and machine intelligence, 38(2): 295-307, 2015.
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of deep network. University of Montreal, 1341(3):1, 2009.
Sina Farsiu, M Dirk Robinson, Michael Elad, and Peyman Milanfar. Fast and robust multiframe super resolution. IEEE transactions on image processing, 13(10):1327-1344 2004.
Baptiste Gauthier, Evelyn Eger, Guido Hesselmann, Anne-Lise Giraud, and Andreas Kleinschmidt. Temporal tuning properties along the human ventral visual stream. Journal of Neuroscience, 32 (41):14433-14444 2012.
Shiming Ge, Shengwei Zhao, Chenyu Li, and Jia Li. Low-resolution face recognition in the wild via selective knowledge distillation. IEEE Transactions Image Processing, 28(4):205122062, 2018.
Daniel Glasner, Shai Bagon, and Michal Irani. Super-resolution from a single image. In 2009 JEEE 12th international conference computer vision, pp. 349-356. IEEE, 2009.
Liron Zipora Gruber and Ehud Ahissar. Closed loop motor-sensory dynamics in human vision. PloS one, 15(10):c0200660 2020.
Liron Zipora Gruber, Shimon Ullman, and Ehud Ahissar. Oculo-retinal dynamics can explain the perception of minimal recognizable configurations. Proceedings of the National Academy of Sciences, 118(3)) 2021.
Muhammad Haris, Greg Shakhnarovich, and Norimichi Ukita. Task-driver super resolution: Object detection in low-resolution images. arXiv preprint 1803v /1316, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.025: 2015.
Shaul Hochstein and Merav Ahissar. View from the top: Hierarchics and reverse hierarchies in the visual system. Neuron, 36(5):691 2002.
Sonja S Hohl and Stephen G Lisberger. Representation of perceptually invisible image motion in extrastriate visual area mt of macaque monkeys. Journal of Neuroscience, 11(46)):6661-1666) 2011.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1704.00704 2017.

---

Published as a conference paper at ICLR 2022
D. H. Hubel and T. N. Wiesel. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. The Journal of Physiology, 160(1):106- Jamuaaa 1962. ISSN 0022- 3751.
www. ncbi n1mm gow. httpp//cc//a.....eee///C../C.11/
Janis Intoy and Michele Rucci. Finely tuned eye movements enhance visual acuity. Nature Com- municalions, 11(1), February 2020. ISSN 2041-1723, doi: 10. 16000///1117000--14466111
URL
https:/ / www. www.aatureee.mmmmm com artii eelccleessee / 01/p///44677220222 00tp://///.rrraeeee.16666000000-
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Junchen Jiang, Ganesh Ananthanarayanan Peter Bodik, Siddhartha Sen, and lon Stoica. Chameleon: scalable adaptation of video analytics. In Proceedings of the 2018 Conference of the ACM Special Interest Group Data Communication, pp. 253-266, 2018.
Hiroki Kanazawa, Yuta Nakamoto, Jiaxin Zhou, and Takashi Komuro. Human detection from low- resolution video images using 3d convolutional neural network. In Fiffeenth International Con- ference on Quality Control by Artificial Vision, volume 11794, pp. 11941GG. International Society for Optics and Photonics, 2021.
Kohitij Kar, Jonas Kubilius, Kailyn Schmidt, Blias B Issa, and James J DiCarlo. Evidence that recurrent circuits are critical to the ventral stream's 's execution of core object recognition behavior. Nature neuroscience, 22(6))974-955 2019.
Tim C Kietzmann, Courtney J Spoerer, Lynn KA Sörensen, Radoslaw M Cichy, Olaf Hauk, and Nikolaus Kriegeskorte. Recurrence is required to capture the representational dynamics of the human visual system. Proceedings of the National Academy of Sciences, 11(443)21854-21863, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers 2009.
of features from tiny images.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
Jonas Kubilius, Martin Schrimpf, Kohitij Kar, Ha Hong, Najib Majaj, Rishi Rajalingham, Elias Issa, Pouya Bashivan, Jonathan Prescott-Roy Roy, Kailyn Schmidt, al. Brain-like like object recognition with high- perforrffomig shallow recurrent anns. arXiv preprint 190.v:1909.06166 2019.
BB Lee, A Elepfandt, and V Virsu. Phase of responses to moving sinusoidal gratings in cells of cat retina and lateral geniculate nucleus. Journal of Neurophysiology, 45(5):807 817, 1981.
WR Levick, BG Cleland, MW Dubin, et al. Lateral geniculate neurons of cat: retinal inputs and physiology. Invest Ophthalmol, 1(5):302-311 1972.
Jianan Li, Xiaodan Liang, Yunchao Wei, Tingfa Xu, Jiashi Feng, and Shuicheng Yan. Perceptual generative adversarial networks for small object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1222-1230, 2017.
Jack Lindsey, Samuel A Ocko, Surya Ganguli, and Stephane Deny. A unified theory of carly visual representations from retina cortex through anatomically constrained deep cnns. arXiv preprint arXiv:1901.00945, 2019.
Tatiana Malevich, Antimo Buonocore, and Ziad M Hafed. Rapid stimulus-driven modulation of slow ocular position drifts. Elife, 9te57595, 2020.
WH Marshall and SA Talbot. Recent evidence for neural mechanisms in vision leading to general theory of sensory acuity. 1942.
Massimo Merenda, Carlo Porcaro, and Demetrio Iero. Edge machine learning for ai-enabled iot devices: A review. Sensors, 20(9):2533, 2020.
12

---

Published as a conference paper at ICLR 2022
Peyman Milanfar. Super-resolution imaging. CRC press, 2017.
Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. Recurrent models of visual attention. In Advances in neural information processing systems, pp. 2204 2212, 2014.
Aran Nayebi, Daniel Bear, Jonas Kubilius, Kohitij Kar, Surya Ganguli, David Sussillo, James J DiCarlo, and Daniel LK Yamins. Task-driven convolutional recurrent models of the visual system. arXiv preprini arXiv:1807 007.00053, 2018.
Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. Advances in neural information processing systems, 29:3387-3395, 2016.
Bence P Ólveczky, Stephen A Baccus, and Markus Meister. Segregation of object and background motion in the retina. Nature, 423(6938):401-408, 2003.
Luis Perez and Jason Wang. The effectiveness of data augmentation in image classification using deep learning. arXiv preprint arXiv:1712.04621,,222 2017.
Kavitha Ratnam, Niklas Domdei, Wolf M. Harmening, and Austin Roorda. Benefits of retinal image motion at the limits of spatial vision. Journal of Vision, 17(1):30, January 2017. ISSN 1534- 7362. doi: 10.1167/17.1300. htttstts: htttp::://www.cbb.... nlm. nih,
PMC52830837]
Pamela Reinagel and Clay Reid. Temporal coding of visual information in the thalamus. Journal of neuroscience, 20( (0(14))5392-5400, 2000.
Michele Rucci, Ramon Iovin, Martina Poletti, and Fabrizio Santini. Miniature eye movements enhance fine spatial detail. Nature, (47(7146):852-88)) 2007.
Michele Rucci, Ehud Ahissar, and David Burr. Temporal coding of visual space. Trends in cognitive sciences, 22(10):883-89) 2018.
Nicole C Rust and James J DiCarlo. Selectivity and tolerance ("invariance") both increase visual information propagates from cortical area v4 it. Journal of Neuroscience, 30(39): 2010.
Nicole Rust, Odelia Schwartz, Anthony Movshon, and Eero P Simoncelli. Spatiotemporal elements of macaque v1 receptive fields. Neuron, 46(6))945-956 2005.
Jason M Samonds, Brian R Potetz, Christopher W Tyler, and Tai Sing Lee. Recurrent connectivity can account for the dynamics of disparity processing in vl. Journal of Neuroscience, 33(7): 2934-2946, 2013.
Daisuke Shimaoka, Kenneth D Harris, and Matteo Carandini. Effects of arousal on mouse sensory 2018.
cortex depend on modality. Cell reports,
Ann M. Skoczenski and Anthony M. Norcia. Late Maturation of Visual Hyperacuity, Psychological Science, 13(6):537-541, November 2002. ISSN 0956-7976. doi: 10.1111/1467-92888 01167792200009999 URL httpt::tp/////// /ttp::::/eeoi.orgg1.11111111111. Inc.
9280. 00494
Publisher: SAGE Publications
Max Snodderly, Igor Kagan, and Moshe Gur. Selective activation of visual cortex neurons by fixational eye movements: implications for neural coding. Visual neuroscience, 18(2):259-277, 2001.
Robert M Steinman and John Z Levinson. The role of eye movement in the detection of contrast and spatial detail. Eye movements and their role in visual and cognitive processes, 4:1 15-21,,, 1990.
David A Van Valen, Takamasa Kudo, Keara M Lane, Derek N Macklin, Nicolas T Quach, Mialy M DeFelice, Inbal Maayan, Yu Tanouchi, Euan A Ashley, and Markus W Covert. Deep learning automates the quantitative analysis of individual cells in live-cell imaging experiments. PLoS computational biology, 12(1 1005505177 2016.
13

---

Published as a conference paper at ICLR 2022
Yi-Zhong Wang, Sarah E. Morale, Robert Cousins, and Eileen E. Birch. The Course of Devel- opment of Global Hyperacuity Over Lifespan. Optometry and vision science official publi- cation of the American Academy of Optometry, 86(6))695-770, June 2009. ISSN 1040-5488. doi: 11 articles/PMc2233222
URL. https:: pestps://www.noee /www./wwwwoomm.mmm/.ooo ncbi nlm. nih. gtv.phc//o/mc/////o////m/m
G. Westheimer. Hyperacuity. In Larry Squire (ed.), Encyclopedia of Neuroscience, pp.
45-50. Academic Press, Oxford, January 2009. ISBN 978-0-08-045046-9. doi: 10.1016/
187580088445046-9000221177 URL https:: //w/:///wwww../// sci encedi rect. wom sc ence/ atticcle/s1//77700855452222
Yue Xi, Jiangbin Zheng, Wenjing Jia, Xiangjian He, Hanhui Li, Zhuqiang Ren, and Kin-Man Lam. See clearly in the distance: Representation learning gan for low resolution object recognition. IEEE Access, 8:53203-53214 2020.
Daniel L. K. Yamins and James DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nature Neuroscience. 19(3), March 2016. ISSN 54461722. doi: 10.1038/mn. 4244. URL URL https:: www. nature com/ wwtaaaatiiclssme 4244
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In
European conference computer vision, pp. 818-833. Springer, 2014.
14

---

Published as a conference paper at ICLR 2022
A FEATURE VISUALIZATION TRAINING PROCEDURE
As a first attempt of feature visualization training we followed the general guidelines of (Erhan ct al, 2009 Nguyen et al. 2016). The code we used was based activation of a feature in a layer j of our network denoted by (8, where is the network's s parameters, and is the input. Using gradient ascent optimize the input to maximize the feature activation Hij- Following [Erhan et al. (2009) x was initialized randomly and the gradient ascent algorithm was applied moditying iteratively. While this procedure has been shown work with typical Convolutional Neural Network layers it failed to converge with the DRC-front-end probably due its higher dimensional input and complexity of the recurrent layers, (Fig. SS4) Therefore, we designed a new training procedure that includes deep generative network (DGN) which generates the synthetic input x, and uses gradient ascent update the DGN weights using the Adam optimizer. The DGN architecture is shown in Table S99 the architecture was modified to enable the constraints we enforced over the system, i.e. using 3D,2D and ID Transpose Convolution for the corresponding spatio-temporal, spatial-only and temporal-only activation maximization.
online available code example The
1.0
1.0
Activation (normalized)
Activation (normalized)
0.8
0.8
0.6
0.6
0.4
0.2
0.2
0.0
0 50
100 150 200 250 300 epochs
50
100 150 200 250 300
epochs
Supplementary Figure S4: Convergence of AM generator on the left, plot of the DGN converging stable AM feature, and on the right, non-convergence the standard gradient ascent algorithm. Data points were downsampled and normalized for clarity
Activation maximization is usually non-convex optimization problem, and therefore we do not expect to reach the same global maxima on each run. We applied a combination of two regularization methods, Batch-normalization over the networks layer output (loffe & Szegedy 2013) and added Gaussian noise, with μ = 0 and a = 0.05 empirically chosen, to obtain repeatability of AM in each individual feature, while maintaining significant difference between features. We averaged the results over ten different initialization of the generator to report more consistent results; see (Fig. S5 for examples of the convergence behavior of the algorithm.
B IMPLEMENTATION OF DRC TRAJECTORIES
B. TRAJECTORIES WITH CONTROLLED CURVATURE
To create a family of trajectories with controllable curvature, we assumed that at each time-step the sensor location 2tt), yt) is updated via polar increment dr, 86. Namely:
((t) = <(t 1) + (t(()
xxt) x(t 1) + 5r (o()))
y(t) = y(t 1 1) 5r sin (o()))
(1)
With do(t) being i.i.d. stochastic variables drawn from a von Mises distribution with controlled parameter Zero corresponds to uniform distribution do(t), positive values of correspond
1earn/
15

---

Published as a conference paper at ICLR 2022
A
30
25
2.0
01.
1.0
1.0
8
3334353637
Feature
Feature
B
25
2.0
2.0
1.0
1.0
0.5
0.5
12345778 101120244117770000002222222
04055667789900132444444441551113 0T0.A224444440000001110155555990 555555555555557775 758586666665
Feature
Feature
C
arb. unit
arb. unit
Feature
Supplementary Figure S5: Boxplots displaying the distribution intra-DRC AM values. The data represents six independent training trials of the generator with a single DRC-front-ent (A) boxplots of spatial only AM, we report the Coefficient of variation (CV) CV 0.043. (B) temporal only AM CV = 0. 134 (C) spatio-temporal AM CV = 0.066. The box represents the quarterlies of the data, and the whiskers mark the rest of the distribution. Outliers are marked in the graph. As can be seen, training the generators leads to a relatively consistent local minimum.
to straighter trajectories and negative к corresponds to more curved ones (here we define that for 0, dΦ dδ' with von Mises(-k)).
The second parameter, år was drawn from a half-normal distribution, so that år = + |ril where ~ NO, 1) and where we set To v2 to ensure that two consequent steps do not fall on the same point for any angle after rounding integer pixel coordinates.
B.2 SPIRALS
Spirals were created by setting:
6(1) の(1 1) + 56(1)
r(1) - r)
x(t) r(t) cos(
y(1) = r() sin()(())
(2)
with 5Ø(() ~ N. g) (00) drawn uniformly from circle, and the polarity of + is fixed along each individual trajectory. The parameters r(0) 3, 8r V(--0.100116" were picked heuristi- cally to optimize Small-net performance as well as prevent trajectories from coinciding. Regarding coinciding and repetition of trajectories, found that approximately 10.3K distinct trajectories were generated for a single pass over 45K large training set, making any fitting to specific trajectory unlikely.
16

---

Published as a conference paper at ICLR 2022
A
2.0
15
0.5
12345678 921487778177
588996616265
Feature
Feature
B
arb. unit
arb. unit
1234567 201121344516177 0155667718000222222222
2340553773350000000 044444444444450555 (200000555225555555555 188966616263
Feature
Feature
C
12
16
arb unit
arb. unit
1234567 9011270985111111111111 AAA.A444SS111100222222220002
23300007733500000000444 SWA.SSSASSSSSEE00EEESESSSSSSSSSS 1889960616663
Feature
Feature
Supplementary Figure S6: Boxplots display the distribution over inter-DRC AM values, comparing different DRC trained on the same teacher. The data represent three independent DRCs, each trained on the same Small-ne' -et teacher. The box notation is in the Fig[S5] Spatial only CV 0.1, temporal only CV = 0.16,
CV = 0.104.
B.3 CURVATURE INDEX DISTRIBUTIONS
The same curvature index as in Gruber & Ahissar (020) was used in order to compare our synthetic trajectories biological drift motion. The index is defined follows: CI 1- ds/l where the length of the trajectory and ds is the final displacement (i.e. the distance between the starting and ending point of the trajectory). Using this definition we get one index per trajectory where highly curved trajectory would result with number close to (and smaller than) one while low curvature trajectories would result with number close to (and greater than) zero.
Comparing the curvature index distributions obtained with with the relevant distributions from Gruber Ahissar (2020) (Figure 3, 'Natural-small' curvature- index distributions), where distribution of curvature index with -1.0 corresponds to 'border' drift trajectories distribution and distribution curvature index with k 0.0 corresponds to 'non- border' drift trajectory distribution, shifted to the left and broader compared to the λ = 1.0 ( ('border') distribution. The means the distributions are also comparable and exhibit the same trend: 0.699 + 0.002 and 0.597 + 0.003 for k 1.0 and i = 0.0 respectively, and 0.65 0.05 and 0.55 + 0.02 for 'border' and 'non-border" conditions respectively.
-1.0 and 0.0 (Fig. STOP
similar trends: The 0.0 ('non-border') distribution
B.4 INTEGRATION POSITIONAL INFORMATION
The positional information fed into the network contained the (x.y) normalized coordinates of the lower left corner each frame relative to the center position of the original CiFAR image. was integrated into the network first broadcasting it to the two dimensions (height and width) of
17

---

Published as a conference paper at ICLR 2022
Supplementary Table S3: DRC network architecture
DRC (time steps 10, for ResNet50: W=H=56, for Small-Net: W=H-8)
Layer
Number of Kernel size outputs
Padding
Activation
Function
(Small-
Net/ResNet)
Input
Input image
10*8*8*3
Input trajectory
10*2
Front-end
Broadcast trajectory
10"8"8"2
Concat
10*8*8*5
Upsample
x7
10*W*H*5
(ResNet50 only)
ConvGRU
30*
same
ELU/ReLU ELU/ReLU ELU/ReLU
ConvGRU
10*
same
COnVGRU
H0*
3 3
same
IveragePoollnggI
W*H*64
10
LayerNormalization
Back-end
Weights Copied from
Teacher
BesNet+Classifeer
10/100
softmax
8 "same" " padding refers to padding to the input image so that the input image gets fully covered by the filter and specified stride
Supplementary Table S4: Baseline ResNet50 based network architecture
Reference Net
Layer
Number of outputs
Kernel size
Padding
Activation
Function
ResNet50
14*14*2048
Classifier
GrobolAvrgPPoling
7*7#2048
2
Fully Connected
1024
ReLU ReLU softmax
Fully Connected
512
Fully Connected
10/100
single input frame (8,8,) and then concatenating with the frame. resulting in a (8,8,5 input for each time-step (three dimensions for RGB and two dimensions for location).
For ResNet+convGRU controls the positional information was fed similarly to the first recurrent layer, with appropriate scaling (approximately square root of the layer depth).
18

---

Published as a conference paper at ICLR 2022
Supplementary Table S5: ResNet+GRU network architecture (applied to time series)
Control Net v2
Layer
Number of outputs
Kernel size
Padding
Activation Function
ResNet50 (applied at each timestep)
5*14414"2048
Classifier
Grrgoonvg
52787"2048
GRU
5*1024
sigmoid-tanh sigmoid-tanh softmax
GRU
512
Fully Connected
10/100
Supplementary Table S6: ResNet+ConvGRU network architecture (applied time series)
Control Net v3
Layer
Number outputs
of
Kernel size
Padding
Activation Function
ResNet50 (applied at each timestep)
5#14*14#2048
Classifier
ConvGRU ConvGRU
5*1024
3*3
sigmoid sigmoid-tanh
512
3+3
Global AvrgPoling
5*7*7#512
Fully Connected
10/100
softmax
19

---

Published as a conference paper at ICLR 2022
Supplementary Table S7: Base Small-net network architecture
Teacher
Layer
Number outputs
Kernel size
Padding
Activation Function
Front-end
Input image
32*32*3
Convolution
32*32*32
3 3
same same
ELU ELU
Convolution
32*32*32
3 3
MaxPooling2D
16*16*32
2 2
Dropout 0.2
Convolution
16*16*64
3 * 3
same same
ELU
Convolution
16*16*64
3
ELU
MaxPooling2D
8*8*64
2 2
LayerNormalization
Back-end
3 3, 128
Convolution ResNet
8*8$64
3 + 128
3
same
ELU
* 3, 64
LayerNormallzztion Fully Connected
128
ELU
Dropout - 0.0
Fully Connected
10/100
softmax
Supplementary Table S8: Small-net v2 network architecture
Teacher Kernel size
Layer
Number outputs
of
Padding
Activation Function
Front-end
Input image
32*32*3
Convolution
32#32"32
same same
ELU ELU
Convolution
32*32"64
3 3
MaxPooling2D Dropout - 0.2
16*16*60
3 3, 128
Convolution ResNet
16*16*6
3 * 3, 128
3
ELU
* 3, 64
MaxPooling2D LayerNormalization
8*8*64
2
Back-end
3 x 128
Convolution ResNet
8*8$64
3 * 128
3
same
ELU
3 3, 64
LayerNormalization
Fully Connected
128
ELU
Dropout 0.0
-.00
Fully Connected
10/100
softmax
20

---

Published as a conference paper at ICLR 2022
Supplementary Table S9: Activation Maximization DGN
Generator of
Layer
Number outputs
Kernel size
stride
Padding
Activation
Function
input
100
Fully Connected
128
LeakyReLU
Reshape
1$1*1$128
CTranSSTe
122
2
2
valid valid valid
LeakyReLU
Conv3 3Transpoaasppose
5*4 128
3
8
2
*
2
LeakyReLU
Cons3ITTranspose
10*8*8*128
LeakyReI.U
Cony3IDTranspose
10*8*8*3
3
*
1 *
tanh
Gradient Ascent Over Pixels
Deep Generative Network Model
Supplementary Figure S7: Comparing results of activation maximization the teacher and student networks. Left: results of gradient ascent over the input pixels for the teacher network. Right: output of DGN trained to generate maximizing input for the student network. The easily observed similarity between the panels confirms the validity of the DGN approach activation maximization methods for spatial networks.
21

---

Published as a conference paper at ICLR 2022
2
8
10
11
12
13
14
15
16
17
18
20
21
22
23
24
25
26
27
28
29
30
31
Supplementary Figure S8: Teacher features vs. student features. Same as Fig, 2 for all the features.
22

---

Published as a conference paper at ICLR 2022
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
Supplementary Figure S9: Teacher features vs student features, continued. Same as Fig. 2A for all the features.
23

---

Published as a conference paper at ICLR 2022
3.0
Probability Density Function
K=-10
2.5
k=0.0
2.0
15
10
0.5
0.0
0.2
0.6 Curvature index
0.4
0.8
Supplementary Figure $10: Dsitributions curvature index. Dsitributions of curvature index of trajectories generated with -1.0 (blue) and к = 0.0 (orange).
24


Reviews:
[{"review_id": "ncWhjtl_h_r", "reviewer": "Reviewer_zsN3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The overall thesis here is very interesting, however there are numerous concerns:\n\nOne of the main claims of this paper is that dynamic retinal input combined with recurrent processing is key to how the brain manages to do hyperacuity.  The results of Table 1 seem at first glance to support this.  However none of the baseline models considered the most important control:  what if you just feed in a series of static images without motion to the DRC-FE?  As is, all that we can conclude from this paper is that a recurrent network in the early stages somehow helps, but it is not clear that the motion in the input image has anything to do with this.  In fact, if the motion did help, it would beg even more questions.  The 8x8 images were created by downsampling from the 32x32 images with bicubic interpolation - essentially smoothing or lowpass filtering.  If you simply move and resample a lowpass filtered image, there is no new information that can be exploited by later information processing, assuming that it was lowpass filtered below the nyquist rate for an 8x8 image (which presumably it was, an important detail that is missing) - this is given by basic signal processing.   It seems plausible that recurrent computation in the early layers helps - it is essentially like making a deeper network - but it would appear the effect has nothing to do with the motion in the input.\n\nThe paper seems motivated by neuroscience and psychophysics, but there is very little attempt to tie anything about the neural architecture of the model to substrates in the brain.  For example it is mentioned that neurons exhibit temporal dynamics with phasic responses, but none of this is incorporated in the model.  This seems like run of the mill deep convnet engineering as opposed to neuroscience.  I'm not sure what we learn here from a neuroscience point of view.\n\nThere is no overall theory presented as to how the brain could benefit from motion of the sensor in building a higher acuity representation enabling tasks such as hyperacuity.  There is much verbal reasoning in the introduction, however there is now much engineering and mathematical know-how about how such problems can be solved - e.g., super-resolution.  These works are mentioned at the end in the discussion, but then almost immediately dismissed because they reconstruct the image rather than doing recognition.  This is a shame because the theory behind these models is exactly what the authors need to implement their idea.  Instead, all of the requisite established theory is tossed aside and the authors resort to training a neural network to solve the problem, yielding a non-transparent solution providing little insight into how the brain might actually solve this problem.\n\nThe introduction does not properly attribute prior work.  First, Rucci et al have been writing and talking about the benefits of image motion for more than a decade now, but you wouldn't know this by reading the intro.  Although Rucci is cited, it is about drift motion in general and not with regard to his theory of *why* image motion is helpful, which is well known in the vision science community.  Burak's (2010) important earlier work is cited but misattributed as providing an account for how how drift motion could improve acuity, which is wrong.  Burak's model shows how the cortex could disentangle shape from motion from retinal spike trains so as to recover shape information on the retina, but does not address the question of why the motion may be beneficial to begin with.  Also missing in the intro is any mention of Ratnam et al. (2017) and Anderson et al. (2020).  Those works are brought up in discussion at the end, but given the high degree of relevance of these prior works to the authors' thesis it is baffling why they are not brought up earlier, especially with regard to what the authors hope to do here that goes beyond or improves upon this prior work.\n", "summary_of_the_paper": "The authors train a neural network to do object recognition on downsampled, moving images of objects.  They show that by using a recurrent neural network in the early layers, it can learn to produce representations that result in recognition performance nearly as good as with static, full resolution images.\n\n", "summary_of_the_review": "An interesting idea but implementation is problematic.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hRJhBlsk0e", "reviewer": "Reviewer_RJzH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The use of an \"active sensor\" (i.e. jittering the input image) is an interesting idea that capitalizes on recent developments in neuroscience and psychology.\n\n2. On first blush the results are relatively strong (however with a caveat that more controls are needed to interpret them)\n\nMajor issues:\n\n1. A central claim made by the authors is that spatio-temporal computations *in the front-end of the network* are important. The main evidence here is that the ResNet+RNN network, i.e. putting the recurrent computations on the *back-end*, does not work nearly as well. In fact, ResNet+RNN appears to do no better than simply averaging the prediction of ResNet over 5 frames, which is surprising. This needs to be evaluated much more systematically, since it is not an apples-to-apples comparison. Why is the RNN only used *after* the global average pooling layer? Here, the DRC is using convGRU while the comparison is made with vanilla GRU units in the ResNet+RNN network, so they do not have access to spatial information. So really, the comparison is spatiotemporal computation for the DRC network and temporal only computation for ResNet+RNN.  It is also unclear how the ResNet+RNN network incorporates spatial information since I could not find the parameters related to \"Input Trajectory\" in the appendix. It is also unclear how the ResNet+RNN network was trained.\n\n2. Figure 2 demonstrates that the DRC network uses a mixture of spatial and temporal computation, but the results are under-analyzed.  Does the network produce a similar distribution across different random initializations? Does the performance covary with these distributions? What happens if units with specific criteria are ablated? Much more analysis is needed to be able to interpret the importance of what's shown in Figure 2. \n\n3. Another central claim is that the trajectory of images over time is important (Fig. 3). In general this result is under-analyzed and difficult to interpret as is. As k becomes more negative and the trajectories more curved, trajectories are likelier to remain closer to the center and have more overlap, yet I could not find any analysis of this. If indeed curvature matters, then the authors must show that curved trajectories are better-performing than other trajectories with less curvature but similar aggregate statistics (e.g. the trajectories are a similar distance from the center-point and have similar degrees of overlap). A very simple control here is to shuffle the trajectories over time, in this case the statistics should be the same, but the degree of curvature from point to point will be destroyed. If the DRC network performs just as well with the shuffle, then the overall statistics matter more than curvature.  This is essential to understanding what allows the DRC network to perform well.\n\nMinor issues:\n\n1. Please check for typos\n2. Figure 3 please provide a legend\n3. Figure 3 I do not understand why the authors are plotting an average of *2* datapoints.  Many more points should be computed to estimate the distribution properly, so we can visualize a reasonable confidence interval for each parameter setting.", "summary_of_the_paper": "Here, the authors attempt to leverage spatio-temporal computations for object recognition on the standard CIFAR-10 and CIFAR-100 datasets. In short, they use a network with a front-end of recurrent units (ConvGRU) to recognize objects given spatially jittered downsampled images – effectively approximating an active sensor. The network is trained in a student-teacher configuration, where weights in a temporal pooling layer after the recurrent layers are trained to match the weights of a feature layer inResNet50. Next, the network is fine-tuned to increase classification accuracy.\n\nAltogether, the authors are asking if spatio-temporal computations are enough to produce a feature layer similar to a larger network train on full-res images and, in turn, if this feature layer supports object recognition on par with full-res performance of ResNet50. The authors demonstrate that their network is almost as performant as ResNet50 with 4x downsampled images, especially when the down-sampled images are jittered in a spiral formation. They also present analysis demonstrating that the network is in fact performing spatio-temporal calculations.", "summary_of_the_review": "In summary, I found the core idea of author's network interesting; however, the author's claims are currently not justified by the results. Many more controls are needed to ensure that the DRC network performs better than alternatives. If the DRC network does perform better than all other control networks, then additional analysis is required to understand how the network is able to improve its performance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IVUcL2GtItb", "reviewer": "Reviewer_9EUA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "pros:\n* as far as I can tell, this is a novel setting and I have not seen much work investigating the impact of low retinal resolution on object recognition models\n* the results on CIFAR-10 and CIFAR-100 are clearly described and show that the recurrent DRC model aided by a full-resolution teacher can regain most of the performance of a standard resolution model\n* the paper provides good background on the biological motivation for modeling low-resolution photoreceptors\n\ncons:\n* lack of connection to biology: the proposed model is motivated from biological observations, but model predictions are never tested against any experimental results. Are the model's resulting features any more brain-like? Does it exhibit the same hyperacuity as observed in biology?\n* requirement of a teacher: the DRC is only tested when learning representations from a teacher which both has a non-obvious connection to biology and is an unfair comparison to the non-recurrent baselines which do not use a teacher. Would any of the baselines perform better when trained with a full-resolution teacher in the same way as the DRC?\n* unclear benefits for computer vision: it is not obvious to me if/where processing sensory data with low resolution but many temporal samples will be helpful to the machine learning community. Some connection is made in the very last paragraph to always-on cameras such as body worn cameras but it is not made clear if those are really in the regime of low-resolution and high temporal sampling.\n\nminor: \n* some more discussion of related recurrent models, e.g. https://papers.nips.cc/paper/2019/hash/7813d1590d28a7dd372ad54b5d29d033-Abstract.html and https://www.pnas.org/content/116/43/21854.short, would be helpful to contextualize the work\n* the second-to-last paragraph on page 4 states that the ResNet+RNN \"achieved accuracy lower by 3.5% and 10%\" respectively for CIFAR-10/100, but Table 2 has its accuracy as 83.94/59.61 compared to the standard resolution 96.83/82.94 -- this seems to be inconsistent\n* it is not clear to me what to take from the visualization of features (figs. 2 and 3). In general, I found the paper a bit hard to follow at times, the consistent story is not clear to me. E.g. how do the feature visualizations support the main claim?\n* figure 3 caption has a typo: \"The wors case\"", "summary_of_the_paper": "The paper's main claim is that recurrence aids to enhance visual acuity in settings with limited resolution, such as the one imposed by limited photoreceptors in the retina. The authors therefore build a convolutional network with recurrent connectivity in its early layers (termed DRC) that receives a time-series of low resolution frames and learns representations -- for classification in CIFAR -- from a teacher network receiving full resolution inputs. DRC outperforms a low-resolution baseline and approaches standard resolution performance. Additionally, the paper visualizes the DRC's learned features.", "summary_of_the_review": "The paper lacks a clear demonstration of usefulness: either an improved fit to biological data (since the motivation starts from limited sampling in the retina), or a clear use case in computer vision. Since neither is demonstrated, I find it really hard to contextualize the work and cannot tell if the proposed model makes any improvements over previous models (see the main review for detailed suggestions). The use of a full-resolution teacher network is also not well motivated especially in connection to biology, and the second half of the paper is a bit hard to follow (i.e. what to take from the feature visualizations).\n\nREBUTTAL UPDATE: I have increased my score following the authors' attempts at connecting to biology more directly, but I still believe key comparisons are missing: either a stronger link to biology and concretely relating model predictions to experimental results, and/or explicit comparisons to alternative models in ML tasks.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GKJ2i8DJcz", "reviewer": "Reviewer_vR8C", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well-written, proposes a highly innovative model that is consistent with behavioral and neural data, and obtains excellent results. The paper addresses a long-neglected aspect of human vision (fixation drift) in neurocomputational models of human vision, and shows that it has efficacy in challenging conditions. They don't stop at demonstrating that the model improves accuracy over a static model that uses multiple images. They develop a method for assessing the dynamical features of their model, because the usual activation maximization technique doesn't work in this setting. Finally, they demonstrate that a recently discovered phenomenon, curved paths in the fixational drift, promotes higher classification accuracy.\n\nThe front end of the model is a two-layer, recurrent convolutional network. It is trained by feature distillation from a layer of ResNet 50. The ResNet 50 is pre-trained on imagenet and then fine-tuned on either CIFAR-10 or CIFAR-100. The recurrent net is provided with 8X8 images, and trained to match the features activated by the 32X32 versions in the ResNet. The inputs are shifted slightly based on dynamical difference equations with random perturbations that determined the x,y coordinates of the next input, simulating fixation drift. The network was trained to reproduce the activations of the teacher network after 5 or 10 inputs. Then, the output of this front end was input to the remaining layers of the ResNet50 network, which was then fine-tuned to improve performance. The baselines are quite reasonable: a network trained directly on the 8X8 images, the same network, but using the average prediction over the 5 or 10 images, a ResNet + RNN network trained on a sequence of 5 8X8 images, with or without positional information. They show that with increasing number of inputs (5 images or 10 images), performance of the DRC improves, while the static network flatlines at 5 images. Positional information also improves performance. In the end, a 10-step DRC network with positional information achieves performance nearly as good as the original 32X32 ResNet50 in both CIFAR 10 and CIFAR 100. \n\nThey then go on to analyze the features. They perform the usual gradient ascent procedure to obtain maximally-activating 32X32 inputs for the features of the ResNet50 network used to train the features of the DRC network. They find that this same procedure doesn't converge for the DRC network, so they have to invent a novel technique for finding the optimal features. They use the idea of the generative network (Nguyen, et al., 2016), modified for their setting. The generative network has to learn to generate a *sequence* of 8X8 images that maximally activate the DRC features. These resemble the corresponding ResNet features they were trained on, but obviously have dynamics. To evaluate the spatial and temporal aspects of these units, they also apply the same procedure, but only allow the generative network to generate one image that is repeated, giving the best spatial activation of the feature, or, they only allow the generative network to vary the images, but all the images have to have the same pixel everywhere, giving the best temporal activation, but without form. These activations generally aren't as high as the unconstrained optimization. It took me a while to parse Figure 2B, but once I figured it out, it was reasonably clear. \n\nFinally, they set up the fixation location dynamics in such a way that they can control the curvature of the drift. They find that more curvature in the drift dynamics, the better the accuracy. In fact, an enforced \"spiral\" dynamics gives the best results. It turns out the performance data is based on this model, which is about 4% better than the less constrained model. This is interesting because it accords with recent human data from Michele Rucci's lab that finds curved drifts are used by subjects when the recognition problem is challenging. (I haven't read that paper, so I don't know how faithful they are to Rucci's data, or that this correctly describes his results).\n\nWeaknesses, with concrete, actionable feedback\n\nThe weaknesses are mainly in the exposition: I had several clarification questions:\n\nIt is unclear what the representation of the positional information is.\n\nIs the RNN an LSTM network?  \n\nIn general, I'm confused about the role of \"Small-net\" in this paper. Please clarify.\n\nThe procedure by which the generative network determines the optimal features is not clear - this could be described more clearly. The supplementary material is insufficient in this regard. You have an unused half-page in the main text, so that should be enough room to elucidate how this is done. \n\nMinor comments, wording, etc.\n\nPage 1, 3rd line from the bottom: dominate -> have dominated\n\nFirst sentence in section 2.1.1: -> We applied a feature distillation learning paradigm...\n\nNext paragraph: therefor -> therefore. Spell check!\n\nAlso in this paragraph, I initially thought you were saying you applied feature distillation to an 8X8 layer using 56X56 features, which is not what you did. This is one of those places where the role of Small-net is unclear. \n\nstackup -> processing stack\n\ncosyne -> cosine\n\nOur model was mostly implemented in *the* Keras package ... with *the* convolutional GRU...\n\nIn the sentence beginning \"The accuracy of the reference (teacher)...\", it isn't clear which entry in the table you are referring to here. I believe it is \"Naive training\", so call it that in this sentence.\n\nMiddle of page 6: \nsaptio-temporal -> spatio-temporal. Spell check!\n\nproperty -> properties\n\nVarious places: use two left apostrophes (below the tilde on the standard keyboard instead of \" in LaTeX on the left side of a word. (e.g., \"spirals\" near the bottom of page 6).\n\nIn Figure 2B, you say you are showing predominantly temporal, predominantly spatial, and mixed examples here. If that's the case, I would expect one call-out to be from the far left point in the upper-left hand corner (predominantly spatial - your choice is reasonable here, but the point to the left of it would be even better), a point in the lower right-hand corner (predominantly temporal), and then the third one you show. The point you use from the lower left hand corner corresponds to 0 temporal and low spatial. So, there isn't a \"predominantly temporal\" example here. Can you pick one from the lower-right hand corner instead?\n\nThird line of Figure 2 caption: students -> student's\n\nthird line from the bottom of page 7: reported at Tables... -> reported in Tables...\n\nlast sentence in Figure 3 caption: wors -> worse. Spell check! Don't annoy your reviewers!\n\nWording suggestion for Discussion: \n\nThis setting is novel and has been hardly addressed in the...->\nThis setting is novel and has been mostly neglected in the...\n\nmiddle of page 8: stack-up -> architecture\n\nlast word in third paragraph from the bottom of page 8 is not the one you want!\n\nprepossessing. -> preprocessing step.\n\nFurthermore -> Furthermore,\n\nto idealistic -> to the idealistic \n\nFirst line, last paragraph: it sets -> our work sets\n\nlast sentence, last paragraph: This is not really a sentence in english. \nRewrite as: This is enabled by a solution...\n\ncaption of supplementary Figure S4: The second sentence is garbled. It needs a \"right\" somewhere, or \"compared to\"\n\nI don't know what \"same\" means in the padding column in your supplementary tables. Same as what?\n", "summary_of_the_paper": "This paper takes inspiration from the biological phenomenon of fixation drift, slow, low-amplitude movements during fixation that are believed to result in hyper-resolution in human vision. They hypothesize that this phenomenon can be explained by a model that has a recurrent convolutional front end that integrates over fixation drift, feeding into a well-trained back-end from a conventional model (ResNet 50). They demonstrate that this \"Dynamical Recurrent Classifier\" (DRC) is capable of restoring performance on 8X8 images to nearly the performance on \"high\" resolution 32X32 CIFAR images (actually, no one would call 32X32 high resolution!). They analyze the representations learned by the model and show they have strong spatio-temporal features, with some learned features emphasizing spatial features, some emphasizing temporal features, but most combine the two. Finally, they show that using curved trajectories improves performance over more random walks, which can potentially explain recent results in humans. They suggest this model can be useful in AI applications involving limited resolution but with multiple samples over time. ", "summary_of_the_review": "This paper is well-written, proposes a highly innovative model that is consistent with behavioral and neural data, and obtains excellent results. The paper addresses a long-neglected aspect of human vision (fixation drift) in neurocomputational models of human vision, and shows that it has efficacy in challenging conditions. This result suggests that it can be used in engineering applications where the stimuli are low-resolution. It has some confusing parts, but these can be fixed by the authors. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores the idea that fixational drift of a sensor over an image (something that primate eyes do) could be used to achieve visual hyperacuity, i.e. image recognition with low resolution images equivalent to what would be achieved with high resolution images. The authors construct networks where the bottom of a deep convnet is replaced by recurrent networks and the network is then trained on low-resolution versions of high-resolution images that are sampled with fixational drift across the image. The authors show that this approach allows their system (dynamical recurrent classifier, or DRC) to get much better classification performance on CIFAR images than can be achieved without the early recurrence and drift. The authors also show that the most robust classification mandates drift trajectories with higher curvature, and they show that this matches some of the properties of visual drift trajectories in humans. 

The reviews on this paper were highly divergent (ranging from 3 to 10). Three of the reviewers felt this paper should be rejected, but one felt very strongly it should be accepted. The primary concerns from the negative reviewers were lack of appropriate controls, lack of insight into why the system works, lack of appropriate references to past work, and lack of connection to biology. The authors made a very concerted effort to attend to all of the reviewers' comments. They ran all of the requested control experiments, updated the text to better reflect past literature, and included some comparison to psychophysics data. In the end, only one reviewer increased their score, though, leading to final scores of 3, 10, 5, and 3. Discussion did not lead to any more consensus. 

Thus, this paper was still very much in the borderline zone, and required AC consideration. After reading through the paper, reviews, and rebuttals, the AC felt that the authors really had addressed the primary concerns as best as could be hoped for in the time-frame for ICLR, and that the paper was sufficiently interesting and informative for ML and neuroscience to be worthy of publication. Some of the negative review points stand, e.g. there are still some mysteries as to why this works and there is certainly a lot more that could be done to make this paper informative for neuroscience. Nonetheless, in total, the AC felt that this paper deserved to be accepted, given that the authors did most of what the reviewers requested of them.
Decision: Accept (Poster)