Visual hyperacuity with moving sensor and recurrent neural computations

1 INTRODUCTION . Biological vision is known to be a dynamical process . Two factors contributing to these dynamics are eye motion and recurrent neuronal connections in the brain . Our eyes move constantly with movements that , kinematically , can be divided into saccades - quick gaze shifts , and drifts - small scanning movements between saccades ( often referred to as “ fixational drift ” ) ( Rucci et al. , 2018 ) . These dynamical aspects of vision are reflected only partially in contemporary computer vision systems . Some works addressed large scale shifts in visual attention resembling saccades ( Mnih et al. , 2014 ) . Others explored properties and benefits of recurrent top down connections ( Nayebi et al. , 2018 ) , reminiscent of top-down processing in biological vision ( Hochstein & Ahissar , 2002 ) . Notably , the dynamics of low-level visual processes , occurring early in the bottom-up visual hierarchy and sensitive to the fixational drift ( Snodderly et al. , 2001 ; Ölveczky et al. , 2003 ; Malevich et al. , 2020 ; Hohl & Lisberger , 2011 ) , remains largely overlooked in models of vision as well as in bio-inspired computer vision systems . In fact , since the seminal studies by Hubel & Wiesel ( 1962 ) , selectivity in primary visual cortex has been traditionally described in terms of static spatial filters ( e.g. , simple and complex spatial fields or Gabors of varying frequency and orientation ) . In convolutional neural networks ( CNNs ) ( Krizhevsky et al. , 2012 ) , which have dominated computer vision over the last decade , features resembling the spatial filters deduced from biological studies emerge spontaneously over the course of training ( Zeiler & Fergus , 2014 ; Lindsey et al. , 2019 ) . In some cases , remarkable correlations were found between spatial neural representations in CNNs and those identified in the biological brain ( Yamins & DiCarlo , 2016 ) . On the other hand , temporal dynamics , and sensitivity to temporal features , characterize visual neurons throughout the visual system , from retinal receptors and ganglion cells to thalamic and cortical neurons ( Berry et al. , 1997 ; Chichilnisky , 2001 ; Lee et al. , 1981 ; Levick et al. , 1972 ; Reinagel & Reid , 2000 ; Shimaoka et al. , 2018 ) . Existing evidence suggests that both eye motion ( Snodderly et al. , 2001 ; Ahissar & Arieli , 2001 ; Ölveczky et al. , 2003 ; Malevich et al. , 2020 ; Gruber et al. , 2021 ; Hohl & Lisberger , 2011 ) and recurrent neuronal connectivity ( Bejjanki et al. , 2011 ; Samonds et al. , 2013 ) contribute to this temporal dynamics . Furthermore , it was found that recurrent connections improve correlates of artificial neural networks to neural activity in visual cortical areas ( Kar et al. , 2019 ; Kubilius et al. , 2019 ; Kietzmann et al. , 2019 ) . One niche where spatio-temporal computation is probably necessary is the perception of tiny objects . It is well known that the acuity of biological vision is not limited by the spatial resolution of retinal photoreceptors ( “ visual hyperacuity ” ; Westheimer ( 2009 ) ; Barlow ( 1979 ) ) . Vernier acuity , for example , is dramatically higher than might be expected from pure spatial acuity derived from the photoreceptor density in the retinal mosaic ( Westheimer , 2009 ) . Whether hyperacuity is obtained via spatial , temporal , or spatio-temporal mechanisms is not yet known ( Rucci et al. , 2018 ) . In any case , it is evident that the visual processing allowing hyperacuity , or perception of any tiny stimulus , should cope with the fixational drift ; if it doesn ’ t , the drift , whose amplitude is at least two orders of magnitude larger than the smallest perceivable spatial offsets , would impair acuity ( Ahissar & Arieli , 2001 ; Rucci et al. , 2018 ; Ratnam et al. , 2017 ) . The same drift motion could potentially improve acuity if spatio-temporal computations are employed . Such computations can be based on the emphasis of high-frequency spatial details ( Rucci et al. , 2007 ) , temporal coding of spatial offsets ( Ahissar & Arieli , 2001 ; 2012 ) , Bayesian inference ( Anderson et al. , 2020 ) , or on any other derivative of the interactions between ocular motion and the external image . Furthermore , it is reasonable to attribute such spatio-temporal computations to early visual areas , which are known to exhibit faster dynamics and shorter integration windows compared to regions upstream in the visual processing chain ( Gauthier et al. , 2012 ) . Indeed , it had been shown that the recurrent neuronal circuitry in early visual areas could enable countering the blurring from retinal motion ( Burak et al. , 2010 ) . Using the information available from over-sampling low-resolution images has an extensive history in computer vision as part of the field of super-resolution ( Milanfar , 2017 ) . Multi-image superresolution ( MISR ) ( Farsiu et al. , 2004 ) , distinguished from single-image super-resolution ( Glasner et al. , 2009 ; Dong et al. , 2015 ) , aims to reconstruct high-resolution images from a set of lowresolution ones ( Arefin et al. , 2020 ; Ge et al. , 2018 ; Bhat et al. , 2021 ; Li et al. , 2017 ) . An adjacent field of research , low-resolution object recognition , investigates algorithms to maximize the performance on a given task ( Xi et al. , 2020 ; Ge et al. , 2018 ) . Both fields use low-resolution images as input but differ in the goal of the training and evaluation . In this paper , we introduce a classifier that exploits spatio-temporal computations in early layers to perceive tiny images . More specifically , we trained a convolutional neural network with recurrent connectivity ( Arefin et al. , 2020 ) introduced to early layers . The network receives a sequence of low-resolution images generated via sensor motion mimicking ocular drift . We used high-resolution images to obtain a set of features that were then used to facilitate learning in a teacher-student framework ( Hinton et al. , 2015 ) . The outcome is a dynamical classifier that suffers from only a small drop in accuracy when tasked with a significant decrease in spatial resolution , a decrease that substantially impairs the accuracy of a comparable static feed-forward classifier . Using a novel generative model , we found that our dynamical classifier developed features that were primarily sensitive to spatial changes , others that were primarily sensitive to temporal changes , and a majority that exhibited sensitivity to mixed spatio-temporal patterns . Finally , when examining the correlations between patterns of motion and accuracy of classification , we observed that curved trajectories are favorable for recognition , consistent with recent findings of the curvature of fixational drift trajectories in humans . ( Intoy & Rucci , 2020 ; Gruber & Ahissar , 2020 ) . 2 RESULTS . 2.1 TASK AND MODELS . To create a synthetic setting reminiscent of ocular drift , we used images from popular CiFAR datasets ( Krizhevsky et al. , 2009 ) , embedded in a large ( 200x200 pixel ) scene padded by zeros . Sensor position was defined in units of pixels on the scene and its motion was modeled by a stochastic process that is discussed below . The sensor ’ s frames were obtained by cropping a 32x32 pixels window from the scene around the sensor position . Resolution was then reduced to 8x8 using a standard OpenCV ( Bradski , 2000 ) function ( that does not include anti-aliasing filter ) with bi-cubic interpolation ( Fig . 1A ) . A ResNet50 ( He et al. , 2016 ) network pre-trained on Imagenet ( Deng et al. , 2009 ) , which is available as a part of Keras package ( Chollet , 2015 ) , was used as a model of reference . The model was finetuned to one of the CiFAR datasets , reaching accuracy of 96.83 and 82.94 percent for CiFAR-10 and CiFAR 100 , respectively ( Table 1 , Standard resolution ( 32x32 ) ) . In order to feed the 32x32 pix CiFAR images to the network , images were upsampled by factor of 7 to ( 32x7 ) x ( 32x7 ) pix ( i.e . 224x224 pix which are the dimensions of ImageNet images ) . In order to verify the generality of our conclusions , we tested another more compact variant of reference CNN with 3M parameter model . This smaller network that we refer to as Small-net ( Table S7 , S8 ) receives 32x32 pix images as input , therefore , eliminating the need to upsample the CiFAR images before feeding them into the network . This smaller network also simplified the analysis of internal representations as explained below . 2.1.1 TRAINING . We applied a feature learning paradigm ( Hinton et al. , 2015 ) . while using our reference network as teachers for the dynamical recurrent classifier ( DRC ) student . Typical CNNs perform a series of spatial pooling operations . Max pooling layers in the reference CNNs effectively reduce spatial resolution while preserving relevant information about the underlying scene . To develop our DRC , we exploited this spatial pooling line-up . We thus took instances of trained CNNs and replaced their bottom layers with recurrent convolutional networks ( Fig . 1B ) . Specifically , we used a stack of ConvGRU layers ( Ballas et al. , 2015 ; Van Valen et al. , 2016 ) without spatial poolings to replace the original network all the way from the input to the point where the CNN ’ s spatial resolution is reduced by the desired factor ( Table S3 ) . In our case , the resolution was decreased by factor of 4 , therefore the appropriate resolution was achieved after the second max-pooling layer . At this point , considering the 2 DRC systems we have tested , the resolution of the ResNet50-based teacher is ( 8x7 ) x ( 8x7 ) ( the ’ x7 ’ factor is due to the upsampling of the original 32x32 images by a factor of 7 to ( 32x7 ) x ( 32x7 ) ) ; while the Small-net-based teacher resolution is 8x8 . We refer to the bottom recurrent part of the DRC as DRC-front-end ( DRC-FE ) . For the rest of the processing stack we reuse the reference ( teacher ) network architecture ( either ResNet50 or Small-net ) . We refer to this reused part of the DRC as DRC- back-end ( DRC-BE ) ( Fig . 1B ) . We trained the DRC in two steps - first , the DRC-FE was trained to reproduce features of the teacher network . Here we used mean-squared loss between the teacher network and the DRC-FE ( other optimization goals , such as mean absolute loss and cosine similarity , resulted in very similar performance and are not shown ) . Positional data were concatenated with the images time series ; see Appendix B.4 for further details . Next , the DRC-BE was fine-tuned using cross-entropy loss . Our model was mostly implemented in the Keras package ( Chollet , 2015 ) , with the convolutional GRU layer adapted from the project of ( Van Valen et al. , 2016 ) 1 . 2.2 PERFORMANCE . 2.2.1 BASELINE . In order to evaluate the performance improvement which can be attributed to the unique architecture of the DRC , we considered a few baseline solutions . 1Anonymized code is provided as supplementary material . The accuracy of the ResNet50 reference ( teacher ) network when applied to a single low resolution image , was chosen as a simplistic baseline . The performance of this network , shown in Table 1 ( ’ Naive training ’ ) , demonstrates a large degradation of accuracy in both datasets ( See also Table S4 for network ’ s architecture ) . To facilitate fair comparison , we also checked trained such a naive classifier with feature learning as done in DRC . The results here did not change significantly . As a more advanced baseline , we considered an averaged prediction ( AP ) of a feed-forward model over the T sampled frames . Namely , the estimated probability p̂k of a class k is given by p̂k = 1 T ∑T t=1 p̂ t k , where p̂ t k are predictions of the above naive baseline . The situation here is similar to test time data augmentation ( Perez & Wang , 2017 ) with sensor motion being the augmenter . Notably , the AP saturated with the number of timesteps while our full system , as described below kept improving ( Table 1 ) . Here application of a teacher slightly improve the accuracy , and in the case of CiFAR-100 , the improvement is significant ( 1.34 % on average ) . Next , we evaluated models where a convGRU ( resp . GRU ) is connected before ( resp . on the top of ) the last global average pooling layer of ResNet ; we denote it as Resnet+convGRU ( resp . ResNet+GRU ) ( Table S6 ) . At their best , these models achieved accuracy lower by approximately 4 % and 7 % for CiFAR-10 and CiFAR-100 datasets respectively , compared to 5-step DRC w/o positional information . The fact that these models and the AP achieve approximately equal performance indicates that trainable recurrent connectivity in top layers has little benefit over-simplistic integration . This is in contrast to the DRC , where recurrent connectivity is implemented in the low layers . This result is not surprising since convolutional networks tend to develop invariance to small shifts ( Zeiler & Fergus , 2014 ) , such as those that DRC relies on , albeit with important caveats ( Azulay & Weiss , 2018 ) . Finally , we refer to a recent work ( Xi et al. , 2020 ) that uses a generative adversarial network to enhance feature representation in CiFAR-10 task with 8x8 resolution . This solution performs slightly better than DRC without positional information and five timesteps but underperforms versus the same DRC setting with ten steps . Furthermore , no results for CiFAR-100 are available in this work .

Summary: Here, the authors attempt to leverage spatio-temporal computations for object recognition on the standard CIFAR-10 and CIFAR-100 datasets. In short, they use a network with a front-end of recurrent units (ConvGRU) to recognize objects given spatially jittered downsampled images – effectively approximating an active sensor. The network is trained in a student-teacher configuration, where weights in a temporal pooling layer after the recurrent layers are trained to match the weights of a feature layer inResNet50. Next, the network is fine-tuned to increase classification accuracy.  Altogether, the authors are asking if spatio-temporal computations are enough to produce a feature layer similar to a larger network train on full-res images and, in turn, if this feature layer supports object recognition on par with full-res performance of ResNet50. The authors demonstrate that their network is almost as performant as ResNet50 with 4x downsampled images, especially when the down-sampled images are jittered in a spiral formation. They also present analysis demonstrating that the network is in fact performing spatio-temporal calculations.