Visual hyperacuity with moving sensor and recurrent neural computations

1 INTRODUCTION . Biological vision is known to be a dynamical process . Two factors contributing to these dynamics are eye motion and recurrent neuronal connections in the brain . Our eyes move constantly with movements that , kinematically , can be divided into saccades - quick gaze shifts , and drifts - small scanning movements between saccades ( often referred to as “ fixational drift ” ) ( Rucci et al. , 2018 ) . These dynamical aspects of vision are reflected only partially in contemporary computer vision systems . Some works addressed large scale shifts in visual attention resembling saccades ( Mnih et al. , 2014 ) . Others explored properties and benefits of recurrent top down connections ( Nayebi et al. , 2018 ) , reminiscent of top-down processing in biological vision ( Hochstein & Ahissar , 2002 ) . Notably , the dynamics of low-level visual processes , occurring early in the bottom-up visual hierarchy and sensitive to the fixational drift ( Snodderly et al. , 2001 ; Ölveczky et al. , 2003 ; Malevich et al. , 2020 ; Hohl & Lisberger , 2011 ) , remains largely overlooked in models of vision as well as in bio-inspired computer vision systems . In fact , since the seminal studies by Hubel & Wiesel ( 1962 ) , selectivity in primary visual cortex has been traditionally described in terms of static spatial filters ( e.g. , simple and complex spatial fields or Gabors of varying frequency and orientation ) . In convolutional neural networks ( CNNs ) ( Krizhevsky et al. , 2012 ) , which have dominated computer vision over the last decade , features resembling the spatial filters deduced from biological studies emerge spontaneously over the course of training ( Zeiler & Fergus , 2014 ; Lindsey et al. , 2019 ) . In some cases , remarkable correlations were found between spatial neural representations in CNNs and those identified in the biological brain ( Yamins & DiCarlo , 2016 ) . On the other hand , temporal dynamics , and sensitivity to temporal features , characterize visual neurons throughout the visual system , from retinal receptors and ganglion cells to thalamic and cortical neurons ( Berry et al. , 1997 ; Chichilnisky , 2001 ; Lee et al. , 1981 ; Levick et al. , 1972 ; Reinagel & Reid , 2000 ; Shimaoka et al. , 2018 ) . Existing evidence suggests that both eye motion ( Snodderly et al. , 2001 ; Ahissar & Arieli , 2001 ; Ölveczky et al. , 2003 ; Malevich et al. , 2020 ; Gruber et al. , 2021 ; Hohl & Lisberger , 2011 ) and recurrent neuronal connectivity ( Bejjanki et al. , 2011 ; Samonds et al. , 2013 ) contribute to this temporal dynamics . Furthermore , it was found that recurrent connections improve correlates of artificial neural networks to neural activity in visual cortical areas ( Kar et al. , 2019 ; Kubilius et al. , 2019 ; Kietzmann et al. , 2019 ) . One niche where spatio-temporal computation is probably necessary is the perception of tiny objects . It is well known that the acuity of biological vision is not limited by the spatial resolution of retinal photoreceptors ( “ visual hyperacuity ” ; Westheimer ( 2009 ) ; Barlow ( 1979 ) ) . Vernier acuity , for example , is dramatically higher than might be expected from pure spatial acuity derived from the photoreceptor density in the retinal mosaic ( Westheimer , 2009 ) . Whether hyperacuity is obtained via spatial , temporal , or spatio-temporal mechanisms is not yet known ( Rucci et al. , 2018 ) . In any case , it is evident that the visual processing allowing hyperacuity , or perception of any tiny stimulus , should cope with the fixational drift ; if it doesn ’ t , the drift , whose amplitude is at least two orders of magnitude larger than the smallest perceivable spatial offsets , would impair acuity ( Ahissar & Arieli , 2001 ; Rucci et al. , 2018 ; Ratnam et al. , 2017 ) . The same drift motion could potentially improve acuity if spatio-temporal computations are employed . Such computations can be based on the emphasis of high-frequency spatial details ( Rucci et al. , 2007 ) , temporal coding of spatial offsets ( Ahissar & Arieli , 2001 ; 2012 ) , Bayesian inference ( Anderson et al. , 2020 ) , or on any other derivative of the interactions between ocular motion and the external image . Furthermore , it is reasonable to attribute such spatio-temporal computations to early visual areas , which are known to exhibit faster dynamics and shorter integration windows compared to regions upstream in the visual processing chain ( Gauthier et al. , 2012 ) . Indeed , it had been shown that the recurrent neuronal circuitry in early visual areas could enable countering the blurring from retinal motion ( Burak et al. , 2010 ) . Using the information available from over-sampling low-resolution images has an extensive history in computer vision as part of the field of super-resolution ( Milanfar , 2017 ) . Multi-image superresolution ( MISR ) ( Farsiu et al. , 2004 ) , distinguished from single-image super-resolution ( Glasner et al. , 2009 ; Dong et al. , 2015 ) , aims to reconstruct high-resolution images from a set of lowresolution ones ( Arefin et al. , 2020 ; Ge et al. , 2018 ; Bhat et al. , 2021 ; Li et al. , 2017 ) . An adjacent field of research , low-resolution object recognition , investigates algorithms to maximize the performance on a given task ( Xi et al. , 2020 ; Ge et al. , 2018 ) . Both fields use low-resolution images as input but differ in the goal of the training and evaluation . In this paper , we introduce a classifier that exploits spatio-temporal computations in early layers to perceive tiny images . More specifically , we trained a convolutional neural network with recurrent connectivity ( Arefin et al. , 2020 ) introduced to early layers . The network receives a sequence of low-resolution images generated via sensor motion mimicking ocular drift . We used high-resolution images to obtain a set of features that were then used to facilitate learning in a teacher-student framework ( Hinton et al. , 2015 ) . The outcome is a dynamical classifier that suffers from only a small drop in accuracy when tasked with a significant decrease in spatial resolution , a decrease that substantially impairs the accuracy of a comparable static feed-forward classifier . Using a novel generative model , we found that our dynamical classifier developed features that were primarily sensitive to spatial changes , others that were primarily sensitive to temporal changes , and a majority that exhibited sensitivity to mixed spatio-temporal patterns . Finally , when examining the correlations between patterns of motion and accuracy of classification , we observed that curved trajectories are favorable for recognition , consistent with recent findings of the curvature of fixational drift trajectories in humans . ( Intoy & Rucci , 2020 ; Gruber & Ahissar , 2020 ) . 2 RESULTS . 2.1 TASK AND MODELS . To create a synthetic setting reminiscent of ocular drift , we used images from popular CiFAR datasets ( Krizhevsky et al. , 2009 ) , embedded in a large ( 200x200 pixel ) scene padded by zeros . Sensor position was defined in units of pixels on the scene and its motion was modeled by a stochastic process that is discussed below . The sensor ’ s frames were obtained by cropping a 32x32 pixels window from the scene around the sensor position . Resolution was then reduced to 8x8 using a standard OpenCV ( Bradski , 2000 ) function ( that does not include anti-aliasing filter ) with bi-cubic interpolation ( Fig . 1A ) . A ResNet50 ( He et al. , 2016 ) network pre-trained on Imagenet ( Deng et al. , 2009 ) , which is available as a part of Keras package ( Chollet , 2015 ) , was used as a model of reference . The model was finetuned to one of the CiFAR datasets , reaching accuracy of 96.83 and 82.94 percent for CiFAR-10 and CiFAR 100 , respectively ( Table 1 , Standard resolution ( 32x32 ) ) . In order to feed the 32x32 pix CiFAR images to the network , images were upsampled by factor of 7 to ( 32x7 ) x ( 32x7 ) pix ( i.e . 224x224 pix which are the dimensions of ImageNet images ) . In order to verify the generality of our conclusions , we tested another more compact variant of reference CNN with 3M parameter model . This smaller network that we refer to as Small-net ( Table S7 , S8 ) receives 32x32 pix images as input , therefore , eliminating the need to upsample the CiFAR images before feeding them into the network . This smaller network also simplified the analysis of internal representations as explained below . 2.1.1 TRAINING . We applied a feature learning paradigm ( Hinton et al. , 2015 ) . while using our reference network as teachers for the dynamical recurrent classifier ( DRC ) student . Typical CNNs perform a series of spatial pooling operations . Max pooling layers in the reference CNNs effectively reduce spatial resolution while preserving relevant information about the underlying scene . To develop our DRC , we exploited this spatial pooling line-up . We thus took instances of trained CNNs and replaced their bottom layers with recurrent convolutional networks ( Fig . 1B ) . Specifically , we used a stack of ConvGRU layers ( Ballas et al. , 2015 ; Van Valen et al. , 2016 ) without spatial poolings to replace the original network all the way from the input to the point where the CNN ’ s spatial resolution is reduced by the desired factor ( Table S3 ) . In our case , the resolution was decreased by factor of 4 , therefore the appropriate resolution was achieved after the second max-pooling layer . At this point , considering the 2 DRC systems we have tested , the resolution of the ResNet50-based teacher is ( 8x7 ) x ( 8x7 ) ( the ’ x7 ’ factor is due to the upsampling of the original 32x32 images by a factor of 7 to ( 32x7 ) x ( 32x7 ) ) ; while the Small-net-based teacher resolution is 8x8 . We refer to the bottom recurrent part of the DRC as DRC-front-end ( DRC-FE ) . For the rest of the processing stack we reuse the reference ( teacher ) network architecture ( either ResNet50 or Small-net ) . We refer to this reused part of the DRC as DRC- back-end ( DRC-BE ) ( Fig . 1B ) . We trained the DRC in two steps - first , the DRC-FE was trained to reproduce features of the teacher network . Here we used mean-squared loss between the teacher network and the DRC-FE ( other optimization goals , such as mean absolute loss and cosine similarity , resulted in very similar performance and are not shown ) . Positional data were concatenated with the images time series ; see Appendix B.4 for further details . Next , the DRC-BE was fine-tuned using cross-entropy loss . Our model was mostly implemented in the Keras package ( Chollet , 2015 ) , with the convolutional GRU layer adapted from the project of ( Van Valen et al. , 2016 ) 1 . 2.2 PERFORMANCE . 2.2.1 BASELINE . In order to evaluate the performance improvement which can be attributed to the unique architecture of the DRC , we considered a few baseline solutions . 1Anonymized code is provided as supplementary material . The accuracy of the ResNet50 reference ( teacher ) network when applied to a single low resolution image , was chosen as a simplistic baseline . The performance of this network , shown in Table 1 ( ’ Naive training ’ ) , demonstrates a large degradation of accuracy in both datasets ( See also Table S4 for network ’ s architecture ) . To facilitate fair comparison , we also checked trained such a naive classifier with feature learning as done in DRC . The results here did not change significantly . As a more advanced baseline , we considered an averaged prediction ( AP ) of a feed-forward model over the T sampled frames . Namely , the estimated probability p̂k of a class k is given by p̂k = 1 T ∑T t=1 p̂ t k , where p̂ t k are predictions of the above naive baseline . The situation here is similar to test time data augmentation ( Perez & Wang , 2017 ) with sensor motion being the augmenter . Notably , the AP saturated with the number of timesteps while our full system , as described below kept improving ( Table 1 ) . Here application of a teacher slightly improve the accuracy , and in the case of CiFAR-100 , the improvement is significant ( 1.34 % on average ) . Next , we evaluated models where a convGRU ( resp . GRU ) is connected before ( resp . on the top of ) the last global average pooling layer of ResNet ; we denote it as Resnet+convGRU ( resp . ResNet+GRU ) ( Table S6 ) . At their best , these models achieved accuracy lower by approximately 4 % and 7 % for CiFAR-10 and CiFAR-100 datasets respectively , compared to 5-step DRC w/o positional information . The fact that these models and the AP achieve approximately equal performance indicates that trainable recurrent connectivity in top layers has little benefit over-simplistic integration . This is in contrast to the DRC , where recurrent connectivity is implemented in the low layers . This result is not surprising since convolutional networks tend to develop invariance to small shifts ( Zeiler & Fergus , 2014 ) , such as those that DRC relies on , albeit with important caveats ( Azulay & Weiss , 2018 ) . Finally , we refer to a recent work ( Xi et al. , 2020 ) that uses a generative adversarial network to enhance feature representation in CiFAR-10 task with 8x8 resolution . This solution performs slightly better than DRC without positional information and five timesteps but underperforms versus the same DRC setting with ten steps . Furthermore , no results for CiFAR-100 are available in this work .

Summary: This paper takes inspiration from the biological phenomenon of fixation drift, slow, low-amplitude movements during fixation that are believed to result in hyper-resolution in human vision. They hypothesize that this phenomenon can be explained by a model that has a recurrent convolutional front end that integrates over fixation drift, feeding into a well-trained back-end from a conventional model (ResNet 50). They demonstrate that this "Dynamical Recurrent Classifier" (DRC) is capable of restoring performance on 8X8 images to nearly the performance on "high" resolution 32X32 CIFAR images (actually, no one would call 32X32 high resolution!). They analyze the representations learned by the model and show they have strong spatio-temporal features, with some learned features emphasizing spatial features, some emphasizing temporal features, but most combine the two. Finally, they show that using curved trajectories improves performance over more random walks, which can potentially explain recent results in humans. They suggest this model can be useful in AI applications involving limited resolution but with multiple samples over time.