Vision at A Glance: Interplay between Fine and Coarse Information Processing Pathways

1 INTRODUCTION . Imagine you are driving a car on a highway and suddenly an object appears in your visual field , crossing the road . Your initial reaction is to slam on the brakes even before recognizing the object . This highlights a core difference between human vision and current machine learning strategies for object recognition . In machine learning , visual object recognition is often viewed as a feedforward , bottom up process , where object features are extracted from local to global in a hierarchical manner ; whereas in human vision , we can capture the gist of a visual object at a glance without processing the details of it , a crucial ability for us ( especially animals ) to survive in competitive natural environments . This strategic difference has been demonstrated by a large volume of experimental data . For examples , Sugase et al . ( 1999 ) found that neurons in the inferior temporal cortex ( IT ) of macaque monkeys convey the coarse information of an object much faster than the fine information of it ; FMRI and MEG studies on humans showed that the activation of orbitofrontal cortex ( OFC ) precedes that of the temporal cortex when a blurred object was shown to the subject ( Bar et al. , 2006 ) ; Liu et al . ( 2017 ) further demonstrated that the dorsal pathway extracts the coarse information of an object in less than 100ms after the stimulus onset , and this coarse information guides the subsequent local information processing . Indeed , the Reverse Hierarchy Theory for visual perception has proposed that although the representation of image features along the ventral pathway goes from local to global , our perception of an object goes inversely from global to local ( Hochstein & Ahissar , 2002 ) . How does this happen in the brain ? Experimental studies have revealed that there exist two anatomically and functionally separated signal pathways for visual information processing ( see Fig.1 ) . One is called the parvocellular pathway ( P-pathway ) , which starts from midget retina ganglion cells ( MRGCs ) , projects to layers 3-6 in the lateral geniculate nucleus ( LGN ) , and then primarily goes downstream along the ventral stream . The other is called the magnocellular pathway ( M-pathway ) , which starts from parasol retina ganglion cells ( PRGCs ) , projects to layers 1-2 of LGN , and then goes along the dorsal stream or the subcortical pathway ( the superior colliculus and downstream areas ) . The two pathways have different neural response characteristics and complementary computational roles . Experimental findings have shown that the P-pathway is sensitive to colors and responds primarily to visual inputs of high spatial frequency ; whereas the M-pathway is color blind and responds primarily to visual inputs of low spatial frequency ( Derrington & Lennie , 1984 ) . It has been suggested that the M-pathway serves as a short-cut to extract coarse information of images rapidly , while the P-pathway extracts fine features of images slowly , and the interplay between two pathways endows the neural system with the capacity of processing visual information rapidly , adaptively , and robustly ( Bar , 2003 ; Wang et al. , 2020 ; Bullier , 2001 ; Liu et al. , 2017 ) . For instance , by extracting the coarse information of an image , the M-pathway can generate predictions about what are expected in the visual field , and this knowledge subsequently modulate the fine information processing in the P-pathway ( Fig . 1 ) . Although the existence of separated P- and M- pathways is well known in the neuroscience field , exactly how they cooperate with each other to facilitate information processing remains poorly understood . In this study , we build up a two-pathway model to elucidate the computational properties associated with the interplay between two pathways ( Fig . 2 ) . We use convolution neural networks ( CNNs ) as the building blocks , since recent studies have revealed that CNNs are effective to model the neuronal response variability along the visual pathway ( Yamins et al. , 2013 ; Kriegeskorte , 2015 ) . Specifically , we model the P-pathway using a relatively deep CNN , which has small-size kernels and receives detailed visual inputs , referred to as FineNet hereafter . The M-pathway is modeled by a relatively shallow CNN , which has large-size kernels and receives blurred visual inputs , referred to as CoarseNet hereafter . Based on the proposed model , we investigate several computational issues associated with the interplay between two pathways , including how CoarseNet learns from FineNet via imitation , and how FineNet benefits from CoarseNet via feedback to leverage its performance . We also use the two-pathway model to reproduce the backward masking phenomenon observed in human psychophysic experiments . 2 THE TWO-PATHWAY MODEL . The structure of our two-pathway model is illustrated in Fig . 2 , where FineNet and CoarseNet mimic the P- and M- pathways , respectively . Notably , FineNet is deeper than CoarseNet , reflecting that the P-pathway goes through more feature analyzing relays ( e.g. , V1-V2-V4-IT along the ventral pathway ) than the M-pathway . FineNet also has smaller convolutional kernels than CoarseNet , reflecting that MRGCs in the retina have much smaller receptive fields than PRGCs . Furthermore , we consider that FineNet receives detailed and colourful visual inputs , reflecting that MRGCs have small receptive fields and are color sensitive ; while CoarseNet receives blurred and gray inputs , reflecting that PRGCs have large receptive fields and are color blind . In the model , we consider that the two pathways interact with each other in three forms : 1 ) Imitation learning . Since CoarseNet has a shallow structure and receives blurred inputs , it is hard to train CoarseNet well for object recognition directly . Hence we consider that CoarseNet learns the feature representations of FineNet via an imitation process . Later we will argue that this has an important biological implication ( Sec . 3.2 ) . 2 ) Association . It is supposed that the M-pathway generates predictions about what might be in the visual scene , which guides the information processing in the P-pathway . We model this by considering that CoarseNet predicts the representation of FineNet through a memory association process . 3 ) Feedback . It is known that coarse information can serve as a cognitive bias guiding the extraction of fine information of images . We model this by feeding the associated prediction back to an earlier layer of FineNet to enhance the fine feature extraction . The details of the two-pathway model are introduced below . 2.1 THE INFERENCE PROCESS OF THE MODEL . Denote the input to FineNet as x and the input to CoarseNet as x̂ . x̂ is obtained by either filtering x with a 2D Gaussian filter or binarizing x. Denote the output of CoarseNet to be pC ( x̂ ) = fC [ gC ( x̂ ; θC ) ; wC ] , where gC ( · ; θC ) and fC ( · ; wC ) represent , respectively , the fea- ture extractor and the linear classifier of CoarseNet , and { θC , wC } the trainable parameters . The output of FineNet is similarly denoted as pF ( x ) = fF { gF [ x , O ( x̂ , x ) ; θF ] ; wF } , where the fea- ture extractor gF ( · ; θF ) has an extra input componentO ( x̂ ; x ) , representing the feedback signal . To generate the feedback signal O ( x̂ , x ) in FineNet , we consider a memory association process . Two types of associations are exploited in this work , static memory association ( SMA ) and dynamic memory association ( DMA ) . They have the similar effect of using coarse features gC ( · ; θC ) as a cue to predict fine features . SMA is simpler , but we also consider DMA , as it introduces temporal dynamics into our two-pathway model necessary for reproducing the backward masking experiment ( see Sec . 3.6 ) . For clearance , we only introduce SMA here ( see Fig . 2B ) , and DMA is described in Appendix F. Specifically , we implement SMA with the cache memory model ( Orhan , 2018 ) , which performs a key-value association . The model stores a pair of a key matrix u ∈ Rd×K and a value matrix v ∈ Rc×K in the memory buffer , with K the number of memory items and d , c the dimensions of the key and value vectors , respectively . The columns uk and vk represent , respectively , the normalized gC ( x̂k ; θC ) of CoarseNet and the flattened feature vector hF ( xk , x̂k ) of the last convolution layer of FineNet . When a specific query vector gC ( x̂ ) of CoarseNet is presented , we first calculate its similarities with all key vectors stored in the memory buffer , which are given by ∆k ( x̂ ) = exp [ βgC ( x̂ ) > uk ] , for k = 1 , . . .K , with β controlling the sharpness of similarity . After that , we calculate the associated result , i.e. , the predicted fine features , which is given by C ( x̂ ) = ∑ k vk∆k ( x̂ ) / [ ∑ k ∆k ( x̂ ) ] . The inference of FineNet forms a continuous loop so that the feedback signal is updated iteratively ( see Fig . 2A ) . At time step t , the feedback signal in FineNet is calculated by Ot ( x̂ , x ) = C ( x̂ ) + hFt−1 ( x , x̂ ) . Notably , at the first step t = 1 , only the associated result from CoarseNet is available , which gives O1 ( x̂ , x ) = C ( x̂ ) . This reflects the fact that the M-pathway is much faster than the P-pathway , which generates the first feedback signal without interacting with high visual areas in the P-pathway . In summary , the inference of the model involves interaction between two pathways : in response to an image , CoarseNet first generates its output and meanwhile predicts the fine features of FineNet through association ; the predicted result is then combined with the deep representations of FineNet to form a feedback signal , which modulates the shallow layer of FineNet for feature extraction ; this feedback loop can go on iteratively to continuously leverage the performance of FineNet . 2.2 THE TRAINING OF THE MODEL . During training , FineNet and CoarseNet are optimized jointly . To get the network output for an input , we run the feedback loop iteratively in FineNet for T steps ( T = 2 used in this study ) . FineNet is optimized through minimizing the cross-entropy loss , which is given by LF = − 1 N N∑ i=1 K∑ j=1 yi , j lnp F j ( xi ) , ( 1 ) where pFj is the jth element of p F , i.e. , the likelihood of the jth class , and yi , j is the jth element of the one-hot label yi for the image xi , which is 1 for the correct class and 0 otherwise . The summation runs over all images N and all classes K. Since CoarseNet receives coarse inputs and has a shallow structure , we optimize it via a combination of classification and imitation losses , which is written as LC = 1 N N∑ i=1 −α K∑ j=1 yi , j ln p C j ( x̂i ) + 1− α 2 ‖gC ( x̂i ) − gF ( xi , x̂i ) ‖2 , ( 2 ) where the symbol ‖ · ‖ denotes L2 normal , and α is a hyper-parameter balancing the cross-entropy loss and the imitation loss . Since SMA aims to store the long-term correlation ( association ) between the feature representations of CoarseNet and FineNet , we update its key and value matrices after every two training epochs .

Summary: This paper proposed a two-pathway neural network to mimic the interplay between the parvocellular (slow and fine-grained) and magnocellular (fast and course) pathways in neural systems. The two pathways are named as FineNet and CourseNet. During inference, the FineNet received recurrent feedback signals from the CoarseNet via an attention layer and memory. During training, cross-entropy loss are used for both pathways, and an "imitation" loss is used to encourage the CoarseNet pathway to mimic the FineNet