Vision at A Glance: Interplay between Fine and Coarse Information Processing Pathways
Authors: Zilong Ji
Xiaolong Zou
Tiejun Huang
Si Wu
Venue:  (2021)
Keywords: Fast pathway
Slow pathway
Interplay
Robustness
Visual backward masking
Biological visual systems
Biological inspried model

Abstract: Object recognition is often viewed as a feedforward, bottom-up process in machine learning, but in real neural systems, object recognition is a complicated process which involves the interplay between two signal pathways. One is the parvocellular pathway (P-pathway), which is slow and extracts fine features of objects; the other is the magnocellular pathway (M-pathway), which is fast and extracts coarse features of objects. It has been suggested that the interplay between the two pathways endows the neural system with the capacity of processing visual information rapidly, adaptively, and robustly. However, the underlying computational mechanism remains largely unknown. In this study, we build a two-pathway model to elucidate the computational properties associated with the interactions between two visual pathways. The model consists of two convolution neural networks: one mimics the P-pathway, referred to as FineNet, which is deep, has small-size kernels, and receives detailed visual inputs; the other mimics the M-pathway, referred to as CoarseNet, which is shallow, has large-size kernels, and receives blurred visual inputs. 
The two pathways interact with each other to facilitate information processing. Specifically, we show that CoarseNet can learn from FineNet through imitation to improve its performance considerably, and that through feedback from CoarseNet, the performnace of FineNet is improved and becomes robust to noises. Using visual backward masking as an example, we demonstrate that our model can explain visual cognitive behaviors that involve the interplay between two pathways. We hope that this study will provide insight into understanding visual information processing and inspire the development of new object recognition architectures in machine learning.

example, we Under review as a conference paper at ICLR 2021
VISION AT A GLANCE: INTERPLAY BETWEEN FINE AND COARSE INFORMATION PROCESSING PATHWAYS
Anonymous authors
Paper under double-blind review
ABSTRACT
Object recognition often viewed as feedforward, bottom-up process in ma- chine learning, but in real neural systems, object recognition is a complicated process which involves the interplay between two signal pathways. One the parvocellular pathway (P-pathway), which is slow and extracts fine features of objects; the other is the magnocellular pathway (M-pathway), which fast and extracts coarse features of objects. It has been suggested that the interplay be- tween the two pathways endows the neural system with the capacity of process- ing visual information rapidly, adaptively, and robustly. However, the underlying computational mechanism remains largely unknown. In this study, we build a two-pathway model to elucidate the computational properties associated with the interactions between two visual pathways. The model consists of two convolu- tion neural networks: one mimics the P-pathway, referred as FineNet, which is deep, has small-size kernels, and receives detailed visual inputs; the other mimics the M M-pathway, referred to as CoarseNet, which is shallow, has large-size kernels. and receives blurred visual inputs. The two pathways interact with each other to facilitate information processing. Specifically, we show that CoarseNet can learn from FineNet through imitation to improve its performance considerably, and that through feedback from CoarseNet, the performnace of FineNet is improved and becomes robust to noises. Using visual backward masking demonstrate that our model can explain visual cognitive behaviors that involve the interplay between two pathways. We hope that this study will provide insight into understanding visual information processing and inspire the development of new object recognition architectures in machine learning.
INTRODUCTION
Imagine you are driving a car a highway and suddenly an object appears in your visual field, crossing the road. Your initial reaction to slam on the brakes even before recognizing the object. This highlights core difference between human vision and current machine learning strategies for object recognition. In machine learning, visual object recognition is often viewed as a feedforward, bottom up process, where object features whereas in human vision, we can capture the gist of a visual object at a glance without processing the details of it, crucial ability for us (especially animals) to survive in competitive natural environ- ments. This strategic difference has been demonstrated by a large volume of experimental data. For examples, (Sugase al. (1999) found that neurons in the inferior temporal cortex (IT) of macaque monkeys convey the coarse information of an object much faster than the fine information of it; FMRI and MEG studies on humans showed that the activation of orbitofrontal cortex (OFC) pre- cedes that of the temporal cortex when a blurred object was shown to the subject (Bar et al.) 2006); Liu et al. (0177) further demonstrated that the dorsal pathway extracts the coarse information of an object in less than 100ms after the stimulus onset, and this coarse information guides the subsequent local information processing.
extracted from local to global in hierarchical manner;
Indeed, the Reverse Hierarchy Theory for visual perception has proposed that although the repre- sentation of image features along the ventral pathway goes from local to global, our perception of an object goes inversely from global to local (Hochstein & Ahissar 2002). How does this happen in the brain? Experimental studies have revealed that there exist two anatomically and functionally sepa- rated signal pathways for visual information processing (see Fig 1. One is called the parvocellular

---

Under review as a conference paper at ICLR 2021
MRGC
EVA
LOC
Plane
Eagle
Kite
P-Pathway
Bat
Retina
Butterfly
Feedforward
M-Pathway
Feedback
PRGC
IPS/SC
PFC
Association
Figure 1: Illustration of the two separated pathways for information processing in the visual system. An image of an eagle is processed through two pathways. Upper panel: the P-pathway processes the detailed information of the image. Lower panel: the M-pathway processes the coarse information of the image rapidly, generates predictions about the image (association), and modulates the informa- tion processing of the P-pathway (feedback). MRGC: midget retina ganglion cell. PRGC: parasol retina ganglion cells. EVA: early visual area. LOC: lateral occipital complex. IPS: intraparietal sulcus. SC: superior colliculus. PFC: prefrontal cortex.
pathway (P-pathway), which starts from midget retina ganglion cells (MRGCs), projects to layers 3-6 in the lateral geniculate nucleus (LGN), and then primarily goes downstream along the ventral stream. The other is called the magnocellular pathway (M-pathway), which starts from parasol retina ganglion cells (PRGCs), projects to layers 1-2 of LGN, and then goes along the dorsal stream the subcortical pathway (the superior colliculus and downstream areas). The two pathways have differ- ent neural response characteristics and complementary computational roles. Experimental findings have shown that the P-pathway is sensitive to colors and responds primarily to visual inputs of high spatial frequency; whereas the M-pathway is color blind and responds primarily visual inputs of low spatial frequency (Derrington & Lennie| 1984). It has been suggested that the M M-pathway serves as short-cut extract coarse information images rapidly, while the P-pathway extracts fine fea- tures of images slowly, and the interplay between two pathways endows the neural system with the capacity processing visual information rapidly, adaptively, and robustly (Bar 2003 Wang al.] 2020 Bullier 2001 Liu et al. 2017). For instance, by extracting the coarse information of an im- age, the M-pathway can generate predictions about what are expected the visual field, and this knowledge subsequently modulate the fine information processing in the P-pathway (Fig. D
Although the existence of separated P- and M- pathways is well known in the neuroscience field, exactly how they cooperate with each other to facilitate information processing remains poorly un- derstood. In this study, we build up a two-pathway model to elucidate the computational properties associated with the interplay between two pathways (Fig. 2 We convolution neural networks (CNNs) as the building blocks, since recent studies have revealed that CNNs are effective to model the neuronal response variability along the visual pathway (Yamins al., 2013 |0 Kriegeskorte 2015). Specifically, we model the P-pathway using a relatively deep CNN, which has small-size kernels and receives detailed visual inputs, referred atively shallow CNN, which has large-size kernels and receives blurred visual inputs, referred to as CoarseNet hereafter. Based on the proposed model, we investigate several computational issues associated with the interplay between two pathways, including how CoarseNet learns from FineNet imitation, and how FineNet benefits from CoarseNet via feedback leverage its performance. We also use the two-pathway model to reproduce the backward masking phenomenon observed in human psychophysic experiments.
FineNet hereafter. The M- M-pathway is modeled by rel-
2 THE TWO-PATHWAY MODEL
The structure of our two-pathway model illustrated in Fig. where FineNet and CoarseNet mimic the P- and M- pathways, respectively. Notably, FineNet is deeper than CoarseNet, reflecting that the P-pathway goes through more feature analyzing relays (e.g.. V1-V2-V4-IT along the ventral pathway) than the M-pathway. FineNet also has smaller convolutional kernels than CoarseNet, reflecting that MRGCs in the retina have much smaller receptive fields than PRGCs. Furthermore, consider that FineNet receives detailed and colourful visual inputs, reflecting that MRGCs have small receptive fields and are color sensitive; while CoarseNet receives blurred and gray inputs, reflecting that PRGCs have large receptive fields and are color blind.
2

---

Under review as a conference paper at ICLR 2021
Fine
M Rx)
(B)
ols loss
O
8 ("(00"))
(valucs)
(prediction) CG3
(simllarit )ا, (2)
Imitation loes
loss
(query)
NoarseNat (1,0%)
momory buffer
Figure 2: Illustration of the two-pathway model. (A) The architecture of the model. The blue and or- ange blocks represent the feedforward convolutional layers in FineNet and CoarseNet, respectively. The purple one represents the feedback convolution block in FineNet (for details, see Sec. [.1 In inference, CoarseNet extracts coarse features (.) from blurred image é which serves as a cue to predict the fine features (()) of the image via association. The associated result is then combined with the deep representations h" (x, æ) to form a feedback signal O(x, æ), and the latter modulates an early layer of FineNet. In training, FineNet is optimized by minimizing the classifica- tion loss, and CoarseNet by minimizing both classification and imitation losses. (B) Illustration of
static memory association (SMA). A query of the coarse features g())) of the input & is associated with a weighted summation of the fine features stored in the memory buffer, where the weighting coefficient (+)) is the similarity between the coarse features and the key vector u.k.
In the model, we consider that the two pathways interact with each other in three forms: 1) Imitation learning. Since CoarseNet has shallow structure and receives blurred inputs, it is hard to train CoarseNet well for object recognition directly. Hence we consider that CoarseNet learns the feature representations of FineNet via an imitation process. Later we will argue that this has an important biological implication (Sec. 3.2). 2) Association. It supposed that the M-pathway generates predictions about what might be in the visual scene, which guides the information processing in the P-pathway. We model this by considering that CoarseNet predicts the representation of FineNet through a memory association process. 3) Feedback. It is known that coarse information can serve as a cognitive bias guiding the extraction of fine information the associated prediction back to earlier layer of FineNet to enhance the fine feature extraction. The details of the two- ww--patwway model are introduced below.
images. We model this by feeding
2.1 THE INFERENCE PROCESS OF THE MODEL
Denote the input to FineNet as and the input to CoarseNet as a. & is obtained by either filtering x with 2D Gaussian filter or binarizing æ. Denote the output of CoarseNet to be p ( _ [ (æ:; 0²): w where (: ()) and (:;: represent, respectively, the fea- ture extractor and the linear classifier of CoarseNet, and (0C, wCC the trainable parameters. The output of FineNet is similarly denoted as pf (a) f {g* xx, O x);; 0f] : }. where the fea- ture extractor of (1: OF) has extra input component O a:: 3)), representing the feedback signal.
To generate the feedback signal O æ)) in FineNet, we consider a memory association process. Two types of associations are exploited in this work, static memory association (SMA) and dynamic memory association (DMA). They have the similar effect of using coarse features g Oc) as cue to predict fine features. SMA is simpler, but we also consider DMA, it introduces temporal dynamics into our two-pathway model necessary for reproducing the backward masking experiment (see Sec. 3.6). For clearance, we only introduce SMA here (see Fig. 2B). and DMA is described in Appendix IF Specifically, we implement SMA with the cache memory model (Orhan 2018)., which performs a key-value association. The model stores a pair of a key matrix u. E RaxA and a value matrix v E Rex in the memory buffer, with K the number of memory items and d, e the dimensions of the key and value vectors, respectively. The columns uy and os represent, respectively, the normalized gf (0x;; 0C) of CoarseNet and the flattened feature vector ht" (æx, &x) of the last convolution layer of FineNet. When a specific query vector g of CoarseNet is presented, first caleulate its similarities with all key vectors stored in the memory buffer, which are given by Ae(t) = exp [3g[ (.) uz]] for k = 1....A. K, with 3 controlling the sharpness of similarity. After that, calculate the associated result, i.e., the predicted fine features, which is
3

---

Under review as a conference paper at ICLR 2021
given by E vx44(3)/ E (+) The inference of FineNet forms a continuous loop so that the feedback signal is updated iteratively (see Fig. AA). At time step t, the feedback signal in FineNet is calculated by O, x) C(Ø) h (x. .) Notably, the first step t 1. only the associated result from CoarseNet is available, which gives (át x) = ( This reflects the fact that the M-pathway is much faster than the P-pathway, which generates the first feedback signal without interacting with high visual areas in the P-pathway.
In summary, the inference of the model involves interaction between two pathways: in response to an image, CoarseNet first generates its output and meanwhile predicts the fine features of FineNet through association; the predicted result is then combined with the deep representations of FineNet to form feedback signal, which modulates the shallow layer of FineNet for feature extraction; this feedback loop can go on iteratively to continuously leverage the performance of FineNet.
2.2 THE TRAINING OF THE MODEL
During training, FineNet and CoarseNet are optimized jointly. To get the network output for an input, we run the feedback loop iteratively in FineNet for steps (T = used in this study). FineNet is optimized through minimizing the cross-entropy loss, which is given by
LF
Voo In In (x:).
where pf is the jth element of pf, i.e., the likelihood of the jth class, and Vi,j is the jth element of the one-hot label y; for the image V;, which for the correct class and 0 otherwise. The summation runs over all images N and all classes K.
Since CoarseNet receives coarse inputs and has a shallow structure, we optimize it via a combination of classification and imitation losses, which is written
NNC
Lc
yh. Inp (at) (dt)) +
1-a 1 1 - g (x 2
j=1
where the symbol || denotes L2 normal, and a hyper-paramend balancing the cross-entropy loss and the imitation loss.
Since SMA aims to store the long-term correlation (association) between the feature representations of CoarseNet and FineNet, we update its key and value matrices after every two training epochs.
3 INTERPLAY BETWEEN TWo PATHWAYS
3.1 IMPLEMENTATION DETAILS
Based on the proposed model, we carry out simulation experiments to explore the computational properties associated with the interplay between two pathways. Three datasets, Pascalvoc-mask, CIFAR-10, and CIFAR-100 are used (see details Appendix (A). To generate blurred inputs to CoarseNet, either low-pase ww-pass filter using 2D Gaussian filter with std 2 or binarizing using shape mask (see examples in Fig. A and the details described in Appendix A.2). In the experiments, FineNet consists of four convolutional layers (Fig. A each of which comprises x3 convolution. followed by a group normalization, a ReLU nonlinearity. The numbers of convolutional filters in 4 layers are (64, 128, 256, 512]. CoarseNet consists of two convolutional layers with the same composition as in FineNet, except that it comprises 128-filter 7 x 7 convolution kernels in the first layer and 512-filter x convolution kernels in the second layer. Both FineNet and CoarseNet have global pooling layer before the readout layer (for generating g @C) and g (æx ((ă æ); OF), convolutional layer,
respectively). The feedback kernel consists of an upsample layer and an followed by group normalization and sigmoid nonlinearity. It takes O x) the input and output a weighting term to modulate the representations the second convolutional layer of FineNet via element- wise multiplication. The balancing term is 0.4 in Eq. 2 During training, the memory buffer in SMA is updated after every two epochs, and = 100. Both FineNet and CoarseNet share the same training settings: the total number of training epochs is 1 150, SGD with momentum term
4

---

Under review as a conference paper at ICLR 2021
0.9 is used to optimize parameters, and the initial learning rate is 0.05 which is multiplied with 0.1 after 100 and 125 epochs. For the training of dynamic memory association (DMA) implemented by RBM, please refer to Appendix F.I
3.2
IMITATION LEARNING IMPROVES COARSENET
(4)
(G)
(D)
60
54
Accuray
54
with imiiation imitation
$6
Imitaion
With imizaion
80
with imilation
imitation
without Imitation
without imitation
64
Number convokstio channels
Kerel Size
Number of convolution channels
Kemel Size
Figure 3: Imitation learning from FineNet improves the performance of CoarseNet. (A-B): perfor-
mances of CoarseNet trained on low-pass filtered images from CIFAR1O. (A) Performances vs. the number of convolution channels. (B) Performances vs. the size of convolution kernel. (C-D); per- formances of CoarseNet trained on the binarized Pascalvoc-mask.
(C) Performance vs. the number (D) Performance vs. the size of convolution kernel. See Appendix A for the details of the training and testing data.
of convolution channels.
In the two-pathway model, CoarseNet is supposed to have good initial guess of the image, which serves as a cognitive bias to facilitate the performance of FineNet. However, since CoarseNet is shallow, has large convolution kernels, and receives coarse inputs, it is hard to train CoarseNet well independently. Therefore, we consider that CoarseNet learns the feature representations of FineNet via imitation learning. This is an important property, which may have some far-reaching implications to brain functions. We therefore carry out a separate experiment to study the effect of imitation learning. In the experiment, we focus exploring whether CoarseNet can learn from FineNet via imitation, without considering other interactions between two pathways (the details of the experiment are described in Appendix B
The results are presented in Fig. 3 which demonstrates that with imitation, the classification accu- racy of CoarseNet is improved considerably, compared to that without imitation over wide range of parameters. Specifically, with respect to the number of convolution kernels in CoarseNet, the improvement significant when the number of kernels is large (Fig. 3 for low-pass filtered inputs; Fig. BC for binarized inputs); with respect to the size of kernels in CoarseNet, the improvement is also significant (Fig. B3 for low-pass filtered inputs; Fig. B3D for binarized inputs). The fact that the effect of imitation learning also depends the network parameters (Fig. (A indicates that in reality there is a trade-off between having a simple structure for the M M-pathway and the capability of the M- M-pathway learning from the P-pathway.
From the computational point of view, the brain faces a difficulty of "designing" the M-pathway. On one hand, the M-pathway needs to be shallow and process coarse visual inputs in order to generate quick responses (which is important in a dangerous environment): on the other hand, the M-pathway needs to efficiently generate approximated, if not accurate, recognition of an object, serving good initial guess for further processing. However, it is a well-known fact that shallow neural network alone is unable to achieve good object recognition (this has actually motivated the devel- opment of deep neural networks). So, how does the brain resolve this dilemma? Here, our study suggests that the strategy of imitation learning proposed in machine learning (Hinton et al., 2015) may provide solution to this challenge, that is. the shallow M M-pathway learns the representations of the deep P-pathway through imitation to improve its performance. Imitation learning may also be involved in other brain functions, such as for knowledge transfer and memory consolidation between hippocampus and ncocortex (Alvarez & Squire 1994 Sirota et al., 2003). During the acquisition of motor skills, it has been observed that neural activities gradually shift from the prefrontal cortex to the premotor. posteriorparietal, and cerebellar areas (the so-called scaffolding-storas storage proposed by [Petersen et al. 1998)), indicating that imitation learning may occur across cortical regions. Notably, the brain also has resources to implement imitation learning, e.g., the widely observed synchronized oscillations between cortical regions (Buzsáki & Draguhn 20044 can modify neuron connections via
5

---

Under review as a conference paper at ICLR 2021
Hebbian plasticity to support the transfer of neural representations. It will be interesting to explore how imitation learning is realized in real neural systems.
3.3 TWO-PATHWAY PROCESSING IMPROVES ROBUSTNESS TO NOISE
Models
Clean
Gaussian noise
Shot noise
Impulse noise
Adversarial noise
FineNet-only (nta= 0)
S6.9
50.0
57.840.8
59.0+0.4
FineNet-only (n = 1)
88. 28.4
56.6+1.2
64.4
61.4+14
FineNet-only (Rea = 3)
88.020.0
59. +1.4
65.9+1.2
63.0+0.1
two-pathway-FFi
88, =0.2
58.8+0.0
65.8+0.2
62.740.6
61.640.1
IWo-pathway-SFl.
88. 0..
56.3
63.041.4
63.240.9
55.4
Our model
86. 26.7
62.2.0.2
68. 68.040.1
66.5
65.7
The two-pathway model outperforms others on diverse noise perturbations. FineNet-only with a = 0, 1, 3, refer FineNet without feedback connection, with loop of feedback inter- action, and with 3 loops of feedback interaction, respectively. The models of two-pathway-FFFL and wwo-pathway-SEF -FL refer to two different ways of fusing the features of FineNet and CoarseNet. FineNet takes clean RGB images and CoarseNets the grayed, tow-pass filtered images. Adversarial noises are generated by attacking FineNet using the Fast Gradient Sign Method (Goodfellow al.) 2014]. The network performances for Gaussian, shot, and impulse noises are obtained by averag- ing over 5 different noise perturbation levels, and the results for adavesarial noises are obtained by averaging over 8 different noise perturbation levels. Mean and std are obtained by averaging over 4 trials. The noise generation details are described in Appendix A.2 Additional experimental results presented in Appendix E
Table 1:
A deep CNN trained for image classification is known overly rely local textures rather than the global shape of objects (Baker et al.] 2018 Geirhos et al. 200 8a b), which is sensitive to unseen noises. In our model, since CoarseNet processes blurred visual inputs, whereby the local texture information is no longer the main cue supporting object classification, we expect that CoarseNet robust to noise corruptions. Furthermore, through association and feedback, we expect that the robustness of FineNet to noises is also leveraged. We carry out simulations to test this hypothesis.
We compare our model with a single single-passway model formed only by FineNet, referred to as FineNet- only. For fair comparison, we also include feedback loops between higher and lower layers in FineNet-only. We train the models on clean CIFARIO dataset and test them by adding various noise perturbations, including Gaussian, shot, impulse and adversarial noises. Tab. shows that compared to the single-pathway model, although the accuracy of our model on clean data is decreased a little bit, its robustness to noises is improved significantly, confirming that CoarseNet does contribute to improving the noise robustness of the model.
We also test different ways of integrating FineNet and CoarseNet. Precious works have studied the integration between two parallel networks (Hou et al.. 2017] Zhu et al.] 2018 Kim et al.) 2019). However, they considered that the networks share the similar structures and receive the same inputs, which are conceptually different from the brain-inspired two- two-pathway model. We borrow their methods of fusing networks integrate FineNet and CoarseNet (while keeping all other hyper- parameters unchanged). Two fusion strategies, simple fusion learning (SFL) and a feature fusion learningFFFL), are adopted (Kim al. 2019). Both of them consider integrating information only at the top layers of two networks (for details, see Appendix ( As shown in Tab. [ our model still has a much better performance on noise robustness than other methods, indicating that the association and feedback module in our model is suitable for integrating information between two pathways.
Remarkably, compared to other methods, our model does not have increased performances on clas- sifying clean images, rather the performances are often decreased little bit compared to that of well-trained deep neural network. The reliable achievement of our model the improved robustness to noises. This highlights important goal for the brain employing two-pathway processing; the P-pathway learns to recognize objects with high fidelity based on fine features that are only available when images clean; when images are ambiguous, the M- M-pathway relying on coarse features of objects compensates for noises. Certainly, there are other computational advantages associated with the two-pathway processing (see below).

---

Under review as a conference paper at ICLR 2021
3.4 ROUGH-TO-HINE PROCESSING IN THE TWO-PASSWAY MODEL
Models
Clean
Gaussian noise
Shot noise
Impulse noise
FineNet-only (m 1)
63.540.3
28.2+1.3
34.9
28.6
Our model
62.5
30.5
36.6
31.0+0.4
Table 2: CoarseNet facilitates the performance of FineNet. The images are taken from CIFAR- 100, which form 20 super-classes and 100 sub-classes (for details, see Appendix AA. CoarseNet and FineNet perform super- and sub- class classifications, respectively. The experiment settings the same as in Tab.
In the above, we have considered that the two pathways process the same categorical information of images. In reality, the two pathways may process different levels of categorical information of images, and object recognition goes from rough to fine, for instance, CoarseNet may recognize the higher category of object (e.g., animal), and FineNet recognizes the lower category of the object (e.g., cat). In such a case, the result of CoarseNet can still serve as cognitive bias to facilitate the performance of FineNet. We carry out experiments confirm this.
We construct a classification task, in which CoarseNet and FineNet are trained to recognize the super- and sub- classes of images, respectively, and the interaction between two networks remains unchanged as before. The results are presented in Tab. 2 which show that indeed the classifica- tion results on super-classes of images by CoarseNet can serve as cognitive bias improve the performance of FineNet on classifying the sub-classes of the images.
3.5 COMPONENT ANALYSIS OF THE TWO-PATHWAY MODEL
We further analyze the contributions of different elements in the model, and confirm that when any one of four elements of the model (FineNet, CoarseNet, the association module (SAM), and the feedback loop) is missing or changed, the robustness of the model to noise is degraded dramatically (see Fig. Notably, these elements are also the key characteristies of the visual signal pathways.
(A) 70
(B) 70
Accuracy
Accuracy
50
FN
X
X
FN
J
J
X
X
J
SMA
X
X
FBI
V
X X x x
FBF
X
X X
Figure 4: Component analysis of the two-pathway model. FN: FineNet. CN: CoarseNet. SAM: static associative memory. FBf: the long-range feedback from CoarseNet and the higher layer of FineNet to the second layer of FineNet; without means short-range feedback the third layer of FineNet. Red dashed line: the performance of the two-pathway model. (A) Model performances with respect Gaussian, impulse and shot noises. (B) Model performances with respect to adver- sarial noise. Experimental details are the same as in Tab. !
3.6 TWO-PATHWAY PROCESSING ACCOUNTS FOR VISUAL BACKWARD MASKING
Visual backward masking is classic experiment widely used in cognitive psychology to investigate attention, awareness and dyslexia. In the cognitive experiment, a masking stimulus is presented after the target stimulus with brief delay (usually 30-70 ms), which incurs failure of the subject to consciously perceive the target (Fig. (BiityeA)..Breemmeeee channel framework, fast transient channel and slow sustained channel, explain the backward
Breitmeyer & Ganz (1976) proposed psychological two-

---

Under review as a conference paper at ICLR 2021
(A)
Backward Masking
(8) 3d
(D)1007 100
Tarper
80
Pafarmenos
S0A
80
80-
20
20.
STM
75 125150
Time
STA dureian
SOA dureion
Figure 5: The two-pathway model reproduces the visual backward masking phenomenon. (A) A
paradigm of the backward masking experiment, adapted from Macknik & Martinez-Conde 2007 SOA: stimulus onset asynchrony, the time interval between the onsets of target and mask; ISI: inter- stimulus interval, the interval between the termination of target and the onset of mask; STA: stimulus termination asynchrony. the interval between the terminations of target and mask. ISI=0 is used in our simulation. (B) Model performance vs. STA. (C) Model performance vs. SOA. (D) The experimental result, adapted from Tang et (2018).
masking phenomenon. The idea is that since the time delay for masking is too short, the neural representation of the mask, which is extracted rapidly through the transient channel, intercepts the neural representation of the target sustained in the slow channel, and hence disturb the perception. The important factors affecting the masking effect are the target and the mask durations, referred to as stimulus onset asynchrony (SOA) and stimulus termination asynchrony (STA), respectively.
Our two-pathway provides natural computational model to explain the backward masking phe- nomenon. To capture the temporal effect, we consider a dynamical memory association process implemented by restrict Boltzmann machine (RBM), which holds the same idea of using coarse features as a cue predict fine features as in SMA. The visible part of RBM composed of the concatenated features from CoarseNet and FineNet, which are associated with each other through hidden variables (for the details, see Appendix [.T] and Appendix F.2). In the simulation, one it- eration in RBM equals
time step of 10ms. As suggested by the neural data (Sugase et al. 1999 Bar et al. 2006 Liu et al. 2017), information processing in the M-pathway proceeds that in the P-pathway for about 50 ms. Therefore, the coarse features fore their corresponding fine features. Because of this time lag, the coarse information of the mask confounded with the fine information of the target, leading wrong association that interferes the perception. The larger the STA or the shorter the SOA, the stronger interference of the mask. Our model successfully reproduces the backward masking effect as observed in the experiment. As shown in Fig. [5] the classification accuracy of the model increases with SOA, agreeing with the experimental findings Tang al. (2018).
loaded into RBM 50 ms be-
CONCLUSION
In the present study, we have proposed a two-pathway model mimicking visual information pro- cessing in the brain. The model is composed of FineNet and CoarseNet, with the former extracting fine information of visual inputs and the latter extracting coarse information of inputs. CoarseNet processes information rapidly, whose result serves feedback to facilitate the performance of FineNet. Our study reveals several appealing properties associated with the interplay between two pathways, which are: 1) through imitation, CoarseNet can learn from FineNet to improve its perfor- mance: 2) through association and feedback from CoarseNet, the robustness to noise of FineNet is improved significantly; 3) the result of CoarseNet serve cognitive bias to leverage the perfor- mance of FineNet, achieving rough-to-fine information processing: 4) the two-pathway model can explain the visual backward masking phenomenon as observed in the experiment. To our knowl- edge, our work the first one that builds up a network model to mimic the structures of the two biological visual pathways and studies their interaction properties. We hope that this study will give us insights into understanding visual information processing and inspire the development of new object recognition architectures in machine learning.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
David H Ackley, Geoffrey E Hinton, and Terrence J Sejnowski. A learning algorithm for boltzmann machines. Cognitive science, 9()::177-19, 1985.
Pablo Alvarez and Larry Squire. Memory consolidation and the medial temporal lobe: a simple network model. Proceedings of the national academy of sciences, 91(15):7041-7045 1994.
Nicholas Baker, Hongjing Lu, Gennady Erlikhman, and Philip Kellman. Deep convolutional networks do not classify based on global object shape. PLoS computational biology, 14(12): e1006613, 2018.
Moshe Bar. A cortical mechanism for triggering top-down facilitation in visual object recognition. Journal of cognitive neuroscience, 15(4):600-609, 2003.
Moshe Bar, Karim S Kassam, Avniel Singh Ghuman, Jasmine Boshyan, Annette M Schmid, An- ders M Dale, Matti S Hämäläinen, Ksenija Marinkovic, Daniel L Schacter, Bruce R Rosen, et al. Top-down facilitation visual recognition. Proceedings of the national academy of sciences, 103 (2):449-454 2006.
Bruno G Breitmeyer and Leo Ganz. Implications of sustained and transient channels for theories of visual pattern masking, saccadic suppression, and information processing. Psychological review, 83(1):1, 1976.
Jean Bullier. Integrated model of visual processing. Brain research reviews, 36(2-3)996-107, 2001.
György Buzsáki and Andreas Draguhn. Neuronal oscillations cortical networks. science, 304 (5679):1966-1929, 2004.
AM Derrington and P Lennie. Spatial and temporal contrast sensitivities of neurones in lateral geniculate nucleus of macaque. The Journal of physiology, 357(1):219-240, 1984.
Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 111(1)98--13,, 2015.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenct-trained cnns biased towards texture; increasing shape bias im- proves accuracy and robustness. arXiv preprint arXiv:1811.1221 2018a.
Robert Geirhos, Carlos RM Temme, Jonas Rauber, Heiko H Schütt, Matthias Bethge, and Fe- lix A Wichmann. Generalisation in humans and deep neural networks. In Advances in neural information processing systems, pp. 7538-7550, 2018b.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv: 1412.657, 2014.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 1033::1503.053.1 2015.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8)11711-180,, 2002.
Geoffrey E Hinton, Simon Osindero, and Yee- Whye Teh. A fast learning algorithm for deep belief nets. Neural computation, 18(7):1527-1554 2006.
Shaul Hochstein and Merav Ahissar. View from the top: Hierarchies and reverse hierarchies in the visual system. Neuron, 36(5):791-804 2002.
John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8):2554-2558. 1982.
Saihui Hou, Xu Liu, and Zilei Wang. Dualnet: Learn complementary features for image recognition. In Proceedings of the IEEE International Conference Computer Vision, pp. 502-510, 2017.

---

Under review as a conference paper at ICLR 2021
Jangho Kim, Minsung Hyun, Inseop Chung. and Nojun Kwak. Feature fusion for online mutual knowledge distillation. arXiv preprint arXiv: 19tii.99940000558.. 2019.
Nikolaus Kriegeskorte. Deep neural networks: a new framework for modeling biological vision and brain information processing. Annual review of vision science, 1:417-446, 2015.
Ling Liu, Fan Wang, Ke Zhou, Nai Ding, and Huan Luo. Perceptual integration rapidly activates dor- sal visual pathway to guide local processing in early visual areas. Plos Biology, (5(11):e224 :)::))0222366 2017.
Stephen Macknik and Susana Martinez-Coniee The role feedback in visual masking and visual processing. Advances in cognitive psychology, 3(1-2):125, 2007.
Emin Orhan. A simple cache model for image recognition. In Advances in Neural Information Processing Systems, pp. 10107-10116, 2018.
Steven E Petersen, Hanneke Van Mier. Julie A Fiez, and Marcus E Raichle. The effects of practice on the functional anatomy of task performance. Proceedings the National Academy of Sciences, 95(3):853-86)) 1998.
Anton Sirota, Jozsef Csicsvari, Derek Buhl, and György Buzsáki. Communication between neocor- tex and hippocampus during sleep in rodents. Proceedings of the National Academy of Sciences, 100(4):2065-2000 2003.
Yasuko Sugase, Shigeru Yamane, Shoogo Ueno, and Kenji Kawano. Global and fine information coded by single neurons in the temporal visual cortex. Nature, 400(6747):869887.. 1999.
Hanlin Tang, Martin Schrimpf, William Lotter, Charlotte Moerman, Ana Paredes, Josue Ortega Caro, Walter Hardesty, David Cox, and Gabriel Kreiman. Recurrent computations for visual pattern completion. Proceedings of the National Academy of Sciences, 115(35)883558840 2018.
Wenbo Wang, Tiangang Zhou, Yan Zhuo, Lin Chen, and Yan Huang. Subcortical magnocellular visual system facilities object recognition by processing topological property. BioRxiv, 2020.
Daniel L Yamins, Ha Hong, Charles Cadieu, and James DiCarlo. Hicrarchical modular optimiza- tion of convolutional networks achieves representations similar to macaque and human ventral stream. In Advances in neural information processing systems, pp. 3093-3101, 2013.
Xiatian Zhu, Shaogang Gong, et al. Knowledge distillation by on-the-fly native ensemble. In Advances in neural information processing systems, pp. 7517-7527, 2018.
10

---

Under review as a conference paper at ICLR 2021
A DATASETS AND MANIPULATION OF THE INPUT
A.1 THREE DATASETS
We three datasets, Pascalvoc-mask, CIFAR-10 and CIFAR-100 to evaluate model. Pascalvoc-mask and CIFAR-10 10 are used to demonstrate the effect of imitation learning in Fig. CIFAR-10 and CIFAR-100 are used test the noise robustness and the rough-to-fine processing property of our model.
dataset created from the Pascalvoc2012 dataset (Everingham et al. 2015), which contains 20 foreground object classes. The goal of the original Pascalvoc2012 dataset to recognize objects from number of visual object classes in realistic scenes. There two main tasks (classification and detection) and two additional competitions (segmentation and action classification). In the current study, we are only interested in those objects with precise annotated segments. Totally, there are 2913 images with 6866 objects having annotated segments. To create the Pascal-mask dataset, firstly, we extract each object image from the raw image and each object segment from the corresponding "SegmentationOhjeer image set according to the bounding box in- formation (see Fig. 6A). Secondly, we remove object images with low resolution (the number of pix- els in width height is less than 50) large aspect ratio (width/height height/width is more than 3). In this way, the number of remaining objects is 4887. Thirdly, to obtain the masked counterpart of each object, we gray the object segment by setting the pixel values for objects to be 1 and back- grounds to be (see Fig. 6B). All object images are resized to 64 x 64 to fit the input of CoarseNet. The new dataset consists of 4887 object images with masks. We split the dataset into 4512 training and 375 testing images, where the testing set is all from the Pascalvoc2007 testing set. See Tab. for the details of Pascalvoc-mask, mask. The dataset can be found at attttaps://rive.eco google. com/ T
Pascalvoc-mask is
aeroplane bicycle bird boat
bottle
bus
car
cat
chair
COW
104/10 147/15 215/11 126/9
92/6
174/12
206/16
261/17 398/33
204/10
diningtable dog horse 115/14
motorbike
person 1017/100
pottedplant
sheep
sofa
tymonitor 138/12 158/16
train
267/16
176/11 162/18
186/14
188/21
178/14
Table 3: The number of samples in each class of Pascalvoc-mask. Digits in each column mean the training/testing numbers. The number of classes is 20 and the total number of training exam- ples/testing examples is 4512/375.
(A)
(B)
Figure 6: Data examples of the dataset Pascalvoc-mask. (A) left panel: raw image and the corre- sponding objects: right panel: the segment of the raw image and the corresponding object segments. (B) Examples of Pascalvoc-mask. Images in the RGB channel and their masked counterparts.
CIFAR-10 consists of 60000 32 x 32 colour images for 10 classes, with 6000 images per class, and they split into 5000 training and 1000 test images in each class.
CIFAR-100 is a harder version of the CIFAR- 10 dataset, which has 100 classes, with 600 images per class, and they split into 500 training and 100 testing images in each class. The 100 classes in
11

---

Under review as a conference paper at ICLR 2021
the LIFAR-100 are further grouped into 20 superclasses, that each image has pair of sub-class and super-class labels (this information is used to test the rough-to- to-fine processing property).
A.2 MANIPULATING THE INPUT WITH DIFFERENT NOISES
(A)
Data example
Guassian noise
Shot noise
Impulse noise
Adversarial noise
Figure 7: Examples of noise disruptions used in the experiments. (A) Examples of visual inputs used for training FineNet and CoarseNet. From up to down, a raw image to FineNet, the corre- sponding low-pass filtered image (blurred) to CoarseNet, and the corresponding binarized image (mask data) to CoarseNet. (B-E) Different kinds of noise disruptions. (B) Examples of Gaussian noise with std 0.04, 0.3, 0.6, respectively. (C) Examples of shot noise with c 100, 3, spectively. (D) Examples of impulse noise with = 0.07, 0.15, 0.3, respectively. (E) Adversarial noise. Up: the adversarial noise of the example image (A)-up, obtained by the Fast Gradient Sign Method (Goodfellow et al., (014)); Middle and down: the adversarial examples with the noise levels of 0.1 and 0.5, respectively.
Evaluating the model performances under different kinds of noise disruption is main task in the current study. Here we describe the details of manipulating inputs with various forms of noise. Four types of noises used, Gaussian, shot, impulse, and adversarial noises. Please Fig. for details.
We obtain the performances of models Gaussion noise, shot noise, and impulse noise dataset by averaging over 5 amplitude levels (Fig. 4). For Gaussian noise, the 5 levels correspond to the noise variance std [0.04, 0.06, 0.08, 0.09, 0.10). For shot noise, the 5 levels correspond to the multiplication parameter e = [500, 250, 100, 75, 50]. For impulse noise, the levels correspond to the probability p [0.01, 0.02, 0.03, 0.05, 0.07]. For adversarial noise, we average 9 different levels
with [0.005, 0.01, 0.015, 0.02, 0.025, 0.03, 0.035, 0.04].
B IMPLEMENTATION DETAILS OF IMITATION LEARNING
In Sec. 3.2 we illustrate the effect of imitation learning CoarseNet. Without loss of generality, FineNet and CoarseNet both adopt simpler structures than that used in the noise robustness task. FineNet used this task consists of three stacked layers, each of which comprises 128-filter 3 x 3 convolution, followed by batch normalization, a ReLU nonlinearity, and 2 x max-pon max-pooling- CoarseNet has two stacked layers with the same composition as in FineNet, except that comprises 64-filter 11 x 11 convolution in the first layer and 128-filter 9 x convolution in the second layer. The balancing term a 0.4 is used when training CoarseNet. Both FineNet and CoarseNet have fully-connected layer of 1000 units before the readout layer. Except for normalizing with the channel-wil mean and standard deviation of the whole dataset, no other pre- pre-processing strategies adopted. All other settings are the same described in Sec. 3.1
12

---

Under review as a conference paper at ICLR 2021
C IMPLEMENTATION DETAILS OF SFL AND FFL
In Sec. 3.3 we mentioned two feature fusion methods SFL and FFL. Here we describe the details. In machine learning society, many feature fusion methods have been proposed to fuse the features from parallel networks. However, these methods focused on fusing features from networks which take the exact same input and share similar model structures, while our two-pathway model takes different inputs and structures. We concatenate the features of top convolutional layers in FineNet and CoarseNet, and then perform the fusion operation through convolutional structure (Kim et al. 2019).
SFL and FFL share the same feature fusion module which consist of dept convolution and pointwise convolution. They are different in training objective functions. In SFL. the sub-network classifiers and the fusion classifier are both trained with cross-entropy losses simultaneously. In FFL, in additional to cross-entropy losses, networks are trained via mutual knowledge distillation losses, which was proved be more effective fusion method (Kim et al. 2019). Note that both FFL and SFL have more trainable parameters than our model (see Tab.
Table 4: The number of trainable parameters in different models. FineNet has trainable pa- rameters than CoarseNet. Networks with FFL and SFL have more trainable parameters than our two-pathway model.
Models
Number of Parameters
FineNet
3, 119, 808
CoarseNet
827, 456. 4, 410, 112
FFL
SFL
4, 410, 112
two-pathway
3,942, 144
D EFFECTS OF 8 IN THE SAM BUFFER
80
CoarseNet
SoarseNet+SAM
Two-pathway
60-
Accuracy
40
20
0
B =1
B 10 B =100 B 1000
Figure 8: Performance of the two-pathway model against the parameter SMA. CoarseNet and the two-pathway model are trained under different 3 settings. When B = 100, both the two- two-pathway model and CoarseNet which reads out via SAM achieve the best performances.
In practice, the value of 3 affects the performance of our two- two-pathway model. We find that a good for CoarseNet, which makes prediction via SAM (Orhan 2018), is always good for the
13

---

Under review as a conference paper at ICLR 2021
corresponding two-pathway model See the three bars when B 100 in Fig. 8). So the value of 3 can be chosen by training a CoarseNet alone, and then we find a good | for CoarseNet through parameter searching.
E ADDITIONAL EXPERIMENTAL RESULTS ON THE NOISE ROBUSTNESS EVALUATION TASK
Models std FFL
Clean
Gaussian noise
Shot noise
Impulse noise
Adversarial noise
88. 20.2
63. 1
69.6g11
66. 26.0
62.8
SFL.
88. 0..1
55.641.0
62.2.4.3
63.440.9
55.2+0.3
wwo-pathway
86.7
64.1+1.0
69.5+1.1
67.3
65.6=0.2
std
88.2
58.8
65.8
62.7
61.61
SFL
88 6
56.3+1.8
63.0+1.4
63.220.9
55.4
two-pathway
86.7
62.2
68.0+0.1
66.5-0.5
65.7=0.5
std =
FFL
88.2 88.240.2
58.3
65.412.6
63.5
60.2
56.5.0.1
63.5
62.0.0.2
56.4+0.4
two-pathway FFL.*
87. 87.5+0.1
60.0+1.5
66.4
65.3=0.1
65.8=0.2
89.4
54.420.2
62.6
62.2
67.7 67.7
Table 5: Performances of different models under different noise levels. CoarseNet in these models takes grayed and low-pass filtered inputs with std = 0, 1, 2, 3. FineNet takes clean inputs. Different from FFL., FFL.* consists of two FineNets. Mean and std are obtained by averaging over trials.
(A) 80
(B)
TwoPassway
FineNee
80
TnnNee
Accuracy
SRL
CoarseNet
Accuracy
CoarseNer
SFL
FFL
FFL
60
60
40
40
3.
Gaussian noise level
Shot noise level
Accuracy (C)
TwoPseeway
(D)
90
WwoPasswaydde
Fine
Accuracy
CoarseNel FinNe::FFL SFL
Finelettpp
noNes-SFL
FFL
60
30
0.01
0.02 0.03
Impulse noise level
Epsilon of adversaral noise
Figure 9: Model performances against noises and adversarial noise perturbations. In (A-C), model performances against noise perturbations. FineNet and CoarseNet are trained independently on the dataset. (D) Model performances against adversarial noise perturbations. and FineNet-FFL means FineNet the two-pathway model, FFL, and SFL, respectively. Noise perturbation details are shown in Appendix A.2 Experimental details are the same as that in Tab.
We also run additional experimentt conditions. As shown Tab. 5 our model outperforms other methods significantly in various noise tasks and with Gaussian filters of diffrent std.
As shown in Fig. | although CoarseNet has much lower accuracy compared to FineNet, it is very robust all kinds of noise disruption and meanwhile its performance decreases much slower than
14

---

Under review as a conference paper at ICLR 2021
other models with the noise disruption level. Then robust CoarseNet can generate robust to SAM to associate relatively clean predictive representation, and the predictive presentation can help to improve the performance of FineNet dramatically.
F IMPLEMENTING BACKWARD MASKING WITH OUR TWO-PATHWAY MODEL
We introduce the details of modeling the backward masking phenomenon using our two-pathway model. First, we introduce RBM used as the association module. Then we elucidate how the two- pathway model works on this task. Last, we introduce some experimental findings the backward masking task.
F.1 RESTRICT BOLTZMANN MACHINE (RBM) AS A DYNAMICAL MEMORY ASSOCIATION MODEL
(A) h
(B)
(1r
(uiky)
data,
t t=
Figure 10: Learning in RBM. (A) Diagram of a RBM with three visible and three hidden units. There are only connections between layers. (B) Illustraiigg the Gibbs sampling process in RBM during training. At time 0, the visible units v are initialized and the hidden units are updated according to h P(h|v). At time i 1, the visible units updated according to P( h)) and the correlations < vahy > are the statistics used for contrastive learning in the RBM. The number of units in the visible layer is 1000, with 500 units for coarse features and 500 units for fine features. For simplicity, we only test our model on the Mnist dataset.
To investigate the effect of different factors on the target visibility, e.g., the task duration (SOA), the mask duration (STA), we modified the similarity- -based association phase into RBM, which intro- duces dynamics in the association phase. RBM is a simplified version of Boltzmann Machine (BM), with the latter being an extension of the Hopfield model with stochastic dynamics (Hinton et al.) 2006). Both BM (Ackley et al. 1985) and the Hopfield model (Hopfield 1982) can be used to cap- ture how memory patterns are stored as stationary states of neural circuits via recurrent connections between neurons. RBM consists of visible and a hidden layers with no within-layed connections. Denote the input to the visible layer as u, activities at the hidden layer as h and the connection matrix between two layers is W. The energy function of RBM is written
E h) -av" - bhh oWh
(3)
where a and b represent the bias vectors in the visible and the hidden layers, respectively. The joint probability of configuration (v, h) is written as
P(v, h) E E(v,b)
(4)
Z
where Z is the partition function given by Z = =IEn E, e E((AA) The probability of specifie is
P(v) = Z
Σ
c E(e,h)
(5)
h
15

---

Under review as a conference paper at ICLR 2021
The appealing property of the bipartite graph structure of RBM is that the conditional distributions P(h)v) and P(w(h) are factorial, i.e..
P(v)h) II P(olh)), P(u, = 1/) = a(a; +
Σ
wiyhe),
(6)
==1
P(h]v)
II P(hloo) P(hy 1/0) = a(b + Σ wybngh
where (x) 1/(1 + + is sigmoid function, with the temperature.
(A) (hidden layer)
(B)
(,00)
C($)
u
v
t=1
t=2
t=k
(keys) (values)
(query)
(prediction)
Figure 11: DMA implemented by RBM. (A) The training phase (see Fig. 10 for the details). (B) The retrivela phase. The coarse probe g (Ø) of visual object fed into the visible layer of the trained RBM and the prediction ( is retrieved through the dynamics of RBM.
To implement the association phase, construct by concatenating the features from both CoarseNet and FineNet, e.g., [oC, "" with vc = ( and v" = g (æ). Given the training examples (features of input images), RBM is optimized through minimizing the negative log-likelihood:
LBBM N Σ log [P(o:)]
1 N
Σ log [Po of)]
(8)
The derivative of the log likelihood with respect to a connection weight is calculated to be,
llog P(v) =< ushy ddata < oely > models
(9)
aW,,
where the first and the second terms in the right hand of the equation denote expectations over the distributions of data and the model, respectively. The first expectation is tractable. For the second expectation, we apply the strategy of contrastive divergence (CD) gradient (Hinton) 2002). which approximates the expectation over the model distribution by sample generated via a number of Gibbs sampling iterations, with the initial state of the visible units being the training sample, as illustrated in Fig 103. More specifically, we use the correlation statistics < u,h; > after k step Gibbs sampling to replace the < vily mmodel to update the connection weights, i.e.,
AW =(<< viyy - oihy >*),
(10)
where e is the learning rate. During the training, e and are sampled from P(h (v) and P(w[h) alternatively. The total number of training epochs is 2000. We use SGD to optimize the RBM with an initial learning rate of 0. which multiplied with 0.1 after 500 and 1000 epochs.
Once the training is finished, can feed partial feature to the visible layer of RBM, and retrieve the complete one. For example, given a partial feature Do at time 0, the hidden representations of RBM is ho P(h 1/2, a(a + We)) and the updated activation in the visible layer is V1 = (b + h,), After & iterations, we can get Us which is a complete feature corresponding do (see Fig. B).
16

---

Under review as a conference paper at ICLR 2021
(A)
(B)
Energy
Ennergy
tinne
CourseNet
CoarseNet
FineNet
FineNet
Target duration (SOA) Masking duration (STA)
in Delay Time
Blank
IGAS AwN.NNAAAOOOA
Suppression Level
Figure 12:
Illustration of the backward masking with our two-pathway model. (A) The effect of small target duration to the network (small SOA). (B) The effect of large target duration to the network (large SOA).
F.2 INPUT TO RBM IN THE TASK
In Sec. 3.6 we showed that the fast representation of the mask processed along CoarseNet will sup- press the slow representation of the target processed along the FineNet, which decreases the target visibility. The dynamics in RBM enables us to investigate which factors, such as the task duration (SOA) and the mask duration (STA), will affect the target visibility. Due slow processing by FineNet, information arrived at RBM from the fine pathway has lag of about 50 ms (biologically) with respect to that from the coarse pathway (see Fig. T2). If we the processing time of each iteration to be 10 ms, then RBM will only receive coarse features v the input at the first iterations. When the target information from FineNet arrives at the visible layer of RBM, will interact with the features from CoarseNet (note that at this moment the features can be the mask or the coarse feature the target, depending on the SOA value). The network evolves for a relatively long time (500 ms) and the final value of the fine part in the visible layer is used for recognition. Fig. 12A (B) shows the effect of a small (large) SOA value to the network performance. Energy is measured in Eq. 33 with the lower the energy, the better the network performance (the better target visibility). Specifically, with small target duration, such as SOA=30, which is smaller than the lag between two pathways, the network state strongly interfered by the mask (red part in Fig. [2] A), and disrupts the target information from Finenet. When SOA is larger than the lag, the network state only partially disrupted by the mask from CoarseNet, leading a better performance. This result has been reported in Fig. 5D.
A
B
100
100
80
Performance
Performance
60
60
Partial
Occluded
40
20
0
25 50
75 100125150
SOA (ms)
0
25 50 75 100125150
SOA (ms)
Figure 13: Performances of human subjects in the backward masking task. The figure is adapted from Tang et al.) (2018). (A) Categorization performance on the partial stimuli on a of 16 exem- plars belonging to categories (chance = 25%, the dashed line). 14 subjects participated in this task. (B) Occluded stimuli was used. Another 15 subjects participated this task. The black lines indi- cate whole objects and the gray lines indicate the partial and occluded objects. Solid lines represents performance with masking and dashed lines without masking.
17

---

Under review as a conference paper at ICLR 2021
F.3 BACKWARD MASKING PSYCHOPHYSICAL EXPERIMENTS
Here we introduce some results of backward masking in psychophysical experiments. They are from Tang et al. (2018), where subjects performed 5-classes recognition task involving categorization of objects that were either partially visible or fully visible. Images were followed by either a gray screen (without masking) or spatially overlapping noise pattern (with masking). SOA varies from 25 to 150 mss in randomly ordered trials (for the detialed setting, please see the original paper). Fig T3 shows that: 1) for whole objects, subjects perfectly recognized the objects, no matter what kind of disruption was used. Even with masking, the performance made no difference with that without masking. We obtaín similar results as in Fig. 5 B&C. Fig. [B, we show that with low mask noise level (can be viewed non-masking). our model achieves high performance. Fi shows that when STA is small (also can be viewed as non-masking), our model achieves a high performance; 2) For disrupted images, subjects' performances were gradually improved when the target duration increased from 25ms to 150 ms (the solid gray lines in both panels). This is also reflected in Fig. 5) in our model. Note that [ang et al. (2018) also proposed attractor-based recurrent model to explain their experiment findings. They proposed that local recurrent connections could explain the backward masking phenomenon, which is different from our two-pathway model.
18


Reviews:
[{"review_id": "lO552gAVDrY", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a two-pathway neural network to mimic the interplay between the parvocellular (slow and fine-grained) and magnocellular (fast and course) pathways in neural systems. The two pathways are named as FineNet and CourseNet. During inference, the FineNet received recurrent feedback signals from the CoarseNet via an attention layer and memory. During training, cross-entropy loss are used for both pathways, and an \"imitation\" loss is used to encourage the CoarseNet pathway to mimic the FineNet \n\nFirst, a coarse-fine two-pathway network is not novel, as coarse-fine or multi-scale pathways is a well-known design used in computer vision applications. Using an attention module for the coarse-to-fine interaction recurrently might be new, but the paper does not show if it could outperform simpler interactions.\n\nSecond, the \"imitation\" loss is essentially model distillation. It is well-known that learn a weaker network by distilling a stronger network can result in stronger results for the weaker network. \n\nThird, the accuracy on Cifar-10/100 are low, far away from the well-established SOTA. Though the paper provides a few ablations, observations made on a too9 weak model are not conclusive. \n\nFinally, the paper might aim to be explanatory, but it falls short in clarity. The model description in Sec 2.1 is elusive and not sufficiently detailed. The choice of the number of feedback steps is ad hoc. The outreach to backward masking oversold a computational analogy as a neural computational model while providing a vague explanation of the background and results of the experiments. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "kEony2UNTPQ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\nThis paper presents a new type of brain-inspired dual-pathway DNN model where the coarse (faster, less accurate) and fine (slower, more accurate) visual pathways augment each other during training and inference (via imitation and feedback) to boost the network's robustness to various noises.\n\n**Pros**\n(1) The topic is interesting and important as the P and M pathways are both crucial elements of the efficient and robust human visual system but their computational models are much less studied.\n(2) The proposed model is new and overall simple to implement.\n\n**Cons**\n(1) The model's design decisions are arbitrary and poorly justified.\n(Imitation learning) It's unclear why CoarseNet activations must mimic FineNet activations, since only FineNet activations are ultimately used for inference (not CoarseNet activations, which are further transformed and used by the FineNet). To justify the necessity of imitation learning, the authors should present full FineNet+CoarseNet results (not just Fig 3) using all 3 losses versus using only FineNet's classification loss.\n(Binary masks) The setting that the CoarseNet can use clean binary masks of objects as inputs doesn't make sense at all in my opinion. Segmentation itself is a complex task that often requires high-level vision so it's unclear to me why the authors assume that PRGCs can provide such information.\n(SMA) It's unclear why the SMA's u and v are implemented as memory buffers (that are updated per 2 epochs, which seem arbitrary and not biologically plausible either) following [Orhan, 2018], instead of end-to-end learned parameters (that are consistently updated every iteration with the rest of the network using backprop gradients). More generally, the authors should present solid reasons why association should also produce results that mimic FineNet activations.\n(DMA and feedback) It's unclear why an RBM is required as an additional component to introduce dynamics, since extending the feedback loop dynamically (i.e. more loops) should also be able to achieve similar results using existing components (which seems more biologically plausible in terms of resources).\n\n(2) The experimental results are weak and incomplete, and comparisons against related work are missing.\n(Noise robustness) As the key benefit claimed in this paper, the noise robustness of the proposed model is however weak (still roughly 20% accuracy drops for all types of noises) and only better than simplified or similar variants of the model. In addition, using only FGSM (targeting the FineNet) to generate adversarial noises doesn't fully test the robustness of the model, since more recent techniques [1, 2] can easily generate smooth adversarial examples that will likely severely affect the CoarseNet (unlike FGSM).\n(Related work) Although the authors argued that existing models are conceptually different, it doesn't mean that architecturally similar models [3, Hou et al.], SOTA in adversarial defense [4, 5], and most importantly other brain-inspired models, targeting robustness [6, 7] or not [8, Tang et al.], shouldn't be compared against to properly prove the value of this work. Also, as the field has started to more directly use neural data to guide better network design [8], it's unclear why the authors seem to have completely omitted this approach.\n(Rough-to-fine processing) The value of this approach is unclear since the accuracies of training both networks using the subclass labels (CIFAR-100) are missing.\n(Backward masking) Visual results alone (Fig 5 and 12) don't properly support the claim that this model \"can explain visual cognitive behaviors that involve the interplay between two pathways\". Please consider providing more detailed statistical analyses (e.g. R^2) if the authors want to make this claim.\n\n(3) The clarity needs improvement.\nThe clarity of this paper is substandard as many key details are ambiguous or completely missing. For example, how does the SMA memory buffer store features? Random sampling over the training set? Is the model also trained on noisy data? How does a CN+SMA model (Fig 4) even work? What exactly are the backward masking stimuli used in the experiments (frame by frame)?\n\n**Recommendation**\nI recommend rejection of the paper given the following two major cons (see details above).\n(1) The model's design decisions are arbitrary and poorly justified.\n(2) The experimental results are weak and incomplete, and comparisons against related work are missing.\n\n**Questions**\nPlease address the cons listed above.\n\n**Additional Feedback**\n(1) Although using an L2 loss for imitation learning is straightforward mathematically, the authors' arguments regarding how the brain may implement imitation learning aren't very convincing (Sec 3.2). For example, is there direct evidence of synchronized oscillation supporting the transfer of neural representations and thus imitation learning?\n(2) Speed seems to be a major potential benefit of the proposed model, which however was not clearly discussed or benchmarked. Please consider adding speed comparisons against SOTA networks in terms of inference speed.\n\n**References**\n[1] Low Frequency Adversarial Perturbation, UAI, 2019\n[2] SmoothFool: An Efficient Framework for Computing Smooth Adversarial Perturbations, WACV, 2020\n[3] U-Net: Convolutional Networks for Biomedical Image Segmentation, MICCAI, 2015\n[4] Feature Denoising for Improving Adversarial Robustness, CVPR, 2019\n[5] Adversarial Defense by Restricting the Hidden Space of Deep Neural Networks, ICCV, 2019\n[6] Brain-inspired Robust Vision using Convolutional Neural Networks with Feedback, NuerIPS-W, 2019\n[7] Biologically Inspired Mechanisms for Adversarial Robustness, arXiv, 2020\n[8] Brain-Like Object Recognition with High-Performing Shallow Recurrent ANNs, NeurIPS, 2019", "extra_scores": {}, "extra_text": {}}, {"review_id": "RjVKKrG4KU-", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a dual-path CNN architecture with complementary roles (FineNet and CoarseNet) which is inspired by parvocellular and magnocellular pathways in the primate brain. The CoarseNet receives blurred inputs and has large kernels while FineNet received high-resolution input, is deep and has small kernels. It is shown that this architecture improves the robustness in object recognition performance over single-pathway architecture and could replicate the behavioral responses in humans during a classic psychological experiment (backward masking). \n\nThe proposed architecture is novel and the results support the main claims regarding the improvements in robust object recognition behavior and replication of the psychological experiment. My main criticism of this work is the lack of comparison with alternative models in the reported results. For example results in table 1 and 2 are only compared to alternative settings of the same model but no other studies or SOTA. \n\nThe authors could further improve the manuscript by considering the following comments. \n\n* what is the neuroscience evidence of relative shallowness of M-pathway?\n* The processes controlling the memory buffer has not been explained at all. \n* The exact images used to produce each of the results are not clearly explained. \n* In Table-1, it is unclear what \"our model” is and how is it different from FFL and SFL?\n* In sectino 3.3, in the FineNet-only models, it is unclear how the feedback loop is resolved in the absence of CoarseNet.\n* In section 3.3 it is stated that “this highlights an important goal for the brain employing two-pathway processing”. While this is a plausible hypothesis about the role of these two pathways, we don't know if that is indeed the evolutionary goal of having two pathways. The dorsal pathway does much more than fast recognition of objects e.g. motion perception. I suggest the authors either remove or revise this statement. \n* In tables 1-2, are the reported results the output of FineNet? It is not currently clear if that is the case \n* page 8: restrict Boltzmann machine —> Restricted Boltzmann Machine \n* For results in Fig-5, how would alternative models like CORnet [1] perform on this task? \n\n[1] Kubilius, Jonas, et al. \"Brain-like object recognition with high-performing shallow recurrent ANNs.\" Advances in Neural Information Processing Systems. 2019.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores a network that has a parvo (fine, detailed, slow)
and magno (low-res, quick) stream.  The ideas are interesting and the
results intriguing, and one reviewer is in favor of acceptance.
Several reviewers criticized the clarity of the paper. and the lack of
details for, explanations of, and critical evaluation of, the design
decisions.  For example, how do the results depend on certain design
decisions?  I think that with a bit more work, this paper has potential to
be a very impactful paper.  I would encourage the authors to follow the 
detailed suggestions and resubmit the work to a high-impact conference or 
journal.


Decision: Reject