SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Directional Tuning: Some V1 and MT neurons respond selectively to a specific range of motion directions. To investigate the directional tuning of the units in our model, we utilized in silico neurophysiology to measure the activation patterns of 256 units in response to drifting Gabor and plaid stimuli. A plaid consists of two superimposed drifting Gabors, as illustrated in Fig. 4 (C). Analysis of the data revealed three distinct groups of units based on their partial correlation to Gabor and plaid stimuli: 1) Component cells, which always respond to the direction of a Gabor component; 2) Pattern cells, which respond to the integrated (coherent) motion direction of plaid; and 3) Un-
Figure 4: In Silico Neurophysiological Test. (A, B, C): Analysis of the direction tuning of different groups of neurons for 1D and 2D stimuli; Stage II-N means the result from iteration  $N$ th in the stage II;  $(\mathbf{D},\mathbf{E},\mathbf{F})$ : Analysis of the spectral receptive field and speed tuning of different groups of neurons. The neural distribution in (A) is redrawn from [29]. Neurons' spectral receptive fields in (D) are redrawn from [53]. Two neurons distributions in (F) are separately taken from [54] and [55].
classified cells, which do not show a clear preference for either component or pattern motions, as illustrated in Fig. 4 (C). The distribution of these cell types is not uniform, with component cells being more commonly found in the primary visual cortex (V1), while pattern cells are more often observed in the MT and MST regions[?]. In agreement with this, as illustrated in Fig. 4 (A) (B), our model demonstrates that in the first stage, most units tend to be component cells, whereas the number of pattern and unclassified cells increases in the second stage. In addition, we employed the maximizing activation method [56] to reversely render their cellular preferences, showing that the second-stage unclassified units respond to more complicated motion patterns consisting of both central and background motions, as presented at the bottom of Fig. 4 (B). This suggests that the classical classification of motion neurons into component and pattern cells might be insufficient to characterize the motion integration properties of these neurons.
Spectral Receptive Field and Speed Tuning: We test the spectral (spatial-frequency-vs-temporal-frequency) receptive field of the model's units using a combination of drifting Gabors with different spatiotemporal frequencies. Two-dimensional oriented Gaussian contours are used to depict the receptive field of the 256 cells, fitted by minimizing the least square error. The results are shown in Fig 4 (E). Visually, the distribution of the receptive field tilt angle spreads from horizontal/vertical directions to oblique directions (Fig. 4, E). This indicates that the units in the stage II have significant speed tuning compared to the stage I. Speed tuning is a characteristic of higher-order visual motion neurons[28] and is often found in the MT area[57]. This tendency can be seen from the distribution of the partial correlation between the actual receptive field and its speed prediction/independent prediction[55], as demonstrated in Fig. 4 (F). Our two-stages process shows a degree of consistency with the change in mammalian neural distribution from the V1 to MT area.
3.2 Psychophysical Analysis
Fourier motion: In the "missing fundamental illusion"[60], as shown in Fig. 5 (C), when the first spatial harmonic is removed from a square-wave grating with a quarter-cycle shift, the perceived motion direction appears reversed. Our model, whose first stage estimates motion from the Fourier
Figure 5: Psychophysical stimulus tests compare human perception and model response. (A, B): Spatial integration of 1D and 2D motions. In the stimulus panels, grey arrows indicate the direction perceived by humans, while red arrows indicate the direction of physical motion. The middle panel of (B) : is the heat map representing the neuron's connectivity from a selected local region (circle center) to other regions, with the warmer color indicating a stronger connection. (C): Fourie-motion-based illusion. The size of the shaded area indicates the pixel-wise standard deviation (STD) of the response. (D): Barber pole illusion. The human psychophysical data in (A), (D) were redrawn from the [58] and [59], respectively.
components[61], can predict this reversal. In contrast, computer vision models designed to infer optical flow based on structural correspondence do not exhibit this bias.
Motion Integration: The direction of 1D motion stimuli, such as drifting Gabors, is ambiguous due to the aperture problem. When presented alone, they are perceived to move in orthogonal directions of stripes. When superimposed with other 1D motion components in different orientations, 2D directions consistent with both components are perceived. Pattern motion neurons in MT may contribute to this phenomenon. The second stage of our model can explain this as well, as shown in Fig. 5. Psychophysics also demonstrates motion integration across space. Fig. 5 (A) shows a psychophysical stimulus consisting of drifting Gabors [58]. These Gabors have a variety of local directions and speeds, yet all of them are consistent with one global 2D motion (downward in this case). When viewed as a whole, humans do perceive coherent downward motion. Stage II of our model predicts both the perceived direction and speed of the global Gabor motion.
Fig. 5 (B) compares spatial motion integration between 1D Gabor motion (left) and 2D plaid motion (right). Humans are able to perceive global downward motion only in the former case: The heat maps depict how units with high activity establish long-distance connections to resolve the aperture problem when subjected to Gabor (ambiguous motion) stimuli. In contrast, the plaid stimuli (determined motion) suppress these long-distance connections. In the latter case, local integration of motion signals takes priority over global integration. Once the local ambiguity is resolved, the global integration process is suppressed. Our model can predict such adaptive motion pooling in human visual processing [58]. Furthermore, the barber pole illusion demonstrates how locally ambiguous 1D motion is affected by the shape of the moving area [62]. Specifically, as the height-width ratio of the visual area varies, human perception of direction shifts from oblique to vertical [59] (Fig. 5, D). Our model can predict the shift in perceived direction in stage II, showing its ability to integrate motion signals with boundary orientations. For more video demonstrations, such as the reverse phi illusion[63], please see the supplementary material.
Comparison of Complex Natural Scenes: The proposed model can effectively handle natural scenes, as demonstrated in Fig. 6 (A). Natural stimuli contain diverse spatiotemporal frequency components, leading to complex activation patterns in Stage I. From the decoded flow field, the motion of stage I is limited to localized areas. For example, local motion cannot be found in untextured road areas. This situation necessitates long-range interactions with the surrounding spatial context, which our recurrent integration process in stage II effectively accomplishes. It is evident from the
Figure 6: Comparison of Performance on Complex Natural Scenes. (A): Our model can estimate a dense natural optical flow with the recurrent integration. (B-D): Similarity of our model estimation to human perceived flow on Sintel dataset. The red circle's size in (D) shows how much closer our model is to the human response than GT.
affinity heat map (right side of subfigure A) that object and background areas are clearly segregated in stage II. This suggests that the integration mechanism based on attention has the potential to combine motion integration and object segmentation into a single framework. These two processes are considered highly relevant in the human visual system [64].
We used psychophysically measured optical flow from the Sintel dataset[16] as a benchmark for naturalistic scene flow perceived by humans. Our model was compared to several optical flow estimation methods used in computer vision, including classical algorithms like Farneback, as well as SOTA DNN-based models. We evaluated officially released DNN models that utilize a wide range of inference structures, such as multi-scale inference[11, 12], spatial recurrent models[14], graph reasoning[65], and vision transformers[15].
As shown in Table 1, we computed both the Pearson correlation coefficients and vector endpoint errors (EPEs) between the model prediction and human response, or ground truth (GT). Additionally, we examined the partial correlation between humans and models while controlling the impact of GT:
$$
\rho_ {\text {m o d e l}} = r _ {\text {r e s p m o d e l} \cdot G T} = \frac {r _ {\text {r e s p m o d e l}} - r _ {\text {r e s p} G T} \cdot r _ {\text {m o d e l} G T}}{\sqrt {1 - r _ {\text {r e s p} G T} ^ {2}} \sqrt {1 - r _ {\text {m o d e l} G T} ^ {2}}}, \tag {7}
$$
where  $r$  is the Pearson correlation. This measure is critical in validating the models' ability to capture the characteristics of the human response, as any model could appear to have a high correlation with the human response by just approximating the GT due to a high correlation between the human response and GT[16].
Quantitatively, the proposed model outperformed all compared models in terms of partial correlation. The RAFT-val in the table is the RAFT framework trained on our dataset as a validation, and indicates that our mixed training set also improves the explanatory power of the human response. Fig. 6 (C) shows that our model significantly improves the partial correlation across iterations in Stage II, indicating that the proposed recurrent motion integration architecture can generate more human-like deviations from GT, which is a trend not present in a similar recurrent network, RAFT. In Fig. 6 (D), one can directly see that our model prediction is more similar to human-perceived flow than the GT flow.
Table 1 also shows the results of three biologically-inspired models. FFV1MT[67] is a model capable of computing dense optical flow using direct decoding of the Simoncelli & Heeger V1-MT mechanism. The other two models are modified versions of MotionNet[20] and DorsalNet[23], recently proposed DNN-based models for the explanation of neural responses to visual motion stimuli. Since the original models are designed to recognize global motion only, we tested general multilayer 3D CNN (a core component of MotionNet and DorsalNet) with residual connections, and a pre-trained DorsalNet with frozen parameters, with a linear flow decoder to compute dense flow, trained on natural dense optical flow datasets. Our model outperforms these biologically-inspired models in predicting human responses. The low performances of FFV1MT model and the modified
Table 1: Model v.s. Human v.s. GT.  $\rho$ : Partial correlation between human & model controlling GT;  $r$ : Pearson correlation coefficient;  $epe$ : vector end-point error;  $uv$ ,  $dir$ ,  $spd$  represent motion components in Cartesian space, direction, and speed, respectively.
Method	ρuv	ρdir	ρspd	v.s. Human	v.s. GT
ruv	rspd	rdir	epe	ruv	rspd	rdir	epe
Farneback[66]	0.27	0.23	0.11	0.41	0.91	0.34	2.02	0.34	0.33	0.92	1.96
FlowNet2.0[11]	0.39	0.26	0.34	0.92	0.90	0.96	0.94	0.95	0.94	0.98	0.47
RAFT[14]	0.20	0.22	0.14	0.92	0.90	0.96	0.93	0.98	0.99	0.99	0.25
RAFT-val	0.43	0.17	0.42	0.92	0.89	0.96	1.01	0.92	0.89	0.98	0.69
AGFlow[65]	0.30	0.16	0.20	0.93	0.90	0.96	0.92	0.98	0.98	0.98	0.27
GMFlow[15]	0.34	0.32	0.17	0.91	0.84	0.96	1.03	0.93	0.90	0.97	0.73
FlowFormer[52]	0.36	0.14	0.32	0.93	0.91	0.95	0.90	0.98	0.97	0.98	0.42
FFV1MT[67]	0.31	0.16	0.31	0.83	0.64	0.92	1.48	0.59	0.84	0.94	1.29
3DCNN	0.27	0.29	0.42	0.83	0.86	0.95	1.31	0.83	0.86	0.96	1.14
DorsalNet[23]	0.17	0.19	-0.10	0.20	-0.08	0.86	2.35	0.20	-0.04	0.86	2.33
Ours-fixed	-0.02	0.12	0.16	0.31	0.23	0.78	2.24	0.35	0.18	0.80	2.29
Ours-Stage I	0.34	0.23	0.35	0.71	0.71	0.92	1.52	0.67	0.67	0.92	1.49
Ours-Stage II	0.57	0.43	0.47	0.91	0.88	0.95	0.98	0.86	0.87	0.95	1.04
DorsalNet also suggest that accurately estimating dense optical flow is a challenging task, requiring specific design considerations to address complex and long-range spatial interactions, large jumps, and boundary effects, the complexities of which are not adequately captured by simple mechanisms.
DNNs have achieved impressive performance in various vision tasks, and their ability to explain the HVS is an active area of research. Recent studies have employed DNNs to model and understand the neural mechanism of visual motion. For instance, Rideaux et al.[19, 20] and Nakamura and Gomi [21] used multilayer feedforward networks, while Storrs et al. [22] used a predictive coding network (PredNet), to model biological visual motion processing, and found similarities to neurophysiological data. De Jong et al.[68] found that the spatiotemporal frequency tuning properties of some units in FlowNet resemble those found in mammalian neurons. DorsalNet[23] uses first-person perspective video stimuli to train a 3D ResNet model to predict self-motion parameters, which helps model recapitulate the neural representation of dorsal visual stream.
With a similar goal in mind, we simplified the biological motion process pipeline and proposed a two-stage architecture that models the complete pathway from images, through neural representations, to the perceptual response. Through end-to-end training using a wide range of datasets, our model generalizes well from simple stimuli to complex natural scenes and partially captures important characteristics of motion-processing neurons, including a change in spatiotemporal tuning from V1 to MT areas. To model the motion integration function, we introduced a novel recurrent process based on the attention mechanism. This process successfully explains a wide range of physiological findings (e.g., a change in the population of component and pattern cells from V1 to MT) and psychophysical findings (e.g., global motion pooling). It also improves the partial correlation with human psychophysical response. The success of the attention mechanism in motion integration could be attributed to its similarity to the human visual grouping mechanism[69] or similar feature grouping/binding that may be accomplished using a top-down attentional selection mechanism[70].
While we show that the attention-based recurrent network is a promising computational model of human visual motion grouping and segmentation, how its complex architecture (where all responses can be influenced by all other responses) is actually implemented in the human brain remains an open question. Another limitation of our current model is that it does not process several important abilities of human motion perception, including non-Fourier (second-order) motion detection [61], and motion integration sensitive to surface layout[71]. Our model does not take into account the benefits of biological processing, such as energy efficiency.
In conclusion, combining classical motion energy and advanced deep learning technology is a promising approach to bridge the gap between human and DNN motion perception systems. Our proposed architecture and recurrent process offer insights into the underlying mechanisms of motion perception and open up new avenues for future research.
This work was supported in part by JST, the establishment of university fellowships towards the creation of science technology innovation, Grant Number JPMJFS2123; in part by the JSPS Grants-in-Aid for Scientific Research (KAKENHI), Grant Numbers JP20H00603 and JP20H05957.
[1] J. J. Gibson, “The perception of the visual world,” 1950.
[2] L. M. Vaina, S. A. Beardsley, and S. K. Rushton, Optic flow and beyond, vol. 324. Springer Science & Business Media, 2004.
[3] D. H. Hubel and T. N. Wiesel, "Receptive fields of single neurones in the cat's striate cortex," The Journal of physiology, vol. 148, no. 3, p. 574, 1959.
[4] D. H. Hubel and T. N. Wiesel, "Receptive fields and functional architecture in two nonstriate visual areas (18 and 19) of the cat," Journal of neurophysiology, vol. 28, no. 2, pp. 229-289, 1965.
[5] C. D. Salzman, C. M. Murasugi, K. H. Britten, and W. T. Newsome, "Microstimulation in visual area mt: effects on direction discrimination performance," Journal of Neuroscience, vol. 12, no. 6, pp. 2331-2355, 1992.
[6] S. Celebrini and W. T. Newsome, "Microstimulation of extrastriate area mst influences performance on a direction discrimination task," Journal of Neurophysiology, vol. 73, no. 2, pp. 437-448, 1995.
[7] C. C. Pack and R. T. Born, “Temporal dynamics of a neural solution to the aperture problem in visual area mt of macaque brain,” Nature, vol. 409, no. 6823, pp. 1040–1042, 2001.
[8] E. P. Simoncelli and D. J. Heeger, “A model of neuronal responses in visual area mt,” Vision research, vol. 38, no. 5, pp. 743–761, 1998.
[9] Y. Weiss, E. P. Simoncelli, and E. H. Adelson, "Motion illusions as optimal percepts," Nature neuroscience, vol. 5, no. 6, pp. 598-604, 2002.
[10] C. Pack and R. Born, "Cortical mechanisms for the integration of visual motion," 2008.
[11] A. Dosovitskiy, P. Fischer, E. Ilg, P. Hausser, C. Hazirbas, V. Golkov, P. Van Der Smagt, D. Cremers, and T. Brox, "Flownet: Learning optical flow with convolutional networks," in Proceedings of the IEEE international conference on computer vision, pp. 2758-2766, 2015.
[12] E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox, "Flownet 2.0: Evolution of optical flow estimation with deep networks," in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2462-2470, 2017.
[13] D. Sun, X. Yang, M.-Y. Liu, and J. Kautz, “Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8934–8943, 2018.
[14] Z. Teed and J. Deng, “Raft: Recurrent all-pairs field transforms for optical flow,” in European conference on computer vision, pp. 402–419, Springer, 2020.
[15] H. Xu, J. Zhang, J. Cai, H. Rezatofighi, and D. Tao, "Gmflow: Learning optical flow via global matching," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8121-8130, 2022.
[16] Y.-H. Yang, T. Fukiage, Z. Sun, and S. Nishida, "Psychophysical measurement of perceived motion flow of naturalistic scenes," iScience in press, 2023.
[17] D. L. Yamins and J. J. DiCarlo, "Using goal-driven deep learning models to understand sensory cortex," Nature neuroscience, vol. 19, no. 3, pp. 356-365, 2016.
[55] N. J. Priebe, C. R. Cassanello, and S. G. Lisberger, “The neural representation of speed in macaque area mt/v5,” Journal of Neuroscience, vol. 23, no. 13, pp. 5650–5661, 2003.
[56] J. Yosinski, J. Clune, A. Nguyen, T. Fuchs, and H. Lipson, "Understanding neural networks through deep visualization," arXiv preprint arXiv:1506.06579, 2015.
[57] N. J. Priebe, C. R. Cassanello, and S. G. Lisberger, “The neural representation of speed in macaque area mt/v5,” Journal of Neuroscience, vol. 23, no. 13, pp. 5650–5661, 2003.
[58] K. Amano, M. Edwards, D. R. Badcock, and S. Nishida, “Adaptive pooling of visual motion signals by the human visual system revealed with a novel multi-element stimulus,” Journal of Vision, vol. 9, no. 3, pp. 4–4, 2009.
[59] N. Fisher and J. M. Zanker, "The directional tuning of the barber-pole illusion," Perception, vol. 30, no. 11, pp. 1321-1336, 2001.
[60] R. O. Brown and S. He, "Visual motion of missing-fundamental patterns: motion energy versus feature correspondence," Vision Research, vol. 40, no. 16, pp. 2135-2147, 2000.
[61] C. Chubb and G. Sperling, "Drift-balanced random stimuli: a general basis for studying non-fourier motion perception," JOSA A, vol. 5, no. 11, pp. 1986-2007, 1988.
[62] P. Sun, C. Chubb, and G. Sperling, “Two mechanisms that determine the barber-pole illusion,” Vision research, vol. 111, pp. 43–54, 2015.
[63] S. M. Anstis and B. J. Rogers, "Illusory continuous motion from oscillating positive-negative patterns: Implications for motion perception," Perception, vol. 15, no. 5, pp. 627-640, 1986.
[64] T. Handa and A. Mikami, “Neuronal correlates of motion-defined shape perception in primate dorsal and ventral streams,” European journal of Neuroscience, vol. 48, no. 10, pp. 3171–3185, 2018.
[65] A. Luo, F. Yang, K. Luo, X. Li, H. Fan, and S. Liu, “Learning optical flow with adaptive graph reasoning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 1890–1898, 2022.
[66] G. Farnebäck, “Two-frame motion estimation based on polynomial expansion,” in Image Analysis: 13th Scandinavian Conference, SCIA 2003 Halmstad, Sweden, June 29-July 2, 2003 Proceedings 13, pp. 363-370, Springer, 2003.
[67] F. Solari, M. Chessa, N. K. Medathati, and P. Kornprobst, "What can we expect from a v1-mt feedforward architecture for optical flow estimation?", Signal Processing: Image Communication, vol. 39, pp. 342-354, 2015.
[68] D. B. de Jong, F. Paredes-Valles, and G. C. de Croon, “How do neural networks estimate optical flow? a neuropsychology-inspired study,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 11, pp. 8290–8305, 2021.
[69] P. Mehrani and J. K. Tsotsos, "Self-attention in vision transformers performs perceptual grouping, not attention," arXiv preprint arXiv:2303.01542, 2023.
[70] J. K. Tsotsos, Y. Liu, J. C. Martinez-Trujillo, M. Pomplun, E. Simine, and K. Zhou, "Attending to visual motion," Computer Vision and Image Understanding, vol. 100, no. 1-2, pp. 3-40, 2005.
[71] J. McDermott, Y. Weiss, and E. H. Adelson, “Beyond junctions: nonlocal form constraints on motion interpretation,” Perception, vol. 30, no. 8, pp. 905–923, 2001.