SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Modelling Human Visual Motion Processing with Trainable Motion Energy Sensing and a Self-attention Network
Zitang Sun<sup>1</sup> Yen-Ju Chen<sup>1</sup> Yung-Hao Yang<sup>1</sup> Shin'ya Nishida<sup>12*</sup>
<sup>1</sup> Graduate School of Informatics, Kyoto University
$^{2}$  NTT Communication Science Laboratories, Nippon Telegraph and Telephone Corporation
{sun.zitang.73u, chen.yenju.44z, yang.yunghao.8v}
@st.kyoto-u.ac.jp; nishida.shinya.2x@kyoto-u.ac.jp
Visual motion processing is essential for humans to perceive and interact with dynamic environments. Despite extensive research in cognitive neuroscience, image-computable models that can extract informative motion flow from natural scenes in a manner consistent with human visual processing have yet to be established. Meanwhile, recent advancements in computer vision (CV), propelled by deep learning, have led to significant progress in optical flow estimation, a task closely related to motion perception. Here we propose an image-computable model of human motion perception by bridging the gap between biological and CV models. Specifically, we introduce a novel two-stages approach that combines trainable motion energy sensing with a recurrent self-attention network for adaptive motion integration and segregation. This model architecture aims to capture the computations in V1-MT, the core structure for motion perception in the biological visual system, while providing the ability to derive informative motion flow for a wide range of stimuli, including complex natural scenes. In silico neurophysiology reveals that our model's unit responses are similar to mammalian neural recordings regarding motion pooling and speed tuning. The proposed model can also replicate human responses to a range of stimuli examined in past psychophysical studies. The experimental results on the Sintel benchmark demonstrate that our model predicts human responses better than the ground truth, whereas the state-of-the-art CV models show the opposite. Our study provides a computational architecture consistent with human visual motion processing, although the physiological correspondence may not be exact. $^{1}$
Visual motion perception is essential not only for humans and animals to perceive and interact with the world but also for artificial agents to process various dynamic visual tasks. As such, visual motion estimation has been extensively studied by both biological vision and computer vision research communities. The key issue lies in estimating optical flow, an array of instantaneous image motion vectors [1, 2].
Vision science has revealed that optical flow estimation in the human visual system (HVS) is primarily served by a pathway that includes the primary visual cortex (V1 [3, 4]) and middle temporal (MT [5, 6]) area. A significant portion of neurons in area V1 are sensitive to the direction of local motion, while those in area MT integrate and segregate local motion signals for global flow interpretation. The MT process also helps to overcome the aperture problem[7]. Several computational models of area MT have been proposed [8, 9], but mechanisms that can fully encapsulate a variety
<sup>1</sup>Project Website: Click here
of motion integration abilities of MT neurons and/or human perception have yet to be developed [10]. Furthermore, previous studies tested the models using only simple artificial stimuli. It remains challenging to create a human-like optical flow extraction mechanism that can derive informative motion flow for a wide range of stimuli including complex natural videos.
On the other hand, optical flow estimation has recently made remarkable progress in the field of computer vision. FlowNet[11] pioneered the use of fully convolutional neural networks for dense optical flow estimation, and various approaches based on deep neural networks (DNNs) have emerged subsequently[12-15]. Owing to the powerful representation ability of DNNs, these models outperform humans in estimating the ground-truth optical flow of natural scenes [16]. Consequently, one might expect DNN-based optical flow estimation algorithms to become promising models of human visual motion processing, similar to how ImageNet-trained DNN models provide good computational models of human object recognition[17]. However, since computer vision optical flow models are designed solely to find local image correspondences between pairs of frames on the image coordinates [11], they cannot explain systematic or adaptive deviations of human perceived motion from local ground truth (GT) [16]. Moreover, existing DNN models often exhibit instability when dealing with non-textured stimuli commonly used in vision science [18].
In this study, we leveraged the flexibility of DNNs to construct an image-computational model of human motion processing. While recent studies successfully used DNNs to elaborate the understanding of the neural mechanism of visual motion processing[19-23], we aimed to make a model that can explain a broader range of physiological and psychophysical phenomena, including those whose neural mechanisms are not yet clear. From an engineering standpoint, we aimed to make a human-aligned optic flow algorithm while maintaining a flow estimation capability comparable to the state-of-the-art (SOTA) CV models. Our model extracts dense informative motion flows for a wide range of inputs in a way consistent with physiologically measured neural responses and psychophysically measured human motion perception. It consisted of two stages. The first stage, which mimicked the function of V1, comprised of neurons with multi-scale spatiotemporal filters to extract local motion energy. Unlike previous models of V1 [24, 8, 25], the filter tunings were learnable to fit natural optic flow computation. The second stage, which mimicked the function of MT recurrently integrated local motion signals, and solved the aperture problem. We constructed an undirected fully connected graph from the map of local motion energy, and used the attention mechanism [26, 27] for adaptive global motion integration and segregation.
We evaluated the performance of our model from several aspects. In in-silico neurophysiology, we found that our model's neurons exhibited direction and speed tunings similar to those observed in mammalian physiological recordings in V1 and MT [28, 29]. In simulations of psychophysical findings, our model showed good generalization from simple artificial stimuli (e.g., drifting Gabor) to complex naturalistic scenes. Our model produced human-like responses for several conventional motion stimuli and illusions, including global motion pooling and the barber-pole illusion. Furthermore, the mode's response to natural scenes was closer to that of humans in comparison to other computer vision models. Our two-stages model provides a computational architecture consistent with human visual motion processing, although the physiological correspondence may not be exact. This achievement is not only valuable in terms of neuroscientific understanding of human visual computation but also for the development of human-aligned machine vision that stably recognizes the world as humans do.
2 Molding of two-stages motion perception system
2.1 First-stage: Local Motion Energy Computation
Spatiotemporal separable Gabor filter: Since we aimed at an image-computable model, the input is a sequence of grayscale images  $\mathbf{S}(\mathbf{p},\mathbf{t})$  for all spatial positions  $\mathbf{p} = (x,y)$  within the image domain  $\Omega$  and for all times  $\mathbf{t} > \mathbf{0}$ . The goal of the first-stage neuron is to capture local motion energy at a specific spatiotemporal frequency, which is associated with the function of a direction-selective neuron in the V1 cortex. The responses of neurons can be modeled as 3D Gabor filters [30, 31]. Gabor filters are known to be optimal in the sense that they achieve maximal resolution in both the spatiotemporal and associated frequency domains[32]. To save computational complexity, we decomposed 3D spatiotemporal filters into filters separable in space and time. The spatial component is described by 2D Gabor filters  $\mathcal{G}(\cdot)$ , and the temporal component  $\mathcal{T}(\cdot)$  is described
Figure 1: Molding of motion perception system, Stage I. (A): The first stage is built using a group of trainable motion energy units to capture local motion energy; (B): Motion energy calculation based on spatiotemporal separable filters, a sub-block of (A); (C): Demo of spatiotemporal separable filters, a sub-block of (B).
by a 1D sinusoidal function with exponential decay. Specifically, given  $x' = x \cos \theta + y \sin \theta$  and  $y' = -x \sin \theta + y \cos \theta$  within the receptive field, the impulse responses of spatial and temporal complex filters are defined as:
$$
\left\{ \begin{array}{l} \mathcal {G} (x, y; f _ {s}, \theta , \sigma , \gamma) = \exp \left(- \frac {x ^ {\prime 2} + \gamma^ {2} y ^ {\prime 2}}{2 \sigma^ {2}}\right) \exp (i (2 \pi f _ {s} x ^ {\prime})), s. t. \{x, y \mid (x ^ {2} + y ^ {2} \leq R ^ {2}) \} \\ \mathcal {T} (t; f _ {t}, \tau) = \exp (- \frac {t}{\tau}) \exp (2 \pi i (f _ {t} t)), s. t. \{t \mid 0 \leq t <   T \} \end{array} \right. \tag {1}
$$
The parameters in red are designed to be trained to fit the dataset, where  $f_{s}$  and  $f_{t}$  denote the spatiotemporal frequency tunings of the filter with the preferred speed  $v \propto \frac{f_t}{f_s}$ , and  $\theta$  determines the preferred moving orientation;  $\sigma$  and  $\gamma$  control the shape of the Gabor filter, and  $\tau$  controls the degree of attenuation of the temporal impulse response. All parameters are subject to certain numerical constraints, such as  $\theta$  being limited to  $[0,2\pi)$  to avoid redundancy;  $f_{s}$  and  $f_{t}$  are limited to less than 0.25 pixels per frame to avoid spectrum aliasing, etc. The response of a simple direction-selective cell  $L_{n}$  to a video stimuli  $\mathbf{S}(\mathbf{p},\mathbf{t})$  can be computed via separate convolutions:
$$
L _ {n} (x, y, t; \Theta) = (\mathbf {S} * \mathcal {G}) * \mathcal {T} = \iiint \mathbf {S} (\mathcal {X}, \mathcal {Y}, \mathcal {T}) \cdot \mathcal {G} _ {n} (x - \mathcal {X}, y - \mathcal {Y}) \cdot \mathcal {T} _ {n} (t - \mathcal {T}) d (\mathcal {X}, \mathcal {Y}) d \mathcal {T} + \alpha_ {1} \tag {2}
$$
where  $\alpha_{1}$  can be learned as spontaneous firing rates. Further, local motion energy is captured by a phase-insensitive complex cell in the V1 cortex, which computes the squared summation of the response from a pair of simple V1 cells with approximately orthogonal spatiotemporal receptive fields[24]. We denote the pair of orthogonal (even and odd) simple V1 cells as:
$$
\left\{ \begin{array}{l} L _ {n} ^ {o} (x, y, t; \Theta) = (\mathbf {S} * \operatorname {I m} [ \mathcal {G} ]) * \operatorname {R e} [ \mathcal {T} ] + (\mathbf {S} * \operatorname {I m} [ \mathcal {G} ]) * \operatorname {I m} [ \mathcal {T} ] \\ L _ {n} ^ {e} (x, y, t; \Theta) = (\mathbf {S} * \operatorname {R e} [ \mathcal {G} ]) * \operatorname {R e} [ \mathcal {T} ] - (\mathbf {S} * \operatorname {I m} [ \mathcal {G} ]) * \operatorname {I m} [ \mathcal {T} ], \end{array} \right. \tag {3}
$$
where  $\mathrm{Re}(\cdot)$  and  $\mathrm{Im}(\cdot)$  extract the real and imaginary parts of a complex number;  $*$  denotes convolution operations. Then, the response of a complex cell  $L_{n}^{c}$  is obtained from a combination of the quadrature pair of the simple cells using the motion energy formulation:
$$
L _ {n} ^ {c} (x, y, t; \Theta) = \left(L _ {n} ^ {o} (x, y, t; \Theta)\right) ^ {2} + \left(L _ {n} ^ {e} (x, y, t; \Theta)\right) ^ {2} \tag {4}
$$
Multi-scale Wavelet Processing: The convolution kernel of our spatial filter has a fixed size of  $15 \times 15$ , which imposes a physical limitation on the receptive field of each unit. To enhance the flexibility of the receptive field size, we employed a multi-scale processing strategy, as shown in Fig. 1 (A). Specifically, we constructed an image pyramid consisting of eight images scaled linearly
Figure 2: Molding of motion perception system, Stage II. (A): The second Stage is constructed by using self-attention mechanism and recurrent processing to simulate the function of global motion integration and segregation. We use an identical flow decoder to visualize the dense optical flow at each iteration. (B): the global motion integration process based on self-attention mechanism, a sub-block of (A).
from  $H \times W$  to  $\frac{H \times W}{16}$ . In total, 256 independent complex cells are deployed across different scales, with the lower scale having a larger receptive field and a preference for a faster motion speed. This approach is computationally efficient for capturing large-scale displacements in engineering applications[33, 34], and enables the representation of different groups of cells sensitive to short- and long-distance motions [35]. The N complex cells  $\{L_{n}^{c}\}_{i}^{N}$  capture motion energy on multiple scales. We applied energy normalization to each cell to ensure consistent energy levels:
$$
\hat {L} _ {n} ^ {c} (t) = \frac {K _ {1} L _ {n} ^ {c} (t)}{\sum_ {\mathrm {i} = 1} ^ {\mathrm {N}} L _ {\mathrm {i}} ^ {c} (t) + \sigma_ {1}}, \tag {5}
$$
where  $\sigma_{1}$  is the semi-saturation constant of the normalization, and  $K_{1} > 0$  determines the maximum attainable response. We interpret the response, denoted by  $\hat{L}_n(t)$ , as the model equivalent of a post-stimulus time histogram (PSTH), which is a measure of the neuron's firing rate. Physiologically, such responses could be computed via inhibitory feedback mechanisms[36, 37]. Considering the spatial arrangement of images, it is expected that motion energy responses should exist at each spatial location generated by the same complex cell groups. Bilinear interpolation is used to resize the multi-scale motion energy into the same spatial size  $= \frac{H \times W}{8}$ . In the context of DNNs, this is mainly to balance the trade-off between the spatial resolution and computational overhead, and the final output of the first stage is a 256-channel feature map  $\mathbf{E} \in \mathbb{R}^{\frac{H}{8} \times \frac{W}{8}}$  that captures the underlying local motion energy, which partially characterizes the cellular patterns of the V1 cortex in a computational manner[24].
2.2 Second stage: Global Motion Integration and Segregation
The receptive field size of first-stage neurons limits their ability to capture only local motion. Advanced spatial integration is an essential requirement for a motion perception system to solve the aperture problem[7]. Spatial integration could involve several mechanisms, such as object recognition and segmentation[38], depth estimation[39], contour structure inference[40], etc., which rely on substantial prior information that may go beyond the capabilities of classical modeling methods. DNNs, with their massive number of parameters and flexibility, offer a suitable solution. However, the spatial integration of local motion requires more flexible connectivity relations than what can be achieved by general convolutions, which are limited by their local receptive field[41]. To overcome this limitation, we propose a computational model based on the attention mechanism and recurrent processing to model the function of motion integration.
Constructing a Graph Structure: First, to establish more flexible connections between cells, we discard the Euclidean space structure within the image and construct topological spaces with an
undirected weighted graph  $\mathbf{G} = \{\mathbf{V},\mathbf{A}\}$ , where  $\mathbf{V}$  is the set of nodes;  $\mathbf{A}$  represents the adjacent matrix based on the graph. Each spatial location  $p(i,j)$  is treated as a node, and the feature of each node is derived from the whole set of local motion energies  $\mathbf{E}(i,j) = \{\hat{L}_n^c (t)\}_{i = 1}^{256}$ . The connection between any pair of nodes is computed using a specific distance metric, and strong connection relationships are formed between nodes whose local motion energy patterns are similar.
Specifically, given a reshaped feature map  $\mathbf{F} \in \mathbb{R}^{HW \times 256}$ , the features at each of its locations are first locally embedded into the graph space, as  $\varphi(\mathbf{F}) = \mathrm{GELU}(\mathbf{F} \cdot \mathbf{W}_{\Theta 1}) \cdot \mathbf{W}_{\Theta 2}$ , where  $\mathbf{W}_{\Theta} \in \mathbb{R}^{256 \times 256}$  is a group of trainable parameters. The distance between any pair of nodes  $(i,j)$  is calculated by the cosine similarity, which is similar to the self-attention mechanism in current transformer structures[26, 42, 27]. We employ the adjacency matrix  $\mathbf{A} \in \mathbb{R}^{HW \times HW}$  to represent the connectivity of the whole topological space, and  $\mathbf{A}$  is a symmetric semi-positive definite matrix defined as:
$$
\mathbf {A} (i, j) = \mathbf {A} (j, i) = \frac {\varphi (\mathbf {F}) _ {i} \cdot \varphi (\mathbf {F}) _ {j}}{\| \varphi (\mathbf {F}) _ {i} \| \| \varphi (\mathbf {F}) _ {j} \|.} \tag {6}
$$
We perform exponential scaling of the connections between graphs using the matrix  $\mathbf{A}$ , given by  $\exp(\mathbf{A}s)$ , where  $s$  is a learnable scalar restricted to the range  $(0,10)$  to avoid overflow. The smaller  $s$ , the smoother the connections across nodes and vice versa. Finally, a symmetric normalization operation is utilized to balance the energy, given by  $\mathbf{A} := \mathbf{D}^{-\frac{1}{2}}\exp(s\mathbf{A})\mathbf{D}^{-\frac{1}{2}}$ , where  $\mathbf{D}$  is the degree matrix with  $\mathbf{D} = \mathrm{diag}\left(\left\{\sum_{j}\exp(s\mathbf{A}_{i,j})\right\}_{i=1}^{n}\right)$ . As such, an energy-normalized undirected graph structure is constructed, as illustrated in the top side of Fig. 2 (B). Intuitively, this adjacency matrix represents the neuron's affinity or connectivity within the space, with strong and global connections built across neurons with related motion responses.
Recurrent Integration Processing: Recurrent neural networks (RNNs) are often used to simulate neurons in the brain, as they are flexible in modelling temporal dependencies and feedback loops, which are fundamental aspects of neural processing in the brain[43]. We use a recurrent network, rather than multiple feedforward blocks, to simulate the process of local motion signals being gradually integrated into MT and eventually converging to a stable state.
As shown in Fig. 2 (A), the local motion energy from the first stage is divided into two recurrent streamlines. One is the motion energy  $\mathbf{E} \in \mathbb{R}^{H \times W \times 256}$ , which is continuously updated in the loop, while the other is embedded in the attention space to generate the graph adjacency matrix to control motion integration, denoted as  $\mathbf{F} \in \mathbb{R}^{H \times W \times 256}$ . In each iteration, the adjacency matrix is first constructed using  $\mathbf{F}$ . Subsequent motion integration is achieved through a simple matrix multiplication, which is computationally similar to the information propagation mechanism in transformers[26, 42] and can also be considered a simplified version of graph convolution[44]. The integrated motion information is passed through two independent Conv-GRU blocks to update the motion energy  $\mathbf{E}$  and feature  $\mathbf{F}$ , respectively. The Conv-GRU represents a gated recurrent unit[45] implemented in a convolutional manner, and we adopt a spatio-temporal separable approach following RAFT[14]. The motion integration process approximates the ideal final convergence state of the motion energy  $\mathbf{E}_k \rightarrow \mathbf{E}^*$  through recurrent iteration.
Decoding the Motion Flow: We adopt the same strategy of decoding the 2D optical flow in each iteration. Initially, the integrated motion is  $\mathbf{E}$  projected back to the energy space with positive value using a square operation, followed by an energy normalization operation:  $\hat{\mathbf{E}}(i,j) = \{K_2\mathbf{E}^2(i,j)\} / \{\sum_{i,j}^{\mathrm{HW}}\mathbf{E}^2(i,j) + \sigma_2^2\}$ . The resulting response  $\hat{\mathbf{E}} \in \mathbb{R}^{H \times W \times 256}$  is interpreted as a post-stimulus time histogram and the motion energy is constrained to the same energy space as the local motion energy from stage I, as we further design an identical flow decoder to project the energy of each spatial location into the optical flow. The structure of the flow decoder is a nonlinear mapping process consisting of multiple  $1 \times 1$  convolution blocks with residual connections, which are referred to in several current advanced optical flow estimation models[33, 46]. Additionally, a convex upsampling strategy[14] is employed to restore the optical flow's original resolution. The entire architecture of stage II is illustrated in Fig. 2 (A, B).
2.3 Training Strategy
Current methods for estimating optical flow using deep neural networks (DNNs) can be categorized as unsupervised/self-supervised and supervised learning approaches. While unsupervised learn
Figure 3: Properties of trained complex cells in stage I. (A): Orientation-speed distribution of trained motion energy units. (B): Spatiotemporal frequency tuning distribution of trained motion energy units. (C), (D): The ability to tune simple motions with a single spatiotemporal frequency component has emerged in the stage I, exemplified by direction tuning of drifting Gabor stimuli.
ing methods are intuitively similar to creatures' interaction with the world, most current methods based on differentiable image warping[47-49] still try to approximate physical motion ground truth. Therefore, we adopt a supervised learning approach in this work, which is more straightforward as recent research suggests that human perception of motion is reasonably similar to physical GT[16]. However, our primary goal lies in evaluating how well the model approximates human motion perception rather than its accuracy in predicting GT. To train and evaluate the model, we construct a dataset containing various natural and artificial motion scenes. Specifically, we incorporate the Sintel benchmark[50], the DAVIS[51] dataset with pseudo-labels generated by FlowFormer[52], as well as self-created multi-frame datasets with non-texture motions and drifting grating motion. Including simple motion stimuli and drifting gratings allows the model to generalize under different non-texture conditions while providing a potential slow-world Bayesian prior[9]. The model is first pre-trained with simple motion and subsequently fine-tuned on complex natural scenes to facilitate convergence[12]. More specific training details can be found in the supplementary materials.
3 Analyses
Fig. 3 shows the distribution of trained parameters in the first stage: (A) presents the distribution of velocity and orientation of the units; (B) displays the spatiotemporal frequency tuning of the units; (C) demonstrates that the complex cells in the first stage are capable of handling single-frequency component motions such as drifting gratings. The design of the trainable motion energy unit allows for optimal fitting to the flow statistics of the dataset. Although the distribution of the trained parameters appears to lack specific characteristics other than uniformity, it does reflect the effect of training. To validate the effectiveness of training the motion energy units, we conducted experiments with a fixed tuning parameter design using a uniform distribution sampled at equal intervals in terms of spatiotemporal frequency and orientation. The results showed that stage I without the fitting function significantly degraded the model's ability to estimate motion. (See Our-fixed in Table 1 for details).
The following three parts are conducted for analysis: 1) In silico neurophysiological test of the activation pattern of the units; 2) Psychophysical stimulus tests comparing human perception and model response; 3) Natural scene test of the generalizability of the model in complicated scenarios.
3.1 In silico Neurophysiological Study