Modeling Human Visual Motion Processing with Trainable Motion Energy Sensing and a Self-attention Network
Authors: Zitang Sun
Yen-Ju Chen
Yung-Hao Yang
Shin'ya Nishida
Venue: NeurIPS 2023 poster (2023)
Keywords: motion perception
optical flow estimation
attention mechanism
psychophysics
In silico neurophysiology
human vision
TL;DR: A two-stage approach that combines trainable motion energy sensing with a recurrent self-attention network for motion integration is proposed to bridge the gap between human and computer vision models of visual motion perception.

Abstract: Visual motion processing is essential for humans to perceive and interact with dynamic environments. Despite extensive research in cognitive neuroscience, image-computable models that can extract informative motion flow from natural scenes in a manner consistent with human visual processing have yet to be established. Meanwhile, recent advancements in computer vision (CV), propelled by deep learning, have led to significant progress in optical flow estimation, a task closely related to motion perception. Here we propose an image-computable model of human motion perception by bridging the gap between biological and CV models. Specifically, we introduce a novel two-stages approach that combines trainable motion energy sensing with a recurrent self-attention network for adaptive motion integration and segregation. This model architecture aims to capture the computations in V1-MT, the core structure for motion perception in the biological visual system, while providing the ability to derive informative motion flow for a wide range of stimuli, including complex natural scenes. In silico neurophysiology reveals that our model's unit responses are similar to mammalian neural recordings regarding motion pooling and speed tuning. The proposed model can also replicate human responses to a range of stimuli examined in past psychophysical studies. The experimental results on the Sintel benchmark demonstrate that our model predicts human responses better than the ground truth, whereas the state-of-the-art CV models show the opposite. Our study provides a computational architecture consistent with human visual motion processing, although the physiological correspondence may not be exact.

Modelling Human Visual Motion Processing with Trainable Motion Energy Sensing and Self-attention Network
Zitang Sun'
Yen-Ju Chen' Yung-Hao Yang' Shin'ya Nishida' 12-
GGaduate School of Informatics, Kyoto University
NTT Communication Science Laboratories, Nippon Telegraph and Telephone Corporation {sun. zitang..aag..................z.. 73u, chen. yenju.njjuu.. 442, yang. yang- yunghao 88v
Pst.kyoto-u.ac.pp sisnada kyoto-u.ac.jp
Abstract
Visual motion processing is essential for humans to perceive and interact with dy- namic environments. Despite extensive research in cognitive neuroscience, image- computable models that can extract informative motion flow from natural scenes in manner consistent with human visual processing have yet to be established. Meanwhile, recent advancements in computer vision (CV), propelled by deep learning, have led to significant progress in optical flow estimation, task closely related to motion perception. Here we propose an image-computable model hu- man motion perception by bridging the gap between biological and CV models. Specifically, we introduce a novel two-stages approach that combines trainable motion energy sensing with a recurrent self-attention network for adaptive motion integration and segregation. This model architecture aims to capture the compu- tations V1-MT, the core structure for motion perception in the biological visual system, while providing the ability to derive informative motion flow for wide range stimuli, including complex natural scenes. In silico neurophysiology re- veals that our model's ss unit responses are similar to mammalian neural recordings regarding motion pooling and speed tuning. The proposed model can also repli- cate human responses to range of stimuli examined in past psychophysical stud- ics. The experimental results on the Sintel benchmark demonstrate that our model predicts human responses better than the ground truth, whereas the kate-offthe--al CV models show the opposite. Our study provides computational architecture consistent with human visual motion processing, although the physiological cor- respondence may not be exact.
Introduction
Visual motion perception is essential not only for humans and animals to perceive and interact with the world but also for artificial agents to process various dynamic visual tasks. As such, visual mo- tion estimation has been extensively studied by both biological vision and computer vision research communities. The key issue lies in estimating optical flow, an array of instantaneous image motion vectors 2)
Vision science has revealed that optical flow estimation in the human visual system (HVS) pri- marily served by pathway that includes the primary visual cortex (V1 4) and middle temporal (MT [5] area. A significant portion of neurons area VI are sensitive to the direction of local motion, while those in area MT integrate and segregate local motion signals for global flow inter- pretation. The MT process also helps to overcome the aperture problem 7 Several computational models of area MT have been proposed g 9 but mechanisms that can fully encapsulate a variety
Projecc Website:
Click here
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

of motion integration abilities of MT neurons and/or human perception have yet to be developed [T]. Furthermore, previous studies tested the models using only simple artificial stimuli. remains challenging to create human-like optical flow extraction mechanism that can derive informative motion flow for a wide range of stimuli including complex natural videos.
On the other hand, optical flow estimation has recently made remarkable progress in the field of com- puter vision. [II pioneered the use of fully convolutional neural networks for dense opti- cal flow estimation, and various approaches based on deep neural networks (DNNs) have emerged subsequently 12 [12 Owing to the powerful representation ability of DNNs, these models out- perform humans in estimating the ground-trun -ruth optical flow of natural scenes [16]. Consequently, one might expect DNN-based optical flow estimation algorithms to become promising models of human visual motion processing, similar to how ImageNet-trainee DNN models provide good com- putational models of human object recognition 17 However, since computer vision optical flow models are designed solely to find local image correspondences between pairs of frames on the image coordinates [TI], they cannot explain systematic or adaptive deviations of human perceived motion from local ground truth (GT) [16]. Moreover, existing DNN models often exhibit instability when dealing with non-textured stimuli commonly used in vision science [1]
In this study, we leveraged the flexibility of DNNs to construct an image-computationall model of human motion processing. While recent studies successfully used DNNs to elaborate the under- standing of the neural mechanism of visual motion processing 23), we aimed to make a model that can explain a broader range of physiological and psychophysical phenomena, including those whose neural mechanisms are not yet clear. From an engineering standpoint, we aimed to make a human-aligned optic flow algorithm while maintaining flow estimation capability comparable to the ifat-oof-the-ar (SOTA) CV models. Our model extracts dense informative motion flows for wide range of inputs in a way consistent with physiologically measured neural responses and psy- chophysically measured human motion perception. It consisted of two stages. The first stage, which mimicked the function of V1, comprised of neurons with multi-scale spatiotemporal filters to extract local motion energy. Unlike previous models of V1 [4 8, 25 the filter tunings were learnable to fit natural optic flow computation. The second stage, which mimicked the function of MT recurrently integrated local motion signals, and solved the aperture problem. We constructed an undirected fully connected graph from the map of local motion energy, and used the attention mechanism 22 for adaptive global motion integration and segregation.
We evaluated the performance of our model from several aspects. In in-silico neurophysiology, we found that our model's neurons exhibited direction and speed tunings similar to those observed in mammalian physiological recordings in VI and MT (28] (2).. In simulations of psychophysical find- ings, our model showed good generalization from simple artificial stimuli (e.g.. drifting Gabor) to complex naturalistic scenes. Our model produced human-like responses for several conventional motion stimuli and illusions, including global motion pooling and the barber-pole illusion. Further- more, the mode's response to natural scenes was closer to that of humans in comparison to other computer vision models. Our two-stages model provides computational architecture consistent with human visual motion processing, although the physiological correspondence may not be exact. This achievement is not only valuable in terms of neuroscientific understanding of human visual computation but also for the development of human- human-aligned machine vision that stably recognizes the world as humans do.
2 Molding of two-stages motion perception system
2.1 First-stage: Local Motion Energy Computation
Spatiotemporal separable Gabor filter: Since we aimed at an image-computable model, the in- put sequence of grayscale images S(, t) for all spatial positions p = (z, y) within the image domain Z and for all times t > O. The goal of the first-stage neuron is to capture local motion energy at a specific spatiotemporal frequency, which is associated with the function of direction- selective neuron in the V1 cortex. The responses of neurons can be modeled as 3D Gabor filters 30 3I] Gabor filters are known to be optimal in the sense that they achieve maximal resolution in both the spatiotemporal and associated frequency domains 32). To save computational complex- ity, we decomposed 3D spatiotemporal filters into filters separable in space and time. The spatial component is described by 2D Gabor filters (-),, and the temporal component T(-) is described

---

S(x.y.)) BEWW
The Inference Pipeline of Stage for Capturing Local Motion Energy (A)
Multi scale Video Pyramid
Interpolate
8
repoase
from camples
F.C.0
( SeaipeedatSSutttrrrnn Seponite Crandatioo
R00
30
Energy Normalization
E.
Spatiotemporal separable convolation
Tempural weightd average pooling & Energy Constrain
Demo of Spatletemporal Receptive field (C)
Gaiureads Elke
Larppeni
Spatial convolation
F00 Bo
Spetwtrrpeo (Qnadotere
Temporal
convolution
Motion energy grree
Figure 1: Molding of motion perception system, Stage (A): The first stage built using group of trainable motion energy units to capture local motion energy: (B): Motion energy calculation based on spatiotemporal separable filters, a sub-block of (A); (C): Demo of spatiotemporal separable filters, a sub-block of (B).
by ID sinusoidal function with exponential decay. Specifically, given x' = cos 0 + sin & and y' sin cos within the receptive field, the impulse responses of spatial and temporal complex filters are defined as:
{ 9(a, (f,,,,,,,,,,1 - =xp 2 20
exp (i (2mfz²))) s.t. {z, 1<x + + s R²)}
(t) cxp (-+) exp (2 (fit)), s.t. (t S <
(1) The parameters red are designed to trained to the dataset, where Le and Js denote the spatiotemporal frequency tunings the filter with the preferred speed v x fs and 0 determines the preferred moving orientation; a and y control the shape the Gabor filter, and controls the degree of attenuation of the temporal impulse response. All parameters constraints, such as 9 being limited to [0,2m) to avoid redundancy; fa and ft are limited to less than 0.25 pixels per frame avoid spectrum aliasing, etc. The response of simple direction-selective cell I,, to a video stimuli S(p, t) can be computed via separate convolutions:
subject to certain numerical
La u, 1; y,,
( (S*
T)
where 01 can be learned as spontaneous firing rates. Further, local motion energy is captured by phase-insensitive complex cell in the VI cortex, which computes the squared summation of the response from a pair simple cells with approximately orthogonal spatiotemporal receptive fields 24). We denote the pair of orthogonal (even and odd) simple V1 cells
{ (,( (S lmlg)) Re T (S Im [ Im T
-
[g(,,,,40) = Re Re[T] (S [m -m
(3)
where Re(-) and Im(-) extract the real and imaginary parts of complex number; * denotes convo- lution operations. Then, the response of complex cell LŞ is obtained from a combination of the quadrature pair of the simple cells using the motion energy formulation:
D (x. y. t; θ) (Z, y, 0)) + (L, y. 0))
(4)
Multi-scale Wavelet Processing: The convolution kernel our spatial filter has a fixed size of 15 x 15, which imposes physical limitation on the receptive field of each unit. To enhance the flexibility of the receptive field size, we employed multi-scale processing strategy, as shown in Fig. (A). Specifically, we constructed an image pyramid consisting of eight images scaled linearly

---

Recurrent Misiion Integration
Integration (IMMIl)
MY
Apumarc Prakkn
Land Maisen Fnergy
Energy recale
Remap
Narealbation
Di
Cun-CRE
goowse
RMMBB
c. gif aifiiraees
Adjacsee Marris
Remop
hirgreliis
RMIB
Kenap
Deceder
E a attrrv
206
Craro
Maltilager Persepttinn Mackias (MIL))
Rennapping io Exerge Sqace (Roenap)
Shared Flom Decoder
Re Roikage Operniion
Mosie Lacce Rocamatos
Ead
SELL
SELA
Normi
Dut product
Stan padpciino
Alu dus Rassa
Eastar
Conn
Bidk uaav side
Figure 2: Molding of motion perception system, Stage (A): The second Stage is constructed by using self-attention mechanism and recurrent processing to simulate the function of global motion integration and segregation. We use an identical flow decoder to visualize the dense optical flow at each iteration. (B): the global motion integration process based on self-attention mechanism, sub-block of (A).
HxW. In total, 256 independent complex cells are deployed across different scales, from H x to 15
with the lower scale having larger receptive field and preference for faster motion speed. This approach is computationally efficient for capturing large-scale displacements in engineering applications [333] 34), and enables the representation of different groups of cells sensitive short- and long-distance motions [35]. The N complex cells {TA]] J capture motion energy on multiple
scales. We applied energy normalization each cell to ensure consistent energy levels:
K1 (t)
(S)
EN (F(() +
where 01 is the cati-saturation constant of the normalization, and K₁ 0 determines the maxi- mum attainable response. We interpret the response, denoted by (t), the model equivalent of a post-stimulus time histogram (PSTH), which is measure of the neuron's 's firing rate. Physiologi- cally, such responses could be computed via inhibitory feedback mechanisms 36) 37.. Considering the spatial arrangement of images, it is expected that motion energy responses should exist at each spatial location generated by the same complex cell groups. Bilinear interpolation is used resize the multi-scale motion energy into the same spatial size #*W In the context of DNNs, this is mainly balance the trade-off between the spatial resolution and computational overhead, and the final output of the first stage is a 256-channel feature map R x that captures the under- lying local motion energy, which partially characterizes the cellular patterns of the V1 cortex in computational manner 24).
2.2 Second stage: Global Motion Integration and Segregation
The receptive field size of first-stage neurons limits their ability to capture only local motion. Ad- vanced spatial integration is an essential requirement for motion perception system to solve the aperture problem[7]. Spatial integration could involve several mechanisms, such as object recog- nition and segmentation (8), depth estimation 39], contour structure inference [40], etc., which rely on substantial prior information that may go beyond the capabilities of classical modeling methods. DNNs, with their massive number of parameters and flexibility, offer suitable solution. However, the spatial integration of local motion requires more flexible connectivity relations than what can be achieved by general convolutions, which are limited by their local receptive field[41]. To overcome this limitation, we propose a computational model based on the attention mechanism and recurrent processing model the function of motion integration.
Constructing a Graph Structure: First, to establish more flexible connections between cells, we discard the Euclidean space structure within the image and construct topological spaces with an

---

undirected weighted graph G = (V,A), where V is the set of nodes; A represents the adjacent matrix based on the graph. Each spatial location p(i.j) is treated node, and the feature of each node derived from the whole set of local motion energies E(i, j) = {t) | The connection between any pair of nodes is computed using specific distance metric, and strong connection relationships are formed between nodes whose local motion energy patterns are similar.
Specifically, given reshaped feature map F E I #W #2Wx2566 x256 the features at each of its locations are first locally embedded into the graph space, as (p(F) = GELU(F We1) Wee where We E RR25xx56 is group of trainable parameters. The distance between any pair of nodes (i.j) calculated by the cosine similarity, which is similar to the self-attention mechanism in current We employ the adjacency matrix A E R #HW to represent the connectivity of the whole topologiaal space, and A is a symmetric semi-positive definite matrix defined as:
transformer structures 26 42 27
(F), (FF); ( (F),
(i. A(i
We perform exponential scaling of the connections between graphs using the matrix A, given by exp(As) (As), where is learnable scalar restricted to the range 10) to avoid overflow. The smaller 8; the smoother the connections across nodes and vice versa. Finally, a symmetric normalization op-
cration is utilized to balance the energy, given by A == D D==eep(=AAD ex)()A)D - where D is the degree undirected graph
matrix with D = diag {{Z, exp(sA..)) - As such, an mergy--orrallee
structure is constructed, illustrated in the top side of Fig. (B). Intuitively, this adjacency matrix represents the neuron's 's affinity or connectivity within the space, with strong and global connections built across neurons with related motion responses.
Recurrent Integration Processing: Recurrent neural networks (RNNs) are often used to simulate neurons in the brain, as they are flexible in modelling temporal dependencies and feedback loops, which are fundamental aspects of neural processing in the brain brain[43]. We use a recurrent network, rather than multiple feedforward blocks, to simulate the process of local motion signals being grad- ually integrated into MT and eventually converging to a stable state.
As shown in Fig. 2 (A), the local motion energy from the first stage is divided into two recurrent streamlines. One is the motion energy E R XXX'x225 which continuously updated in the loop, while the other is embedded in the attention space to generate the graph adjacency matrix to control motion integration, denoted as E R.. x x256 In each iteration, the adjacency matrix is first con- structed using F. Subsequent motion integration is achieved through simple matrix multiplication, which is computationally similar to the information propagation mechanism in transformers (26mnnrrs[2]] 42] and can also be considered a simplified version of graph convolution (44) The integrated motion information is passed through two independent Conv-GRU blocks to update the motion energy E and feature F, respectively. The Conv-GRU represents gated recurrent unit[45) implemented in a convolutional manner, and we adopt a spatio-temporal separable approach following RAFT[14). The motion integration process approximates the ideal final convergence state of the motion energy Eg + E through recurrent iteration.
We adopt the same strategy of decoding the 2D optical flow in each iteration. Initially, the integrated motion is E projected back the energy space with pos- itive value using a square operation, followed by an energy normalization operation: E(i,j) = (i,,)
Decoding the Motion Flow:
E²(i,)) + +2²}. The resulting response È E #x x256 is interpreted as a post-stimulus time histogram and the motion energy is constrained to the same energy space as the local motion energy from stage I, as further design an identical flow decoder project the energy of each spatial location into the optical flow. The structure of the flow decoder is a nonlinear mapping process consisting of multiple 1 X convolution blocks with residual connections, which are referred to in several current advanced optical flow estimation models(33) 46). Additionally, a convex upsampling strategy [14] is employed to restore the optical flow's original resolution. The entire architecture of stage II is illustrated in Fig. 2 (A, B).
2.3 Training Strategy
Current methods for estimating optical flow using deep neural networks (DNNs) can be categorized as unsupervised/seffsupprvii and supervised learning approaches. While unsupervised learn-
5

---

60
OrientationSpeed Distribution
B. Spalioemmoral Frequeccy Distribution
C. Loral Mution Procrasing Stage
D. Direction Tuning of
Neurons in Stage I
70
[Teappre frequeeecl greloframmcc
Stinnll
Medion Requans
209
Pspatial Frequeecc] cyclolgiee
225
6.0
0.15
3.50
$35
Karrgy Reqpanse 149
Normanaed Speed
Figure 3: Properties of trained complex cells in stage I. (A): Orientation-speed distribution of trained motion energy units. (B): Spatiotemporal frequency tuning distribution trained motion energy units. (C), (D); The ability to tune simple motions with a single spatiotemporal frequency component has emerged in the stage I, exemplified by direction tuning of drifting Gabor stimuli.
ing methods intuitively similar to creatures' interaction with the world, most current methods based on differentiable image warping 47H49) still try to approximate physical motion ground truth. Therefore, we adopt supervised learning approach in this work, which is more straightforward as recent research suggests that human perception motion reasonably similar to physical GT(16]. However, our primary goal lies in evaluating how well the model approximates human motion per- ception rather than its accuracy in predicting GT. To train and evaluate the model, we construct dataset containing various natural and artificial motion scenes. Specifically, we incorporate the Sintel benchmark [50], the DAVIS[51] dataset with pseudo-labels generated by FlowFormer (52), as well self-created multi-frame datasets with non-texture motions and drifting grating motion. In- cluding simple motion stimuli and drifting gratings allows the model to generalize under different non-texture conditions while providing potential slow-world Bayesian prior [
The model first pre-trained with simple motion and subsequently fine-tuned on complex natural scenes to facilitate More specific training details can be found in the supplementary materials.
convergence 12
3 Analyses
Fig. shows the distribution of trained parameters in the first stage: (A) presents the distribution of velocity and orientation of the units; (B) displays the spatiotemporal frequency tuning of the units: (C) demonstrates that the complex cells in the first stage are capable of handling single- frequency component motions such as drifting gratings. The design the trainable motion energy unit allows for optimal fitting to the flow statistics of the dataset. Although the distribution of the trained parameters appears to lack specific characteristics other than uniformity, it does reflect the effect of training. To validate the effectiveness of training the motion energy units, conducted experiments with a fixed tuning parameter design using uniform distribution sampled at equal intervals in terms of spatiotemporal frequency and orientation. The results showed that stage I without the fitting function significantly degraded the model's 's ability to estimate motion. (See Our- fixed in Table for details).
The following three parts are conducted for analysis: 1) In silico neurophysiological test of activation pattern of the units; 2) Psychophysical stimulus tests comparing human perception and model response; 3) Natural scene test of the generalizability of the model complicated scenarios.
3.1 In silico Neurophysiological Study
Some VI and MT neurons respond selectively to a specific range of motion directions. To investigate the directional tuning of the units in our model, we utilized silico neu- rophysiology to measure the activation patterns of 256 units in response to drifting Gabor and plaid stimuli. plaid consists of two superimposed drifting Gabors, as illustrated Fig. (C). Analysis of the data revealed three distinct groups of units based on their partial correlation to Gabor and plaid stimuli: 1) Component cells, which always respond to the direction of Gabor component; 2) Pattern cells, which respond to the integrated (coherent) motion direction of plaid; and 3) Un-
Directional Tuning:

---

Lait's Pattern distributions
Unit's Category variation Iterations (8)
Wastrathi cell Patterns (C)
Nearan Reconding
Percenage
Patter
Model Reseene
Componcnt
Pattern
Unclnsiifed
Unstamiled
Spectral Receptine Finki Destriration (P)
Orientation Spectral Receplive (Fald
Classification Speed Independent Call:(F)
Neuron recoreling
Dierbutii of
Lerpert Popprts
Seage
Specil
Stege
simple
50
1.00
Stage
Partial Cuer dalien Indepeadent
Seutal Receaaay
Figure 4: In Silico Neurophysiological Test. (A, B, C): Analysis of neurons for ID and 2D stimuli; Stage II-N means the result from iteration Nth in the stage II; (D, E, F): Analysis of the spectral receptive field and speed tuning of different groups of ncurons. The ncural distribution Two neurons
direction tuning of different groups
in (A) is redrawn from (29.. Neurons' spectral receptive fields in (D) are redrawn from 53 distributions (F) are separately taken from and [55
classified cells, which do not show a clear preference for either component or pattern motions, as illustrated in Fig. 4 (C). The distribution of these cell types is not uniform, with component cells being more commonly found in the primary visual cortex (V1), while pattern cells are more often observed in the MT and MST regions[? In agreement with this, illustrated in Fig. (A) (B), our model demonstrates that in the first stage, most units tend to be component cells, whereas the number of pattern and unclassified cells increases in the second stage. In addition, employed the maximizing activation method [56] to reversely render their cellular preferences, showing that complicated motion patterns consisting of both central and background motions, as presented at the bottom of Fig. 4 (B). This suggests that the classical classification of motion neurons into component and pattern cells might be insufficient to characterize the motion integration properties of these neurons.
the second-stage unclassified units respond
frequeeccy-s---eeeeeeeeeeeee frequency) receptive field of the model's units using a combination of drifting Gabors with different spatiotemporal frequencies. Two-dimensional oriented Gaussian contours used to depict the receptive field of the 256 cells, fitted by minimizing the least square error. The results are shown in 4 Visually, the distribution of the receptive field tilt angle spreads from horizontal/vertical directions to oblique directions (Fig. 4 E). This indicates that the units in the stage II have significant speed tuning compared the stage Speed tuning is characteristic of higher-order visual motion neurons (88 and is often found in the MT area[57] [5]]. This tendency can be seen from the distribution of the partial correlation between the actual receptive field and its speed prediction/indeeendent prediction 55), as demonstrated in Fig. (F). Our two-stages process shows a degree of consistency with the change in mammalian neural distribution from the VI to MT area.
Spectral Receptive Field and Speed Tuning: We test the spectral
3.2 Psychophysical Analysis
Fourier motion: In the "missing fundamental illusion" [60], as shown in Fig. 5 (C). when the first spatial harmonic removed from a square-wave wave grating with quarter-cycle shift, the perceived motion direction appears reversed. Our model, whose first stage estimates motion from the Fourier
7

---

Global Motion Inlegration (A)
Horos prinival Eincion
Motion Inlegration-Pruui Preiooty-ee
Missing Fundameural (C)
Local slyvai moio
Stage Mation)
EAAFT
Furechid Directinn
Local-Plaid
ARr
(Giebal Mudial
Barber Pole Illusion (D)
Tourk
STD: 9666
Intogration Process
Speol
37
Optical
Optical
Pial
Grodie
Iteration
Figure 5: Psychophysical stimulus tests compare human perception and model response. (A, B) Spatial integration of ID and 2D motions. In the stimulus panels, grey arrows indicate the direction perceived by humans, while red arrows indicate the direction of physical motion. The middle panel of (B) is the heat map representing the neuron's 's connectivity from selected local region (cirele center) to other regions, with the warmer color indicating a stronger connection. (C): Fouric- pnurii-motion-basee illusion. The size of the shaded area indicates the pixel-wise standard deviation (STD) of the response. (D): Barber pole illusion. The human psychophysical data in (A), (D) were redrawn from the (88 and [59] respectively.
components 61]. can predict this reversal. In contrast, computer vision models designed to infer optical flow based on structural correspondence do not exhibit this bias.
The direction of ID motion stimuli, such as drifting Gabors, is ambiguous due the aperture problem. When presented alone, they are perceived to move orthogonal di- rections of stripes. When superimposed with other ID motion components in different orientations, 2D directions consistent with both components are perceived. Pattern motion neurons in MT may contribute to this phenomenon. The second stage of our model can explain this as well, as shown Fig. [ Psychophysies also demonstrates motion integration across space. Fig. [5] (A) shows psychophysical stimulus consisting of drifting Gabors [58]. These Gabors have variety of local directions and speeds, yet all them are consistent with one global 2D motion (downward this case). When viewed as whole, humans do perceive coherent downward motion. Stage II of our model predicts both the perceived direction and speed of the global Gabor motion.
Motion Integration:
Fig. S (B) compares spatial motion integration between ID Gabor motion (left) and 2D plaid mo- tion (right). Humans are able perceive global downward motion only in the former case: The heat maps depict how units with high activity establish long-distance connections to resolve the aperture problem when subjected
Gabor (ambiguous motion) stimuli. In contrast, the plaid stimuli (de- termined motion) suppress these long-disaance connections. In the latter case, local integration of motion signals takes priority over global integration. Once the local ambiguity resolved, the global integration process is suppressed. Our model can predict such adaptive motion pooling in human visual processing (58] Furthermore, the barber pole illusion demonstrates how locally ambiguous 1D motion is affected by the shape of the moving area (62] Specifically, as the height-width ratio of the visual area varies, human perception direction shifts from oblique to vertical [59] (Fig, [ D). Our model can predict the shift in perceived direction in stage II, showing its ability to integrate motion signals with boundary orientations. For more video demonstrations, such as the reverse phi illusion 63), please see the supplementary material.
Comparison of Complex Natural Scenes: The proposed model can effectively handle natural scenes, as demonstrated in Fig. 6 (A). Natural stimuli contain diverse spatiotemporal frequency components, leading to complex activation patterns Stage I. From the decoded flow field, the mo- tion of stage I is limited to localized areas. For example, local motion cannot be found in untextured road areas. This situation necessitates long-range interactions with the surrounding spatial context, which recurrent integration process in stage II effectively accomplishes. It is evident from the

---

on Naturalisti Secuario (A)
Flw
B122 Fllwr
Stage
Affisty adictel
repanse
Drno Humon Ground Truth (i)
9
Ineration
Figure 6: Comparison of Performance on Complex Natural Scenes. (A): Our model can estimate dense natural optical flow with the recurrent integration. (B-Dy: Similarity our model estimation to human per- ceived flow on Sintel datasct. The red circle's size in (D) shows how much closer our model is to the human response than GT.
affinity heat map (right side of subfigure A) that object and background areas are clearly segregated in stage II. This suggests that the integration mechanism based on attention has the potential to com- bine motion integration and object segmentation into single framework. These two processes are considered highly relevant in the human visual system (64]
We used psychophysically measured optical flow from the Sintel dataset [6] as benchmark for naturalistic scene flow perceived by humans. Our model was compared to several optical flow estimation methods used computer vision, including classical algorithms like Farneback, well as SOTA DNN-based models. We evaluated officially released DNN models that utilize wide range of inference structures, such as multi-scale inference graph reasoning 65), and vision transformers 15]
12 spatial recurrent models [4],
As shown in Table 1 computed both the Pearson correlation coefficients and vector endpoint emors (EPEs) between the model prediction and human response, or ground truth (GT). Additionally, examined the partial correlation between humans and models while controlling the impact of GTT:
Tresp model
Tresp GT
mmodel GT
Pruxdel
Tresp model GTT
resp GT
model GT
where is the Pearson correlation. This measure is critical in validating the models' ability to capture the characteristics of the human response, as any model could appear to have a high correlation with the human response by just approximating the GT due to high correlation between the human response and GT[[6).
Quantitatively, the proposed model outperformed all compared models in terms of partial correla- tion. The RAFT-val in the table the RAFT framework trained on our dataset as validation, and indicates that our mixed training set also improves the explanatory power of the human response. Fig. (C) shows that our model significantly improves the partial correlation across iterations in Stage II. indicating that the proposed recurrent motion integration architecture can generate more human-like deviations from GT, which trend not present similar recurrent network, RAFT. In Fig. 6 (D), one can directly see that our model prediction is more similar to human-perceived flow than the GT flow.
Table also shows the results of three biologically-inspinn models. FFVIMT (67 is model ca- pable of computing dense optical flow using direct decoding the Simoncelli & Heeger VI-MT mechanism. The other two models are modified versions of MotionNet and DorsalNet 23), re- cently proposed DNN-based models for the explanation neural responses visual motion stimuli. Since the original models are designed to recognize global motion only, we tested general multi- layer 3D CNN (a core component MotionNet and DorsalNet) with residual connections, and a pre-trained DorsalNet with frozen parameters, with linear flow decoder to compute dense flow, trained on natural dense optical flow datasets. Our model outperforms these polloiccally-inppreee models in predicting human responses. The low performances of FFV 1MT model and the modified

---

Table 1: Model v.s. Human v.s. GT. p Partial correlation between human model controlling GT; Pearson correlation coefficient; epe: vector end-point error, ut, dir, spd represent motion components Cartesian space, direction, and speed, respectively.
Method
Puv
Pdir
Pspd
v.s.
Human
V.s. GT
Tuo
Papd
Fair
epe
Fuo
Payd
Tike
cpe
Farneback [66]
0.27
0.23
0.11
0.41
0.91
0.34
2.02
0.34
0.33
0.92
1.96
FlowNet2.0f111
0.39
0.26
0.34
0.92
0,90
0.96
0,94
0.95
0.94
0,98
0.47
RAFT 14
0.20
0.22
0.14
0.92
0.90
0.96
0.93
0.98
0.99
0.99
0.25
RAFT-val
0.43
0.17
0.42
0.92
0.89
0.96
1.01
0.92
0.89
0,98
0.69
AGFlow 65
0.30
0.16
0.20
0.93
0.90
0.96
0.92
0.98
0.98
0.98
0.27
GMFlow 15
0.34
0.32
0.17
0.91
0.84
0.96
1.03
0,93
0.90
0.97
0.73
FlowFormer 52
0.36
0.14
0.32
0.93
0.91
0.95
0.90
0.98
0.97
0.98
0.42
FFVIMT
0.31
0.16
0.31
0.83
0.64
0.92
1.48
0.59
0.84
0,94
1.29
3DCNN
0.27
0.29
0.42
0.83
0.86
0.95
1.31
0.83
0.86
0.96
1.14
DorsalNet 23
0.17
0.19
-0.10
0.20
-0.08
0.86
2.35
0.20
-0.04
0.86
2.33
Ours-fixed
-0.02
0.12
0.16
0.31
0.23
0.78
2.24
0.35
0.18
0.80
2.29
Ours-Stage
0.34
0.23
0.35
0.71
0.71
0.92
1.52
0.67
0.67
0.92
1.49
Ours-Stage II
0.57
0.43
0.47
0.91
0.88
0.95
0.98
0.86
0.87
0.95
1.04
NersalNet also suggest that accurately estimating dense optical flow is challenging task, requiring specific design considerations to address complex and long-range spatial interactions, large jumps, and boundary effects, the complexities of which not adequately captured by simple mechanisms.
4 Discussion and Conclusion
DNNs have achieved impressive performance in various vision tasks, and their ability to explain the
HVS an active area of research. Recent studies have employed DNNs to model and understand the
neural mechanism of visual motion. For instance, Rideaux et al. 19 20 and Nakamura and Gomi 21 used multilayer feedforward networks, while Storrs et al. (22] used a predictive coding network (PredNet), to model biological visual motion processing, and found similarities to neurophysiologi- cal data. De Jong et al. (68] found that the spatiotemporal frequency tuning properties of some units in FlowNet resemble those found in mammalian neurons. DorsalNet uses first-person perspec- tive video stimuli to train a 3D ResNet model to predict self-motion parameters, which helps model recapitulate the neural representation of dorsal visual stream.
With similar goal in mind, we simplified the biological motion process pipeline and proposed a two-stage architecture that models the complete pathway from images, through neural representa- tions, to the perceptual response. Through end-to-end training using wide range of datasets, our model generalizes well from simple stimuli to complex natural scenes and partially captures im- portant characteristies of motioon-rocesssng neurons, including a change in spatiotemporal tuning from V1 to MT areas. To model the motion integration function, introduced a novel recurrent process based on the attention mechanism. This process successfully explains a wide range of phys- iological findings (c.g., a change in the population of component and pattern cells from V1 to MT) and psychophysical findings (e.g., global motion pooling). It also improves the partial correlation with human psychophysical response. The success of the attention mechanism in motion integration could be attributed to its similarity to the human visual grouping mechanism [69] or similar feature grouping/binding that may be accomplished using top-down attentional selection mechanism [70).
While we show that the attention-based recurrent network promising computational model of human visual motion grouping and segmentation, how its complex architecture (where all responses can be influenced by all other responses) is actually implemented in the human brain remains an open question. Another limitation of our current model is that it does not process several important abilitics of human motion perception, including non-Fourier (sccond-order) motion detection [61], and motion integration sensitive to surface layout/7[. Our model does not take into account the benefits of biological processing, such as energy efficiency.
In conclusion, combining classical motion energy and advanced deep learning technology is promising approach to bridge the gap between human and DNN motion perception systems. Our proposed architecture and recurrent process offer insights into the underlying mechanisms of motion perception and open up new avenues for future research.
10

---

Acknowledgement
This work was supported in part by JST, the establishment of university fellowships towards the creation of science technology innovation, Grant Number JPMJFS2123; in part by the JSPS Grants- in-Aid for Scientific Research (KAKENHI), Grant Numbers JP20H00603 and JP20H05957.
References
[1] J. Gibson, "The perception of the visual world.," 1950.
[2] L. M. Vaina, S. A. Beardsley, and S. K. Rushton, Optic flow and beyond, vol. 324. Springer Science & Business Media, 2004.
[3] D. H. Hubel and T. N. Wiesel, "Receptive fields of single neurones in the cat's striate cortex," The Journal of physiology, vol. 148, no. 3, p.p. 574, 1959.
[4] D. H. Hubel and T. N. Wiesel, "Receptive fields and functional architecture in two nonstriate visual areas (18 and 19) of the cat," Journal of neurophysiology, vol. 28, no. 2, pp. 229-289, 1965.
[5] C. D. Salzman, C. M. Murasugi, K. H. Britten, and W. T. Newsome, "Microstimulation in visual area mt: effects on direction discrimination performance," Journal of Neuroscience, vol. 12, no. 6, pp. 2331-2355, 1992.
[6] Celebrini and W. Newsome, "Microstimulation of extrastriate area mst influences perfor- mance on a direction discrimination task," Journal of Neurophysiology, vol. 73, no. 2, pp. 437- 448, 1995.
[7] C. C. Pack and R. Т. Born, "Temporal dynamics of a neural solution to the aperture problem in visual area of macaque brain," Nature, vol. 409, no. 6823, pp. 1040-1042, 2001.
[8] E. Simoncelli and J. Heeger, model of neuronal responses in visual area mt," Vision research, vol. 38, no. pp. 743-761, 1998.
[9] Y. Weiss, E. P. Simoncelli, and E. H. Adelson, "Motion illusions as optimal percepts," Nature neuroscience, vol. 5, no. 6, pp. 598-604, 2002.
[10] C. Pack and R. Born, "Cortical mechanisms for the integration of visual motion," 2008.
[11] A. Dosovitskiy, P. Fischer, E. Ilg, P. Hausser, C. Hazirbas, V. Golkov, P. Van Der Smagt, D. Cremers, and T. Brox, "Flownet: Learning optical flow with convolutional networks." in Proceedings the IEEE international conference on computer vision, pp. 2758 278827766,22 2015.
[12] E. 1lg, N. Mayer, Saikia, M. Keuper, Dosovitskiy, and Brox, "Flownet 2.0: Evolution of optical flow estimation with deep networks." in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2462-2470, 2017.
[13] D. Sun, X. Yang, M.-Y. Liu, and J. Kautz, "Pwe-net: Cnns for optical flow using pyramid, warping, and cost volume." in Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8934-8943, 2018.
[14] Z. Teed and Deng, "Raft: Recurrent all-pairs field transforms for optical flow," in European conference on computer vision, pp. 402-419, Springer, 2020.
[15] H. Xu, J. Zhang, Cai, H. Rezatofighi, and D. Tao, "Gmflow: Learning optical flow via global matching," in Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 8121-8130, 2022.
[16] Y.-H. Yang. Т. Fukiage, Z. Sun, and S. Nishida, "Psychophysical measurement of perceived motion flow of naturalistic scenes," iScience in press, 2023.
[17] D. Yamins and J. J. DiCarlo, "Using goal-driven deep learning models to understand sensory cortex," Nature neuroscience, vol. 19, no. 3, pp.
356-56336666 356-365, 2016.

---

[18] Z. Sun, Y.J. Chen, Y.-H. Yang, and S. Nishida, "Comparative analysis of visual motion per- ception: Computer vision models versus human vision," in Conference on Cognitive Compu- lalional Neuroscience, (Oxford, UK), August 24-27 2023.
[19] R. Rideaux and A. E. Welchman, "But still it moves: static image statistics underlie how we see motion," " Journal of Neuroscience, vol. 40, no. 12. pp. 2538 2552, 2020.
[20] R. Rideaux and A. E. Welchman, "Exploring and explaining properties of motion processing in biological brains using neural network," Journal of Vision, vol. 21, no. 2, pp. 11-11, 2021.
[21] D. Nakamura and H. Gomi, "Decoding self- oeff-motion from visual image sequence predicts dis- tinctive features of reflexive motor responses to visual motion," Neural Networks, vol. 162, pp. 0p.516-330 2023.
[22] K. Storrs, O. Kampman, R. Rideaux. G. Maiello, and R. Fleming, "Properties of v1 and mt mo- tion tuning emerge from unsupervised predictive learning." Journal of Vision, vol. 22, no. 14, pp. 4415-4415, 2022.
[23] P. Mineault, S. Bakhtiari, B. Richards, and C. Pack, "Your head is there to move you around: Goal-driven models of the primate dorsal pathway," Advances in Neural Information Process- ing Systems, vol. 34, pp. 28757-28771, 2021.
[24] E. H. Adelson and J. R. Bergen, "Spatiotemporal energy models for the perception of motion," Josa a, vol. 2, no. 2. pp. 299, 1985.
[25] A. Perrone, "A visual motion sensor based on the properties of vl and mt neurons," Vision research, vol. 44, no. 15, pp. 1733-1755, 2004.
[26] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, E. Kaiser, and I. Polo- sukhin, "Attention all you need," Advances neural information processing systems, vol. 30, 2017.
[27] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. hghani, M. Minderer, G. Heigold, S. Gelly, et al., "An image is worth 16x 16 words: Trans- formers for image recognition at scale," arXiv preprint arXiv: 11999, 2020.
Zhai, T. Unterthiner, M. De-
[28] J. A. Perrone and A. Thiele, "Speed skills: measuring the visual speed analyzing properties of primate mt neurons," Nature neuroscience, vol. 4, no. 5, pp. 526-532, 2001.
[29] Movshon, E. Adelson, M. Gizzi, and W. T. Newsome, "The analysis of moving visual pat- terns," in Frontiers in cognitive neuroscience, MIT Press, 1992.
[30] J. P. Jones, A. Stepnoski, and L. A. Palmer, "The two-dimensional spectral structure of simple receptive fields cat striate cortex," Journal of Neurophysiology, vol. 58, no. 6, pp. 1212- 1232, 1987.
[31] P. Jones and L. A. Palmer, "An evaluation of the two-dimensional gabor filter model of simple receptive fields cat striate cortex," Journal of neurophysiology. vol. 58, no. pp. 1233-1258, 1987.
[32] R. N. Bracewell and R. N. Bracewell, The Fourier transform and its applications, vol. 31999. McGraw New York, 1986.
[33] D. Sun, X. Yang, M.-Y. Liu, and J. Kautz, "Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume," in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8934-8943, 2018
[34] J.-Y. Bouguet el al., "Pyramidal implementation of the affine lucas kanade feature tracker description of the algorithm."" Intel corporation, vol. 5, no. 1-10, p. 4, 2001.
[35] E. Castet and J. Zanker, "Long-range interactions in the spatial integration of motion signals.." Spatial Vision, vol. 12, no. 3, pp. 287-307, 1999.
[36] D. J. Heeger, "Modeling simple-cell direction selectivity with normalized, half-squared, linear operators," Journal of newrophysiolog, vol. 70, no. 5, pp. 1885-1898, 1993.
12

---

[37] M. Carandini and D. Heeger, "Summation and division by neurons in primate visual cortex," Science, vol. 264, no. 5163. pp. 1333-1336, 1994.
[38] S. Gilaic-Dotan, "Visual motion serves but is not under the purview of the dorsal pathway," Neuropsychologia, vol. 89, pp. 378-392, 2016.
[39] A. Noest and A. Van Den Berg, "The role of early mechanisms in motion transparency and coherence.." Spatial Vision, vol. 7, no. pp. 125-147, 1993.
[40] J. Lorenceau and M. Shiffrar, "The influence of terminators motion integration across space," Vision research, vol. 32, no. 2, pp. 263-273, 1992.
[41] Y. Wei, H. Xiao, H. Shi, Z. Jie, J. Feng, and T. S. Huang, "Revisiting dilated convolution: A simple approach for weakly-and remi-supervised semantic segmentation." in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7268-7277, 7277,20 2018.
[42] X. Wang, R. Girshick, A. Gupta, and K. He, "Non-local neural networks," " in Proceedings of the IEEE conference computer vision and pattern recognition, pp. 7794-7803, 2018.
[43] T. Serre, "Deep learning: the good, the bad, and the ugly," Annual review of vision science, vol. 5, pp. 399-426, 2019.
[44] N. Kipf and M. Welling, "Semi supervisd classification with graph convolutional net- works," arXiv preprint (609.26090029000 2016.
[45] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, "Empirical evaluation of gated recurrent neural networks on sequence modeling," arXiv preprint arXiv: #xXvv:1422.3555, 2014.
[46] L. Liu, J. Zhang, R. He, Y. Liu, Y. Wang, Y. Tai, D. Luo, C. Wang, J. Li, and Huang, "Learning by analogy: Reliable supervision from transformations for unsupervised optical flow estimation," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6489-644 6498,, 2020.
[47] R. Jonschkowski, A. Stone, J. T. Barron, A. Gordon, K. Konolige, and A. Angelova, "What matters in unsupervised optical flow," in Computer Vision-ECCV 2020: 16th European Con- ference, Glasgow, UK, August 23 23-28, 2020, Proceedings, Part Il 16, pp. 557-572, Springer, 2020.
[48] A. Stone, D. Maurer, A. Ayvaci, A. Angelova, and R. Jonschkowski, "Smurf: Self-teaching multi-frame unsupervised raft with full-image warping," in Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 3887-3896, 2021.
[49] S. Meister, J. Hur, and S. Roth, "Unflow: Unsupervised learning of optical flow with bidirec- tional census loss," in Proceedings of the AAAI conference on artificial intelligence, vol. 32, 2018.
[50] D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black, "A naturalistic open source movie for optical flow evaluation," " in European Conf. on Computer Vision (ECCV) (A. Fitzgibhon et al. (Eds.), ed.). Part IV, LNCS 7577, pp. 61 1--55, Springer- Verlag, Oct. 2012.
[51] F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine- Hornung, "A benchmark dataset and evaluation methodology for video object segmentation," in Computer Vision and Pattern Recognition, 2016.
[52] Z. Huang, X. Shi, C. Zhang, Q. Wang, K. C. Cheung, H. Qin, J. Dai, and H. Li, "Flowformer: transformer architecture for optical flow." in Computer Vision-ECCV 2022: 17th Euro- pean Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVII, pp. 668-685, Springer, 2022.
[53] N. Priebe, S. G. Lisberger, and J. A. Movshon, "Tuning for spatiotemporal frequency and speed in directionally selective neurons of macaque striate cortex," " Journal of Neuroscience, vol. 26, no. 11, pp. 2941-2950, 2006.
[54] F. LeDue, M. Zou, and N. Crowder, "Spatiotemporal tuning mouse primary visual cortex," Neuroscience letters, vol. 528, no. 2, pp- 165-169, 2012.
13

---

[55] N. J. Priebe, C. R. Cassanello, and S. G. Lisberger, "The neural representation of speed in macaque area mt/v5," Journal of Neuroscience, vol. 23, no. 13, pp. 5650-5661, 2003,
[56] Yosinski, Clune, A. Nguyen, Fuchs, and H. Lipson, "Understanding neural networks through deep visualization," arXiv preprini arXiv:/506.06577 2015.
[57] N. J. Priebe, C. R. Cassanello, and S. G. Lisberger, "The neural representation of speed in macaque area mt/v5," Journal of Newroscience, vol. 23, no. 13, pp. (650-5661, 2003.
[58) K. Amano, M. Edwards, D. R. Badcock, and S. Nishida, "Adaptive pooling of visual motion signals by the human visual system revealed with a novel multi-element stimulus," Journal of vision, vol. 9, no. 3, pp. 4-4, 2009.
[59] N. Fisher and J. M. Zanker, "The directional tuning of the barber-poll illusion." " Perception, vol. 30, no. 11, pp. 1321-1336, 2001.
[60] R. O. Brown and S. He, "Visual motion of missing-fundamenta patterns: motion energy versus feature correspondence," Vision Research, vol. 40, no. 16. pp. 2135-2141 2000.
[61] C. Chubb and G. Sperling, "Drift-balanced random stimuli: general basis for studying non- fourier motion perception," JOSA A, vol. 5, no. 11, pp. 1986-2007, 1988.
[62] P. Sun, C. Chubb, and G. Sperling, "Two mechanisms that determine the barber-pole illusion," Vision research, vol. 111, pp. 43-54, 2015.
[63] S. M. Anstis and J. Rogers, "Illusory continuous motion from oscillating gositive-negative patterns: Implications for motion perception," Perception, vol. 15, no. 5, pp. 627-640, 1986.
[64] T. Handa and A. Mikami, "Neuronal correlates of motion-defined shape perception in primate dorsal and ventral streams," " Eurupean journal of Neuroscience, vol. 48, no. 10, pp. 3171-3185, 2018.
[65] A. Luo, F. Yang, K. Luo, X. Li, H. Fan, and S. Liu, "Learning optical flow with adaptive graph reasoning," in Proceedingy of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 1890-1898, 2022.
[66] G. Farnebäck, "Two-frame motion estimation based on polynomial expansion," " in Image Anal- ysis: 13th Scandinavian Conference, SCIA 2003 Halmstad, Sweden, June 29-July 2, 2003 Proceedings 13, pp. 363-370, Springer, 2003.
[67] Solari, M. Chessa, N. K. Medathati, and Kornprobst, "What can we expect from a vl-mt feedforward architecture for optical flow estimation?," Signal Processing: Image Communica- tion, vol. 39, pp. 342-354, 2015.
[68] D. B. de Jong, F. Paredes-Vallee and G. C. de Croon, "How do neural networks estimate optical flow? Machine Intelligence, vol. 44, no. 11, pp. 8290-8305, 2021.
hwwpppssscooooooospppeeeee study," IEEE Transactions on Pattern Analysis and
[69] P. Mehrani and J. K. Tsotsos, "Self-attention in vision transformers performs perceptual group- ing, not attention," arXiv preprint arXiv:2303.01542. 2023.
[70] K. Tsotsos, Y. Liu, J. C. Curtinez-Trujillee M. Pomplun, E. Simine, and K. Zhou, "Attending to visual motion," Computer Vision and Image Understanding, vol. 100, no. 1-2. pp. 3-40, 2005.
[71] McDermott, Y. Weiss, and E. H. Adelson, "Beyond junctions: nonlocal form constraints on motion interpretation." Perception, vol. 30, no. 8, pp. 905-923--3 2001.
14


Reviews:
[{"review_id": "ZTUMQXA84o", "reviewer": "Reviewer_tFBw", "summary": "This paper proposes a new V1-MT model using a normalized Gabor model of V1, followed by a recurrent self-attention stage. It uses dense optic flow as a supervised objective. The authors perform extensive in silico neurophysiology to show the model units qualitatively look like V1 and MT. They also show that this model is a better match to human visual perception on natural scenes than computer-vision models.", "questions": "-", "limitations": "-", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Well-written and clear\n- Breadth of in silico experiments\n- The model makes sense and I’m excited about the idea of segregation vs. integration as an explanation for complex receptive fields in MT", "weaknesses": "- Pretty incremental in a crowded field\n- Little comparison to SOTA\n- Mostly qualitative and little quantification\n\nI really want to like this paper: full disclosure, I’m a big fan of the work of Orban et al. and more recently Cui et al. (2013) that shows that MT cells have complex receptive fields capable of integration and segregation. However, this paper kind of rubbed me the wrong way by ignoring the state-of-the-art in this field: “Despite extensive research in cognitive neuroscience, image-computable models that can extract informative motion flow from natural scenes in a manner consistent with human visual processing have yet to be established”. There are plenty of image-computable models that can extract information relevant to dense optic flow from natural scenes. Everything from the old work of Simoncelli and Heeger to the receptive field models empirically derived from MT of Nishimoto and Gallant (2011) to MotionNet from Rideaux et al. and especially to Mineault et al. (2021), inexplicably uncited in this manuscript despite being published in these very pages and having very similar motivation to this manuscript. The onus is on the manuscript to show us some failings of these previous models and how the model does better according to some axis. \n\nTo be fair, the paper could say that some of these networks–especially generic 3d CNNs without multi-scale representations and explicitly organized direction and speed tuning–do not extract quantitative optic flow information *explicitly*. However, consider this thought experiment: if I wanted to read information from a patch of MT, as a psychophysical observer presumably needs to do to solve the tasks in the paper, I wouldn’t have access to a neat organization of direction and speed tuning, either; I would need to do a readout on top of the patch. I don’t buy the premise that the brain needs to form an explicit estimate of dense optic flow at every point in space, e.g. in MT. I think it’s easy enough to turn these old models into ones that estimate dense optic flow using a linear readout, and these should be compared to the proposed model.\n\nThe paper spends a lot of time rehashing the same kind of qualitative receptive field exploration that’s a hallmark of this field: tuning curves, component vs. pattern, speed tuning curves, distribution of preferences for direction tuning, barber poles, reverse phi, etc. This is a very crowded field between all the papers from Rideaux, Welchman, Fleming, Mineault, Bakhtiari and Pack: these kinds of demonstrations have been done over and over again. It’s nice to have it in the paper but I really consider these a sanity check, not a finding, especially since this paper bakes in Gabor receptive fields in the first layer: how could they not learn direction selectivity? My advice, speed run through these to give more space for the end of the paper, which is where things get interesting.\n\nDefinitely the most interesting thing about the paper, in my view, is Table 1. It shows that the network knows something about how human motion estimation operates that is not captured by CV models. It’s a bit buried in the paper and explained a bit fast. The authors should add ablations to figure out what it is about this network that makes it more brain-like - the Gabors? the normalization? the attention mechanism? I also think they need to add in quantitative comparisons with MotionNet from Rideaux et al. and DorsalNet from Mineault et al. with a linear decoder on top–to be clear, I’d be fine with a linear decoder trained on another task. You could avoid doing a linear decoder by using an RSA-based analysis, or you could use other methods of alignment which are more restrictive than a linear decoder (e.g. from Alex Williams et al.), if you believe this is not an apples-to-apples comparison. \n\nOverall, I think this could be a valuable contribution to the field, but it needs to be very explicit about its specific contribution in light of plentiful previous work, and it needs explicit comparisons to this previous work. I would be happy to accept granted the authors cite and address previous literature and include quantitative comparisons to MotionNet and DorsalNet, provided their model comes out on top either according to the metrics they have in Table 1 and Figure 5 or some other relevant metric they find.\n\nNitpicks:\n\n- I liked the convention of using red color for trainable parameters, but the authors only use that once and that drop that later. Would recommend doing it consistently throughout the methods.\n- Bottom of Page 4: “Top side of Fig. 3” → Should be a reference to figure 1. There are a couple more instances of this, so the authors should go carefully through the manuscript to find if there are more instances of that.\n- The model is fairly bespoke but it’s not particularly biologically plausible with its attention mechanism. If the point is to implement recurrence, why not just use a plain transformer with tied weights at every layer instead? To be clear, the authors don’t have to fit this model, just a sentence or two to justify why they picked this architecture rather than something more off-the-shelf.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for thoroughly reading our research and comprehensive comments.\nFirstly, we appreciate the reviewer's nitpicks and will make the suggested modifications.\n\n---\n\n**General Response:**\n- We should clearly state the purpose of our study. From a scientific perspective, we aim to explain a wide range of psychophysical phenomena, including those whose physiological mechanisms are not yet clear, while aligning the model's internal representations with physiological ones. Engineering-wise, we aim to make a human-aligned model that maintains competitive performance with SOTA CV models. In contrast, the purpose of MotionNet and DorsalNet was to explain neural responses to visual motion stimuli. They are good models for this purpose, but they do not give a dense optical flow. We cannot compare the outputs of these models with human motion perception, our model, and CV models.\n\n- We do not believe in an explicit representation of dense optical flow in the brain. However, given that it is possible to reproduce a high-resolution flow from human responses (Yang et al., 2023) and that many motion phenomena include interactions of local motion signals, constructing a model like ours would be useful for a computational understanding of motion perception.\n\n- The reviewer suggested that MotionNet/DorsalNet with a simple linear encoder to compute a dense optical flow might outperform our model. Accurate dense optic flow estimation is a difficult task that requires complex and long-range spatial interactions to tackle large jumps and the boundary effect. FlowNet (Dosovitskiy et al., 2015), the first optical flow model to use a multi-layer stacked CNN, has more than ten times the parameters of MotionNet but still lags behind SOTA models in Table I. The CV SOTA models employ many sophisticated strategies to match GT and better predict human responses. In the attached PDF, we provide additional results obtained with three relevant models: a pre-trained DorsalNet with a linear flow decoder, a general 3D CNN (basic structure for MotionNet and DorsalNet), and FFV1MT (Solari et al.,2015), the only model known to calculate a dense optical flow with simple decoding of the Simoncelli & Heeger V1-MT mechanism. These models fall short in terms of dense optical estimation accuracy or fail to account for the global integration of local motions. This outcome suggests that a simple linear decoder cannot handle dense optical flow, particularly in complex real scenes. Furthermore, in addition to flexible spatial pooling and accurate estimation of dense optical flow, our model displays consistent human-perceived Fourier motion in missing fundamental stimuli by motion energy computation simulating the V1 neurons. Comparison with other models shows how difficult it is to achieve such diverse capabilities **simultaneously** in a single model. \n \n- We understand the concerns regarding our model being 'over-designed' compared to general 3-D CNNs, and we provide more explanation in the appendix.\n             \n- It is not a simple sanity check to show that our model achieves internal representations similar to MT concerning component/pattern motion and spectral receptive field, since stage II of our model is a complex recurrent self-attention network. RSA-based model comparison is an interesting idea to test in the future, but in this study, we do not propose our model outperforms MotionNet and DorsalNet in explanation of MT/MST responses to specific types of motion stimuli used previously in the labs. Our preliminary analysis suggests that our model best explains human-perceived flow in naturalistic scenes because the attention network can do something similar to vector decomposition (Johansson, 1973).\n\n- The neural implementation of the attention mechanism in our model design might look too complicated and biologically implausible. However, substantial psychophysical evidence necessitated a function for flexible spatial information integration, which the attention mechanism and graph structure capably deliver. Simple convolution stacking may be able to yield similar outcomes but will make the model more complex and biologically uninterpretable.\n\nWe appreciate the reviewer's insightful feedback. In the revised manuscript, we will clarify how our study differs from previous studies, especially DorsalNet and MotionNet, and make modifications based on the reviewer's minor comments.\n\nBest, Authors\n\n ---\n\n**Appendix:**\n\nTo model a function that allows flexible integration of spatial information, we proposed the attention mechanism and graph structure to accomplish this effectively. Stacking simple convolutions could achieve similar results but is restrictive, setting an upper limit on long-range spatial relationships. For instance, the large 2K resolution Sintel dataset we used for comparison with humans requires hundreds of (3x3) convolution layers for single-time information propagation (Zhao et al., 2017). Stacking such many layers is unreasonable, leading to model bloating, limited long-distance interaction, and convergence issues. In contrast, our model enables graph topology, and the attention mechanism allows global interaction with one-layer operation. MotionNet's simplicity is due to its limited testing stimulus of 32x32, which is manageable with a few CNN layers. However, this approach isn't suitable for larger images due to physical long-distance information limits. Plus, with too many CNN layers, it becomes challenging to comprehend the exact functionality of each layer.  Instead, our model is designed for clarity of function: the first layer extracts local energy, while the second layer manages motion integration and segregation.\nOur use of the attention mechanism also offers the possibility of understanding motion integration by visualizing the adjacency matrix, as shown in Appendix Fig. 1 of our supplementary materials. This level of understanding would not be achievable by simply stacking CNN layers. \n"}, {"review_id": "FW8bJvrKsA", "reviewer": "Reviewer_n3Ed", "summary": "This paper proposes a novel model of motion analysis. It takes inspiration from biological motion processing to try and solve the aperture problem, leveraging a combination of biologically inspired constrained structure with learned parameters. This produces strong partial correlations of motion components in empirical tests, but also interesting emergent phenomenon when investigating the distribution of unit responses between the different network components.", "questions": "- In the caption for Table 1 it states that uv = Cartesian space, dir = speed, and spd = direction. I assume dir should be direction and spd should be speed?", "limitations": "Limitations seem to be adequately discussed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This paper does a good job of anchoring model development with a strong motivation and justifications based on biological understanding of motion processing. This is not an easy task, and the paper does a good job of finding compromise between the performance-driven approach of data fitting with the desire for an explanatory model that provides insight into why a certain performance or behaviour is achieved.\n\n- The paper is quite dense with a wide range of ways to explore the problem and the model. This is both a strength and a weakness, as it sometimes makes the paper difficult to follow, but the range of analysis is impressive.", "weaknesses": "- Missing reference: Tsotsos et al., \"Attending to visual motion\", CVIU 2005\n  -- This paper provides a biologically motivated model of motion including attentional effects and structure informed by our understanding of visual areas V1, MT, MST, and 7a in the primate visual system.\n\n- Figure 1 is rather confusing; there's a lot going on, and I found flow of information rather unclear. If possible I would recommend breaking it up into separate figures or finding a way to show how the different components connect in a clearer manner. For example, is C a sub-component of A, or just some example receptive fields? If the latter, why is it part of the network diagram? Similarly, in the text F is given as having dimensions HWx256 (Line 140), but in the RMIB section of Figure 1 it looks like it is supposed to have HxWxC dimensions. Is this a different F? This was not clear.\n\n- The partial correlation metric should be clearly defined; I did not see a definition for this in the paper. It would be better to have more discussion of the aspects that the model does not excel at compared to competing models (e.g. FlowFormer for v.s. Human and RAFT for vs. GT in Table 1).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThanks for the constructive feedback from the reviewer. \n\n -----\n\n**Missing reference**:\n\nWe will include the recommended reference: Tsotsos et al., \"Attending to visual motion,\" CVIU 2005, in the introduction and discussion part.\n \n**Clarification of Figure 1**:\n\nWe understand that Figure 1 seems overcrowded and potentially confusing. We will work on refining it for improved clarity and coherence.\n \nAs for the reviewer's specific questions, 'Fig. 1C' is indeed a sub-component of 'Fig. 1A'. We designed the model such that there are 256 different motion energy units in Stage I and 'Fig. 1C' visualizes one of these units. This unit consists of a quadrature pair of spatial and temporal filters, forming a spatiotemporal slanted receptive field. We will make this relationship clearer in the revised manuscript.\n \nAs for the dimensions of F, we apologize for the confusion. The correct description should be $F \\in \\mathbb{R}^{H×W×C}$ where $C = 256.$ (Line 140)\n \n*Here we provide some additional context:*\n\nIn our model, C represents the number of channels in a given feature map, which in this case is 256. Hence, we have $H×W×256$. This is equivalent to $HW×256$; the only difference lies in the shape in which we represent the same quantity. When we say $F \\in \\mathbb{R}^{H×W×C}$, we refer to a 3D space. On the other hand, $HW×C$ represents the same set of elements but flattened along one dimension—akin to reshaping a 2D image into a 1D sequence. This change in representation is primarily for the convenience of matrix product calculation to get the adjacency matrix. While this convention is commonly used when defining tensor shapes in the context of neural networks, we acknowledge that it may not be mathematically rigorous.\n \nWe will include these clarifications in our revised manuscript.\n \n \n**The partial correlation metric should be clearly defined; I did not see a definition for this in the paper. It would be better to have more discussion of the aspects that the model does not excel at compared to competing models (e.g., FlowFormer for v.s. Human and RAFT for vs. GT in Table 1).**\n \nThe partial correlation we used is defined as:\n\n$$\n\\rho_{\\text {model }}=r_{\\text {resp model } \\cdot G T}=\n\\frac{\nr_{\\text {resp model }}-r_{\\text {resp } G T}\\cdot r_{\\text {model } G T}} { \\sqrt{1-r^2_{\\text {resp } G T}} \\sqrt{1-r^2_{\\text {model } G T}}}\n$$\n\nwhere $r$ is the Pearson correlation. Due to page limitations, we briefly mentioned the partial correlation from Line 283 but did not detail the specific equation definition. To ensure clarity, we will provide the definition of partial correlation in our supplementary material.\n\nHere, we would like to explain the reason for using partial correlation, combining the question of \"*FlowFormer for v.s. Human and RAFT for vs. GT.*\"\n \nWe recognize that our model's optical flow prediction performance may not be as high as other SOTA models. However, it's important to note that our main goal is to model human motion perception. Therefore, we prioritize biological plausibility over pure performance.\n\nThe SOTA models are intentionally designed to achieve the best performance in matching ground truth (GT) data. In contrast, we incorporate additional constraints on our current models to trade off biological plausibility, such as motion energy computation. Hence, it is not appropriate to rely solely on a simple correlation to GT as an index for comparing the capabilities of our model with other SOTA models.\n \n\nOn the other hand, we believe that considering the correspondence of the model prediction to human response provides a fairer index for comparing models. However, previous psychophysical studies have highlighted that human response is inherently highly correlated with GT [ref]. This correlation might lead to the observation that any models trained to fit the GT are also highly related to human response, regardless of the model design. Following the previous study's strategy, to address this potential confusion between human-model correspondence (true evaluation) and human-GT correspondence (confounding), we employ partial correlation, controlling of GT, to purify the true model-human correspondence. More intuitively, we exclude the covariance between GT and the model prediction and also between the GT and human response. Therefore the partial correlation we used is only related to how strong model prediction could explain the variance in human response, which turns out to be a pure correlation between humans and the model.\n \nBased on the partial correlation metric evaluation, our model surpasses the state-of-the-art models listed in Table I. In addition, the pure EPE data proves that our model closely matches human responses in complex natural environments, whereas other state-of-the-art models tend to align more with the ground truth.\n \nWe hope the above explanation provides you with a better understanding.\nWe will detail these points further in our revised manuscript.\n \n \n**Caption for Table 1:**\n\nThanks for noticing this mistake. 'dir' should indeed denote 'direction,' and 'spd' should denote 'speed.' We will rewrite the caption to:\" uv, dir, spd represent motion components in Cartesian space, direction, and speed, respectively. \"\n\n\n[ref]\n\n- Yang, Y. H., Fukiage, T., Sun, Z., & Nishida, S. Y. (2023). Psychophysical measurement of perceived motion flow of naturalistic scenes. Available at SSRN 4414877\n\n ----\n\nWe are appreciative of your time and effort spent reviewing our work and giving us valuable insights.\n \nBest,\nAuthors"}, {"review_id": "rNDY4fowql", "reviewer": "Reviewer_gk6R", "summary": "I have read the authors' rebuttal and will maintain my already high rating.\n\nThis paper proposes a new model of the dorsal pathway (V1->MT) using a two-stage architecture. The first stage uses spatiotemporal filters tuned by supervised learning, while the second stage uses a dynamic connection between motion detectors based on the similarity of motion responses. The model is able to capture several important aspects of human motion processing, including both neurophysiological properties and psychophysical responses.", "questions": "1. What do the script symbols in Equation 3 represent? They aren’t defined.\n\n\nLine 67: mode’s -> model’s\n\n123: ability to capture local motion only ->\n        ability to capture only local motion.\n\n132: process -> processing\n\nbottom of page 4: At this point, it would be great to give an intuitive description of what this adjacency matrix represents. It looks to me like this strongly and globally  connects neurons with similar motion responses, which would make intuitive sense, but since it is dynamically computed, it isn’t obvious how this would be implemented neurally.\n\n184: our primary lies -> our primary goal lies \n\nIt’s unclear what’s being shown in Figure 3D, especially the graphs in the first row on the right (is this still part of D?). Presumably, lines 227-231 are describing what this is, but I don’t understand the explanation.\n\nline 251 refers to figures 3A and 4A. Are both supposed to be 4A?\n\nAlso, panel E should be to the left of panel F. \n\nFigure 4B middle panels. What are these depicting?", "limitations": "Barber pole illusion is not quite captured properly (Figure 4D), although the end points are.\n\nThe neural implementation of this model is unclear. This is mentioned as a limitation.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "-The paper is very well written, although there are a few places where it is not clear. \n\n-The model is quite convincing in its fits to the data, both neurophysiological and psychophysical. \n\n-The models fits to data are compared to 8 different SOTA models of optical flow, and when the correlation to the ground truth is factored out, the model is superior to all others (by having a high correlation to the ground truth motion, the models will necessarily\thave a high correlation with the human data). This is an important test.\n\n-The model can not only account for responses to low-level stimuli (drifting gabors), but can also give convincing responses to natural and artificial movies. I am not an expert in motion processing, but to my knowledge, this is the first model to do this. Perhaps other reviewers will know of other models with this capability.\n\n-The model is a sophisticated one, using a dynamic graph construction that connects the motion detectors in stage 2, integrating the motion detectors in stage 1 based on similarity of response. In this way, the model can capture global motion, solving the aperture problem.", "weaknesses": "-The model’s sophistication is also a weakness: The dynamic graph construction assumes all responses can be influenced by all other responses, depending on the response similarity. That is, dynamically, any response integrator can be connected to any other, no matter how far away spatially. This is what allows the network to integrate the local motion into global motion. How this could be implemented neurally is very unclear. However, this is an argument from lack of imagination. Also, the paper acknowledges this limitation.\n\n-Some of the presentation is unclear/unconventional: For example, figure captions don’t label all components of the figure (e.g., the caption to Figure 5 only describes 5D), although they may be discussed in the text. It is not always clear what is being shown in the Figures, which are relatively complex.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n\nThanks for the reviewer's appreciative comments regarding our work, and I'd like to address the reviewer's concerns as follows.\n\n---\n\n**On the neural implementation of dynamic graph construction**:\n\nWe admit that the specific neural implementation of the attention mechanism is still unclear. This is a complex neuroscientific issue and one of the limitations we acknowledge in our paper. However, numerous psychophysical findings suggest the presence of global motion integration capabilities. For instance, human visual motion computation exhibits substantial interactions over vast spatial separations (refer to Maruya, K., Holcombe, A. O., & Nishida, S., 2013). To simulate this, we required a function capable of flexibly integrating spatial information, which is what the attention mechanism and graph topology effectively provide.\nAs the reviewer correctly pointed out, the current version of our model assumes, dynamically, any response integrator can be connected to any other, no matter how far away spatially. This may sound physiologically plausible. However, we could incorporate position embedding to mirror retinal topological information when calculating the adjacency matrix. Consequently, as illustrated in our supplementary material (Fig. 1.2.3). The visualized connections primarily focus on the local area of the selected regions. \nHowever, considering that human visual motion computation exhibits substantial interactions over 100 deg (see, e.g., Maruya, Holcombe & Nishida, 2013), designing biologically plausible position embedding is not an easy task. It is possible that the neural implementation for this computation is different and more efficient, and we will acknowledge this limitation in the manuscript.\n\n\n**Addressing presentation clarity**:\n\nWe apologize for any confusion stemming from our figures and captions. Our revised manuscript aims to enhance figure clarity and provide more detailed captions to explain each component thoroughly.\n\n**Questions**:\n\n- **Q1: What do the script symbols in Equation 3 represent?**\n\nA1: We apologize for the lack of explanation. The symbols $\\Re$ and $\\Im$ extract the real and imaginary parts of a complex number, while $*$ denotes convolution operations. We will add these definitions in our revised manuscript for clarity.\n\n- **Q2: some typo errors**\n\nA2: Thank you very much for the careful check and correction. We will correct them in our revised manuscript.\n\n- **Q3: bottom of page 4: give an intuitive description of what this adjacency matrix represents**\n\nA3: Thanks for your suggestion. Intuitively, this adjacency matrix represents the neuron's affinity or connectivity within the space. We will provide a more in-depth explanation in the revised manuscript.\n\n- **Q4. It's unclear what's being shown in Figure 3D, especially the graphs in the first row on the right (is this still part of D?).**\n\nFirstly, considering the clarity, we consider relocating the graphs in the first row to the supplementary material or appendix.\n\n Fig. 3D and its accompanying content are designed to investigate the spectral properties of the units in our model. To accomplish this, we subjected the model to a combination of drifting-Gabor stimuli with varying spatiotemporal frequency components, which we can interpret as testing the spectral receptive field of each unit.\n\nTo analyze these receptive fields, we employed a 2D Gaussian profile fitting. This mathematical approach provides key parameters such as the central location and oblique angle of the fitted profile. These parameters allow for a quantitative examination of the spectral field distribution for each neuron. The upper-right section of Fig 3D demonstrates the distribution of these fitted Gaussian profiles. Each small dot corresponds to the central location of a unit's receptive field in the frequency domain. The slanted bar associated with each dot signifies the oblique angle of the spectral receptive field for the corresponding neuron. The length of the bars represents the eccentricity of the receptive field. This collection of oblique angles is further visualized as a bar plot in Figure 4E.\n\nHowever, we want to stress that Fig 3D primarily serves to introduce Figure 4E. It does not aim to advance any argument. Please refer to the supplementary material (Line 117) for more details.\n\n- **Q5. line 251 refers to figures 3A and 4A...**\n\nThanks for your correction! In line 251, we changed 'Fig.3(A)' to 'Fig. 4'.\nWe will also adjust the arrangement of panels E and F.\n\n- **Q6. Figure 4B middle panels...**\n\nThe middle panels of Figure 4B aim to represent the neuron's connectivity by visualizing the adjacency matrix. The heat map indicates the neuron's connectivity (cosine similarity in the adjacency matrix) from a selected local region to other regions. The warmer the color, the higher the similarity. The two figures depict how units with high activity establish long-distance connections to resolve the aperture problem when subjected to Gabor (ambiguous motion) stimuli. In contrast, the plaid stimuli (determined motion) suppress these long-distance connections. We will provide details in captions. \n\n*More details*:\nFig. 4 (B) compares spatial motion integration between 1D Gabor motion (left) and 2D plaid motion (right). Humans are able to perceive global downward motion only in the former case. In the latter case, local integration of motion signals takes priority over global integration. Once the local ambiguity is resolved, the global integration process is suppressed. Our model can predict such adaptive motion pooling in human visual processing. (K. Amano et al., 2009)\n\n---\n\nWe hope our responses address the reviewer's concerns and appreciate the reviewer's constructive feedback.\n\nBest,\nAuthors"}, {"review_id": "CEcKLQR4c3", "reviewer": "Reviewer_rMjX", "summary": "This paper proposes an image-computable model of human motion perception, bridging the gap between biological computation and CV models. The proposed model contains a two-stage approach that combines trainable motion energy sensing with a recurrent self-attention network for adaptive motion integration and separation. The similarity of the proposed model to human visual motion processing is demonstrated by computer neurophysiology experiments and psychophysics experiments.", "questions": "See the above statements and the following questions.\nQ1: As this study just uses cosine similarity, is the phrase “self-attention” suitable? Maybe it’s better to use a term from biological computation mechanisms.\nQ2: The meaning of some items in Equation 3 is not clear.\nQ3: What’s the origin of F in Fig.1E? Is it equal to E? Because the whole procedure is quite complicated, it needs clearer explanations/descriptions.\nQ4: What’s the meaning of “Stage II-1/2/3/4” in Fig.3 or Fig.5?\nQ5: Mixed usage of “two-stage”, “two-stages” and “two stages”. Please be consistent.\nQ6: In Table 1 caption, “speed, and direction” should be “direction, and speed”?", "limitations": "This paper has discussed its limitations in modeling human motion perception and its potential influence on future research. Maybe it’s better to add some limitations regarding to its performance on optical flow prediction, compared to other SOTA models.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. This study applies DNNs to construct a model of human motion perception that extracts informative motion flows for a wide range of inputs. Specifically, a two-stage model is proposed that simply mimics mammalian V1 and MT functions, respectively. \nS2. The two-stage model’s neurons exhibit direction and speed tunings similar to those observed in mammalian physiological recordings in V1 and MT.\nS3. Human-like responses are generated to several traditional motion stimuli and illusions, including the global motion pool and the barbershop illusion, showing good generalization from texture-free stimuli (e.g., drifting Garbo) to complex natural scenes.\nS4. This paper is clear and well-written overall.", "weaknesses": "W1: Although the primary purpose of this paper is to better model human motion perception, there is a gap between the two-stage model and state-of-the-art models in the CV field regarding the optical flow prediction performance, which needs more explanations/analyses.\nW2: The proposed two-stage model is relatively complicated, but this study lacks complexity analysis for each part. Considering the low energy consumption property of the human brain, energy efficiency may be another advantage of this model compared with other SOTA models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThank you for the reviewer's comprehensive comments and for recognizing the value of our work. \n\n--- \n\n**The gap between our two-stage model and SOTA models in the CV field regarding optical flow prediction performance:**\n\nWe recognize that our model's optical flow prediction performance may not be as high as other SOTA models. However, it's important to note that our primary goal is to model human motion perception. Therefore, we need to trade off biological plausibility over pure performance.\n\nOur model integrates classical motion energy computation with a graph connection and attention mechanism to solve dense optical flow in real-world scenarios effectively. In the design stage, using many stacked CNN layers instead of motion energy computation might enhance performance but would obscure each layer's specific functionality, deviating from our initial intention. Instead, our model is designed for clarity of function: the first layer extracts local energy, while the second layer manages motion integration and segregation.\nCV models are designed to match Ground Truth (GT) data best, but our approach adds extra constraints to maintain a balance of biological plausibility. Therefore, a simple correlation to GT shouldn't be the sole comparison index between our model and other SOTA models.\n\nRegarding Table I, our model is comparable to SOTA CV models when the object is the human response. Some SOTA models demonstrate a higher correlation with human responses. However, as a recent study (Yang et al. 2023) points out, there's a substantial inherent correlation between GT and human responses. As a result, any model trained to fit the ground truth must consider if the high correlation with human responses is merely a result of fitting the ground truth well. This led us to employ a partial correlation metric, which evaluates the correlation between the model and human responses while controlling for the impact of the GT. When evaluated using the partial correlation, our model outperforms the SOTA models with a considerable gap in Table I. Our preliminary analysis suggests that our model best explains human-perceived flow in naturalistic scenes because the attention network can do something similar to vector decomposition (Johansson, 1973).\n\n**The proposed two-stage model is relatively complicated and lacks complexity analysis for each part:**\n\nOur current focus is to make a computational model similar to humans and competitive with SOTA CV models without seriously prioritizing the complexity/efficiency of computation.\n\nThe choice of our framework is based on its algorithmic affinity to complex neural computations in the human brain. The two-stage concept is a simplified yet representative version of the real-world structure of the human brain's motion processing system. Considering the complex topology and dynamic temporal characteristics of brain neurons, we have incorporated graph structures and recurrent neural networks to deal with the motion integration tasks, a function of the MT regions. With a total of 14.7 million trainable parameters, our model remains leaner compared to generic CNN backbones.\n\nOur study suggests what kinds of computational mechanisms are necessary to explain human perception. It is possible that the neural implementation for this computation is different and more efficient.\n\n**Questions**:\n\n- Q1: As this study just uses cosine similarity, is the phrase \"self-attention\" suitable? \n\nThanks for your suggestion. According to our understanding, the \"self-attention/ attention mechanism\" in computer vision and deep learning fields is a kind of calculating the similarity between each pair of spatial location/ temporal step across the 2-D feature map / 1-D sequence. Most works, such as current transformer architecture, use dop-product distance as the distance metric. The cosine similarity used in this work is also a mathematical distance metric. Using different distance metrics wouldn't deviate from the same concept of self-attention. The cos similarity can be decomposed into dot-product with a normalization factor, and it could normalize numeric value into [-1,1]. In our experiment, we found that this feature helps the numeric stability of the model when training. \n\n- Q2: The meaning of some items in Equation 3 is not clear.\n\nApologies for the confusing mathematical symbols. The symbol $\\Re$ and $\\Im$ denote acquiring the Real and Imaginary parts from a complex value, respectively.\nWe will replace that with **Im[.]** and **RE[.]** for a better understanding. In addition, $*$ means the convolution operation.\n\n- Q3: What's the origin of F in Fig.1E? Is it equal to E? needs clearer explanations/descriptions.\n\nIf we refer to Fig.1 D, the RMIB block corresponds to Fig 1. In the initial recurrent stage, the red dot (F) and blue dot (E) both originate from the same motion energy in Stage I. However, in subsequent recurrent stages, 'F' and 'E' begin to diverge. 'F' represents the guiding attention for global motion integration, whereas 'E' signifies the motion energy response utilized to decode the optical flow.\nWe acknowledge that this process could use more clarification and will enhance the captions to explain the model's workings better. Furthermore, we found a labeling error in Fig 1 legend (in the lower right corner), where the descriptions of the blue and red dots should be switched. These will be corrected in our revised manuscript. \n\n- Q4: What's the meaning of \"Stage II-1/2/3/4\" in Fig.3 or Fig.5? \n\nStage II is a recurrent stage, and we analyze the result of each recurrent iteration. Stage II -1/2/3/4 means the result from iteration 1/2/3…\nWe will provide details in the revision.\n\n- Q5: Mixed usage of \"two-stage\", \"two-stages\" and \"two stages\". \n\n- Q6: In Table 1 caption, \"speed, and direction\" should be \"direction, and speed\"?\n\nThanks for the suggestion. We will modify it.\n\n--- \n\nWe hope our responses sufficiently address the reviewer's concerns. \n\nBest,\n\nAuthors\n"}, {"review_id": "d2gRAPWB2d", "reviewer": "Reviewer_QXFT", "summary": "In this paper, the author tries to build up a visual system like human eyes. In this study, the authors propose a two-stage model that combines trainable motion energy sensing with a recurrent self-attention network to capture the computations in V1-MT, the core structure for motion perception in the biological visual system. The model's unit responses are similar to mammalian neural recordings regarding motion pooling and speed tuning and replicate human responses to various stimuli. The model outperforms several state-of-the-art computer vision models in explaining human responses that deviate from the ground truth.", "questions": "None", "limitations": "Please refer to weaknesses", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. This paper has a deep exploration of the human visual system based on on-the-shelf computer vision modules.The idea of using two stage framework to build up this system is insightful. \n2. The design of this paper is intuitive and this framework achieves reasonable results.\n\n", "weaknesses": "1. In this paper, the authors do not provide much visual evidence to prove their system.\n2. In some experiments, their method is still worse than state-of-the-art", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nWe genuinely appreciate the time and effort of the reviewer. We respond to the concerns pointed out by the reviewer as follows.\n\n----\n\n**Do not provide much visual evidence to prove their system**:\n\nWe entirely agree that for a topic as intricate as motion perception, dynamic visual demonstrations would offer a much more comprehensive and understandable insight than text alone. Unfortunately, due to page constraints in the conference proceedings, it was impractical to provide a thorough visual display of the main content of the paper. To supplement this, we have provided a wealth of visual evidence in the supplementary materials. These include multiple demonstration videos and the results produced by our model. We've added hyperlinks for your convenience, and we trust that these materials will give a more in-depth understanding of our work and its significance. \n\n**In some experiments, their method is still worse than state-of-the-art**:\n\nWe recognize that our model's optical flow prediction performance may not be as high as other SOTA models. However, it's important to note that our main goal is to model human motion perception. Therefore, we prioritize biological plausibility over pure performance.\n\nThe SOTA models are intentionally designed to achieve the best performance in matching ground truth (GT) data. In contrast, we incorporate additional constraints on our current models to trade off biological plausibility, such as motion energy computation. Hence, it is not appropriate to rely solely on a simple correlation to GT as an index for comparing the capabilities of our model with other SOTA models.\n\nRegarding the comparison with SOTA models, our model is comparable to SOTA CV models when the object is the human response. Table I indicate that some SOTA models might demonstrate a higher correlation with human responses. However, as another study [ref] points out, there's a substantial inherent correlation between the ground truth and human responses. As a result, any model trained to fit the ground truth must consider if the high correlation with human responses is merely a result of fitting the ground truth well. This led us to employ a partial correlation metric, which evaluates the correlation between the model and human responses while controlling for the impact of the ground truth. More intuitively, we exclude the covariance between GT and the model prediction as well as between the GT and human response. Therefore, the partial correlation we used is only related to how strong model prediction could explain the variance in human response, which turns out to be a pure correlation between humans and the model. We believe this offers a more accurate reflection of the model's performance on human response.\n\nWhen we refer to the partial correlation, our model outperforms the SOTA models in Table I. The likely reason is that the current SOTA models in the computer vision field are primarily designed to match the ground truth as closely as possible, which is not the objective of this study. Instead, we aim to consider a wider range of factors associated with the human visual motion system when developing the model. The data in Table I substantiates that when we eliminate the effect of the Ground Truth, our model tends to align more closely with human responses in complex natural scenes.\n\n\n\n\n\n---\n\nWe trust these clarifications address the reviewer's concerns and offer a more transparent understanding of our work. \nThanks again for the reviewer's constructive contribution to our work.\n\nBest,\nAuthors\n\n\n[ref] \n- Yang, Y. H., Fukiage, T., Sun, Z., & Nishida, S. Y. (2023). Psychophysical measurement of perceived motion flow of naturalistic scenes. Available at SSRN 4414877\n"}]

Author rebuttal:
We provided more comparison results in the attached PDF.

Meta-review: There was general agreement among the reviewers that the proposed model of the dorsal stream adds value to an already crowded field of computational models. In particular, the reviewers noted that the model appears sophisticated compared to prior models and that the breadth of in-sillico experiments presented is to be lauded. Reviewers asked for clarifications and additional experiments that were all answered in the rebuttal. The AC recommends this paper be accepted.
Decision: Accept (poster)