SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\| \vec{F} \| = \sqrt{X_0^2 + Y_0^2 + Z^2}$ ,  $\cos \Theta_0 = Z / \| \vec{F} \|$ , and  $\tan \Phi_0 = Y_0 / X_0$ . It turns out what we are looking for is a rotation by an amount  $\Theta_0$  about a unit vector  $\hat{u}$  that depends only on  $\Phi_0$ . That is,
$$
\hat {u} \left(\Phi_ {0}\right) = \left( \begin{array}{c} - \sin \Phi_ {0} \\ \cos \Phi_ {0} \\ 0 \end{array} \right), \tag {20}
$$
and the rotation matrix is given by  $R_{\hat{u}}(\Theta_0) = I + (\sin \Theta_0)L(\hat{u}) + (1 - \cos \Theta_0)L(\hat{u})^2$
Here,  $L(\vec{\omega})$  is the skew-symmetric matrix representation of a vector  $\vec{\omega}$ , that is
$$
L (\vec {\omega}) := \vec {\omega} \cdot \vec {L} = \omega_ {X} L _ {X} + \omega_ {Y} L _ {Y} + \omega_ {Z} L _ {Z}, \tag {21}
$$
where  $L_{i}$  are the familiar basis matrices for  $\mathfrak{so}(3)$ , the Lie algebra of  $SO(3)$ :
$$
L _ {X} = \left( \begin{array}{c c c} 0 & 0 & 0 \\ 0 & 0 & - 1 \\ 0 & 1 & 0 \end{array} \right), \quad L _ {Y} = \left( \begin{array}{c c c} 0 & 0 & 1 \\ 0 & 0 & 0 \\ - 1 & 0 & 0 \end{array} \right), \quad L _ {Z} = \left( \begin{array}{c c c} 0 & - 1 & 0 \\ 1 & 0 & 0 \\ 0 & 0 & 0 \end{array} \right). \tag {22}
$$
The rotation matrix in its full form is expressed as
Fixed coordinate system:
Observer's coordinate system:
$$
(X, Y, Z)
$$
$$
(x, y, z)
$$
Fixation point:
$$
\vec {F} = (X _ {0}, Y _ {0}, 1)
$$
Any given point in the scene:
$$
\vec {P} = (X, Y, Z)
$$
Observer:
$$
\vec {O} = (0, 0, 0)
$$
These points are all defined in the fixed coordinates
Figure 18: Setup. (a) Background and coordinate systems. (b) The rotation of the fixed coordinates  $\left(\hat{X},\hat{Y},\hat{Z}\right)$  onto the observer-centric coordinates  $(\hat{x},\hat{y},\hat{z})$  can be accomplished by applying a rotation of angle  $\Theta_0$  around the unit vector  $\hat{u}$  as defined in Equation 20. Note that by definition,  $\hat{z}$  is always parallel to the fixation point. See the text for more details. (c) Perspective from the observer's point of view. The retinotopic coordinates of point  $\vec{P}$  are denoted by  $\alpha_{x}$  and  $\alpha_{y}$ , measured in radians.
$$
R := R _ {\hat {u}} \left(\Theta_ {0}\right) = \left( \begin{array}{c c c} s ^ {2} + c ^ {2} \cos \Theta_ {0} & - c s [ 1 - \cos \Theta_ {0} ] & c \sin \Theta_ {0} \\ - c s [ 1 - \cos \Theta_ {0} ] & c ^ {2} + s ^ {2} \cos \Theta_ {0} & s \sin \Theta_ {0} \\ - c \sin \Theta_ {0} & - s \sin \Theta_ {0} & \cos \Theta_ {0} \end{array} \right), \tag {23}
$$
where we have defined  $c\coloneqq \cos \Phi_0$  and  $s\coloneqq \sin \Phi_0$
Any point  $\vec{P} = (X,Y,Z)$  defined in the fixed coordinate system can be mapped to a point  $\vec{p} = R^T\vec{P}$  with components  $\vec{p} = (x,y,z)$  given in the observer's system. For example, it can be shown that  $R^{T}\vec{F}$  is parallel to  $\hat{z}$ , which is how we defined  $R$  in the first place. See Fig. 18.
In conclusion, we have a rotation matrix  $R$  that is fully determined as a function of the fixation point  $\vec{F}$ . Applying  $R$  rotates the fixed coordinate system onto the observer's system
$$
\hat {x} := R \hat {X},
$$
$$
\hat {y} := R \hat {Y}, \tag {24}
$$
$$
\hat {z} := R \hat {Z}.
$$
This way, the observer's coordinates  $(x,y,z)$  are obtained ensuring that the fixation point is always parallel to  $\hat{z}$ .
13.1.2 Adding motion
Consider a point  $\vec{P} = (X, Y, Z)$ . In the previous section, we showed that the rotation matrix in Equation 23 can be used to relate  $\vec{P}$  to its observer-centric coordinates
$$
\vec {P} = R \vec {p}. \tag {25}
$$
Differentiate both sides to get
$$
\vec {V} := \frac {d \vec {P}}{d t} = \frac {d R}{d t} \vec {p} + R \frac {d \vec {p}}{d t}, \tag {26}
$$
where  $\vec{V}$  is the velocity of the point relative to the observer, with components measured in the fixed coordinate system.
13.1.3 Derivative of the rotation matrix
Here we will derive a closed-form expression for  $\frac{dR}{dt}$  in terms of the observer's angular velocity. This is important because it essentially captures the effects of eye movement.
Let us use  $\vec{\omega} = (\omega_{X},\omega_{Y},\omega_{Z})$  to denote the angular velocity of observer's coordinates, with components expressed in fixed coordinates. The time derivative of  $R$  is given as follows
$$
\frac {d R}{d t} = L (\vec {\omega}) R, \tag {27}
$$
where  $L(\vec{\omega})$  is given in Equation 21. Taken together, equations 3, 4, 8, and 9 yield
$$
\begin{array}{l} \boxed {\frac {d \vec {p}}{d t} = R ^ {T} \vec {V} - \left[ R ^ {T} L (\vec {\omega}) R \right] \vec {p}} \\ = R ^ {T} \vec {V} - \left[ R ^ {T} \left( \begin{array}{c c c} 0 & - \omega_ {Z} & \omega_ {Y} \\ \omega_ {Z} & 0 & - \omega_ {X} \\ \omega_ {Y} & \omega_ {X} & 0 \end{array} \right) R \right] \vec {p} \tag {28} \\ \end{array}
$$
This equation allows relating the velocity of a moving point expressed in fixed coordinates,  $\vec{V}$ , to its observer-centric equivalent,  $\vec{v} \coloneqq d\vec{p} / dt$ . Importantly, Equation 28 is a general relationship valid for any combination of velocities and rotations.
13.1.4 Motion perceived by the observer
Before discussing specific applications, we need to solve one remaining piece of the puzzle: what is  $\vec{v}$ , and how is it related to the motion perceived by the observer? To address this, we need to map the retinotopic coordinates of the observer (measured in radians), to points in the environment (measured in the observer-centric coordinates).
Denote the retinotopic coordinates using  $(\alpha_{x},\alpha_{y})$ . The goal is to relate each  $(\alpha_{x},\alpha_{y})$  pair to a point in the world  $\vec{p}$  with coordinates  $(x,y,z)$ . It can be shown that
$$
\tan \alpha_ {x} = x / z = \tan \theta \cos \phi ,
$$
$$
\tan \alpha_ {y} = y / z = \tan \theta \sin \phi ,
$$
where  $\theta$  and  $\phi$  are the polar coordinates of  $\vec{p}$ . See Fig. 18c for an illustration.
Differentiate both sides of Equation 29 and rearrange terms to obtain the time derivative of  $(\alpha_{x},\alpha_{y})$  as a function of  $\vec{v} = (\dot{x},\dot{y},\dot{z})$  and  $\vec{p}$ . That is,
$$
\begin{array}{l} \dot {\alpha_ {x}} = \frac {\dot {x} z - \dot {z} x}{z ^ {2} + x ^ {2}} \\ \dot {\alpha_ {y}} = \frac {\dot {y} z - \dot {z} y}{z ^ {2} + y ^ {2}} \end{array} \tag {30}
$$
Thus, we obtained a relationship between retinotopic velocity  $(\dot{\alpha}_x,\dot{\alpha}_y)$  (measured in radians/s) and velocity of objects in the world (measured in  $m / s$ ). In conclusion, equations 28 and 30 enable us to compute the observer's perceived retinotopic velocity for any type of motion.
13.2 Example: maintaining fixation
We are now ready to apply the formalism developed in the preceding section to create optical flow patterns in realistic settings. In this section, we will consider a specific example: self-motion while maintaining fixation on a background point. This is an important and ecologically relevant type of motion, as we often tend to maintain fixation while navigating the environment—think about this next time you are moving around. In fact, one can argue that fixation occurs more naturally during behaviors such as locomotion, as opposed to typical vision experiments where the animals are stationary and passively viewing a screen.
What does it mean to maintain fixation while moving? It means the eyes rotate to cancel out the velocity of the fixation point. Assume the observer is moving with velocity  $\vec{V}_{\text {self }}$ . Equivalently, we can imagine the observer is stationary while the world is moving with velocity  $\vec{V} = -\vec{V}_{\text {self }}$  (this is the same  $\vec{V}$  that appears in Equation 28).
Any eye movement can be represented mathematically as a rotation of the observer's coordinates with respect to the fixed coordinates. Use  $\vec{\omega}$  to denote the angular velocity of the rotation. We can determine  $\vec{\omega}$  by requiring that the rotation cancels out the velocity of the fixation point. To this aim, we first compute the normal component of  $\vec{V}$  with respect to the fixation point  $\vec{F}$  as follows
$$
\vec {V} _ {\perp} := \vec {V} - \frac {\vec {V} \cdot \vec {F}}{\| \vec {F} \| ^ {2}} \vec {F}. \tag {31}
$$
Use the above equation to compute the angular velocity
$$
\vec {\omega} = \frac {\vec {F} \times \vec {V} _ {\perp}}{\| \vec {F} \| ^ {2}}. \tag {32}
$$
Plug Equation 32 in Equation 28 to find  $\vec{v}$ , and use Equation 30 to find  $(\dot{\alpha}_x, \dot{\alpha}_y)$ . This way, we just calculated the motion perceived by the observer during self-motion while maintaining fixation.
13.3 The algorithm
Start by choosing a fixation point  $\vec{F} = (X_0, Y_0, Z)$ , field-of-view extent  $(fov)$ , and retinal resolution  $(res)$ . For example,  $fov = 45^\circ$  and  $res = 5.625^\circ$ . Use this to create a mesh grid for  $(\alpha_x, \alpha_y)$ . The grid covers  $(-fov, +fov)$  degrees in each dimension and has shape  $2 * f_{ov} / res + 1$ , or  $17 \times 17$  in our example. Use Equation 29 to compute  $(\theta, \phi)$  pair for each point on the  $(\alpha_x, \alpha_y)$  grid.
Because the simulation is scale-invariant, we can always fix the distance between the observer and the background wall. Assume  $Z = cte$ .  $= 1$ , and use equations 23 and 25 to find  $\vec{p} = (x,y,z)$  per grid point. Note that we assume infinite resolution for the points in the environment.
So far, we partitioned the retina into a grid and found the observer-centric coordinates of a point in the real world that falls on each grid point. Now we can sample a random self-motion velocity  $\vec{V}_{self}$  and use the results described in the previous sections to find  $(\dot{\alpha}_x,\dot{\alpha}_y)$ . This provides us with the  $2D$  retinal velocity vector on the grid and concludes the algorithm for the simple case of fixate-0 (i.e., no objects).
Please refer to our code for additional details, including instructions on how to incorporate independently moving objects into the scene.
13.3.1 Valid fixation points
The observer-wall distance is constrained to be a positive value. That is, the observer is always situated on the left side of the wall ( $Z > 0$  for all points; Fig. 18). This constraint, combined with a given  $fov$  value and Equation 23 results in a theoretical bound on which fixation points are valid. To derive this bound, we start by considering the  $\hat{Z}$  component of Equation 25
$$
(R \vec {p}) _ {3} = R _ {3 1} x + R _ {3 2} y + R _ {3 3} z = Z > 0. \tag {33}
$$
Divide both sides by  $z$  and use Equation 29 to get
$$
R _ {3 1} \tan \alpha + R _ {3 2} \tan \beta + R _ {3 3} > 0. \tag {34}
$$
Plug the matrix entries from Equation 23 and rearrange to get
$$
\begin{array}{l} \sin \Theta_ {0} (\cos \Phi_ {0} \tan \alpha + \sin \Phi_ {0} \tan \beta) <   \cos \Theta_ {0}, \\ \Rightarrow \quad X _ {0} \tan \alpha + Y _ {0} \tan \beta <   Z. \\ \end{array}
$$
For a given  $fov$  value, we have  $-\tan (fov) \leq \tan \alpha$ ,  $\tan \beta \leq \tan (fov)$ . Assuming  $Z = 1$ , we find that the  $(X_0, Y_0)$  coordinates of a valid fixation point should satisfy the following inequality
$$
\left| X _ {0} \right| + \left| Y _ {0} \right| <   \frac {1}{\tan (f o v)}. \tag {36}
$$
In other words, the  $L_{1}$  norm of  $(X_0,Y_0)$  should be less than  $1 / \tan (fov)$ . For example, when  $fov = 45^\circ$ , the upper bound is 1. This result can be used to choose an appropriate  $fov$ : smaller values allow a larger phase space to explore for fixation points.
13.4 Conclusions
We introduced Retinal Optic Flow Learning (ROFL), a novel synthetic data framework to simulate realistic optical flow patterns. The main result is given in Equation 28, which shows the simulation has the potential to generalize to various other types of motion. For instance, the observer might engage in smooth pursuit of a moving object instead of maintaining fixation, a scenario that can be modeled simply by adjusting the choice of  $\vec{\omega}$  in Equation 28. This flexibility enables us to generate a diverse repertoire of realistic optical flow patterns, which can be used in training and evaluating unsupervised models, among other applications.
We believe that our framework will prove valuable not only to neuroscientists studying motion perception but also to machine learning researchers interested in disentangled representation learning. By bridging these domains, we aim to foster interdisciplinary collaboration and advance our understanding of both biological and artificial information processing systems.