SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lujie Xia $^{1,2}$ , Ziluo Ding $^{2,3}$ , Rui Zhao $^{1,2}$ , Jiyuan Zhang $^{1,2}$
Lei Ma $^{5}$ , Zhaofei Yu $^{1,2,4}$ , Tiejun Huang $^{1,2}$ , Ruiqin Xiong $^{1,2*}$
$^{2}$ National Engineering Research Center of Visual Technology (NERCVT)
$^{4}$ Institute for Artificial Intelligence, Peking University
$^{5}$ National Biomedical Imaging Center, College of Future Technology, Peking University
{lujie.xia, ziluo, lei.ma, yuzf12, tjhuang, rqxiong}@pku.edu.cn
{ruizhao, jyzhang}@stu.pku.edu.cn
Efficiently selecting an appropriate spike stream data length to extract precise information is the key to the spike vision tasks. To address this issue, we propose a dynamic timing representation for spike streams. Based on multi-layers architecture, it applies dilated convolutions on temporal dimension to extract features on multi-temporal scales with few parameters. And we design layer attention to dynamically fuse these features. Moreover, we propose an unsupervised learning method for optical flow estimation in a spike-based manner to break the dependence on labeled data. In addition, to verify the robustness, we also build a spike-based synthetic validation dataset for extreme scenarios in autonomous driving, denoted as SSES dataset. It consists of various corner cases. Experiments show that our method can predict optical flow from spike streams in different high-speed scenes, including real scenes. For instance, our method achieves  $15\%$  and  $19\%$  error reduction on PHM dataset compared to the best spike-based work, SCFlow, in  $\Delta t = 10$  and  $\Delta t = 20$  respectively, using the same settings as in previous works. The source code and dataset are available at https://github.com/Bosserhead/USFlow.
Optical flow is defined as the apparent motion of individual pixels on the image plane and is prevalent for being an auxiliary tool for various vision tasks, e.g. frame rate conversion [15], scene segmentation [43, 38], and object detection [49]. In high-speed scenes, the optical flow estimation may suffer from blurry images from low frame-rate traditional cameras. Obtaining the data with a device that can precisely record the continuous light intensity changes of the scene is the key to addressing this issue. Recently, neuromorphic cameras have been developed greatly, such as event camera and spike camera. They can record light intensity changes in high-speed scenes. Especially, for spike camera, each pixel of it responds independently to the accumulation of photons by generating asynchronous spikes. It records full visual details with an ultra-high temporal resolution (up to  $40\mathrm{kHz}$ ). With these features, spike camera has demonstrated superiority in handling some high-speed scenarios, such as video frame interpolation [35] and dynamic scene reconstruction [40, 42, 7, 8, 48].
Since spike camera can record details of high-speed moving objects, it has enormous potential for estimating more accurate optical flow in high-speed scenes. Considering that deep learning has
achieved remarkable success in frame-based optical flow estimation [31, 32], it seems reasonable to directly apply frame-based architecture in spike data. However, the data modality of spike stream output by spike camera is quite different from frame images. For each pixel in spike camera, a spike is fired and the accumulation is reset when photons accumulation at a pixel exceeds a set threshold. At each timestamp, the spike camera outputs a binary matrix, denoted as spike frame, representing the presence of spikes at all pixels. Previous work [12] utilizes spike stream as naive input representation for one timestamp, which consists of a series of spike frames within a predefined time window. However, a too-long window can incur lots of misleading frames, while a short window is sensitive to noise and cannot provide enough information. Therefore, deliberate modifications are needed for input representation to extract salient information more flexibly and efficiently from spike streams before optical flow estimation architecture takes over.
In addition, the ground truth of optical flow is scarce in the real world, especially for high-speed scenes. To cope with the lack of labeled real-world data, it is necessary to study spike-based optical flow estimation in an unsupervised manner. As described above, the light intensity information is contained within the spike intervals. This difference in data characteristics makes it unreasonable to directly apply the frame-based unsupervised loss to spike streams. Therefore, the light intensity should first be extracted from spike streams in the spike-based unsupervised loss. This is also the core of constructing the illuminance consistency on spike streams. Moreover, we argue the field of autonomous driving is a good place to validate spike camera since it is suitable for high-speed scenes. In autonomous driving, it is nearly impossible to collect real data for complex, diverse, high-speed extreme scenarios, e.g. vehicle collisions and pedestrian-vehicle accidents. However, these scenarios are of great significance to improve the safety of this field and should be highlighted. Therefore, in order to verify that the spike-based algorithm can handle extreme scenarios, we propose a spike-based synthetic validation dataset for extreme scenarios in autonomous driving, denoted as the SSES dataset.
In this paper, we propose an unsupervised method for spike-based optical flow estimation with dynamic timing representation, named USFlow. In our unsupervised loss, we propose two strategies, multi-interval-based and multi-time-window-based, to estimate light intensity in regions with different motion speeds. The estimated optical flow is utilized to distinguish regions with different motion speeds and generates corresponding weights for light intensity fusion. Then the final approximate light intensity participates in loss calculation. As for addressing the fixed time window issue, there is a way that apply a dynamic time window to the different spike streams. To this end, we propose Temporal Multi-dilated Representation(TMR) for spike streams. In more detail, we apply multi-layer dilated convolutions to operate on the temporal dimension of spike streams. Multi-layer dilated convolutions enable the network to have different receptive fields and each layer can be regarded as summarizing spike stream with one different time window. We also design a Layer Attention(LA) module to extract salient features and filter the redundant ones.
Following the settings in previous works [12], we train our method on SPIFT [12] dataset and evaluate it on PHM [12] and our proposed SSES datasets. We demonstrate its superior generalization ability in different scenarios. Results show that USFlow outperforms all the existing state-of-the-art methods qualitatively and quantitatively. USFlow shows visually impressive performance on real-world data.
Deep Learning in Optical Flow Estimation. Frame-based optical flow estimation is a classical task in the computer vision area through the years and has been solved well. PWC-Net[31] and Liteflownet[13] introduce the pyramid and cost volume to neural networks for optical flow, warping the features in different levels of the pyramid and learning the flow fields in a coarse-to-fine manner. RAFT[32] utilizes ConvGRU to construct decoders in the network and iteratively decodes the correlation and context information in a fixed resolution. Due to their excellent performance, PWC-Net and RAFT are the backbones of most algorithms[37, 34, 41, 21, 20, 18, 17, 36, 16, 39, 30] in frame-based optical flow estimation. TransFlow [22] utilizing the long-range temporal association to recover more information by introducing transformer architecture. In addition, many frame-based unsupervised optical flow networks[25, 21, 20, 18, 23, 30] were proposed to discard the need for labeled data. Similar to the traditional optimization-based methods, Yu et al.[14] employ photometric loss and smoothness loss to train a flow estimation network. Unflow[25] applies a forward-backward
check on bidirectional optical flow to estimate the occlusion area, where the backpropagation of photometric loss is stopped.
Deep learning has also been applied to event-based optical flow[45, 46, 19, 6, 11, 26, 5]. EV-FlowNet[45] can be regarded as the first deep learning work training on large datasets with a U-Net architecture[28]. As an updated version of EV-FlowNet, an unsupervised framework has been proposed by Zhu et al.[46] based on contrast maximization. Spike-FlowNet[19] and STE-FlowNet[6] use spiking neural networks and ConvGRU to extract the spatial-temporal features of events, respectively.
The research on spike-based optical flow estimation is just getting started. SCFlow[12], is the first deep-learning method. It proposes a large dataset, the SPIFT dataset, to train an end-to-end neural network via supervised learning. However, our work aims to fill in the blanks of unsupervised learning.
Event-based and Spike-based Input Representation. Normally, asynchronous event streams are not compatible well with the frame-based deep learning architecture. Therefore, frame-like input representation is needed and is expected to capture rich salient information about the input streams. Apart from many handcrafted representations[46, 27, 24, 29, 6], some end-to-end learned representations have been proposed, making it possible to generalize to different vision tasks better. Gehring et al.[10] simply uses multi-layer perceptrons as a trilinear filter to produce a voxel grid of temporal features. Cannici et al.[4] propose Matrix-LSTM, a grid of Long Short-Term Memory (LSTM) cells that efficiently process events and learn end-to-end task-dependent event surfaces. Similarly, Event-LSTM [1] utilizes LSTM cells to process the sequence of events at each pixel considering it into a single output vector that populates that 2D grid. Moreover, Vermprala et al.[33] presents an event variational autoencoder and shows that it is feasible to learn compact representations directly from asynchronous spatio-temporal event data.
Directly using spike stream as input might incur lots of misleading information or miss something necessary if the time window is not chosen appropriately. Therefore, frame-like input representation should also be carefully designed to extract sufficient information from spike stream. SCFlow [12] uses estimated optical flow to align spike frames to eliminate motion blur.
Different from all previous works, we aim to train an end-to-end input representation with the function of a dynamic time window by multi-layer dilated convolutions.
3.1 Spike Camera
Spike camera works by an "integrate-and-fire" mechanism, which asynchronously accumulates the light on each pixel. The integrator of spike cameras accumulates the electrons transferred from incoming photons. Once the cumulative electrons exceed a set threshold, the camera fires a spike and resets the accumulation. The process of accumulation can be formulated as,
$$
\mathbf {A} (\mathbf {x}, t) = \int_ {0} ^ {t} \alpha I (\mathbf {x}, \tau) d \tau \mod \theta , \tag {1}
$$
where  $\mathbf{A}(\mathbf{x}, t)$  is the cumulative electrons at pixel  $\mathbf{x} = (x, y)$ .  $I(\mathbf{x}, \tau)$  is the light intensity at pixel  $\mathbf{x}$  at time  $\tau$ .  $\alpha$  is the photoelectric conversion rate.  $\theta$  is the threshold. The reading time of spikes is quantified with a period  $\delta$  of microseconds. The spike camera fires spikes at time  $T$ ,  $T = n\delta$ ,  $n \in \mathbf{Z}$ , and generate an  $H \times W$  spike frame  $s$ . As time goes on, the camera produces a spatial-temporal binary stream  $S_{t}^{N}$  in  $H \times W \times N$  size, as shown in Figure 2. The  $N$  is the temporal length of the spike stream.  $H$  and  $W$  are the height and width of the sensor, respectively.
Given two timestamps  $t_0$  and  $t_1$ , we have two spike streams centered on  $t_0$  and  $t_1$ , noted as  $S_{t_0}^L$  and  $S_{t_1}^L$ , respectively. Then we estimate a dense displacement field  $\mathbf{f} = (f^u, f^v)$  from  $t_0$  to  $t_1$  using
Figure 1: The overall architecture of the USFlow. First, The input spike stream is through the temporal multi-dilated representation, and the layer attention fuses the multi-layer outputs. Second, the fused output is fed to the optical flow estimation backbone. Finally, it uses estimated optical flow to learn weights for light intensity estimation.
these two sub-spike streams, mapping each pixel  $\mathbf{x} = (x,y)$  at  $t_0$  to its corresponding coordinates  $\mathbf{x}' = (x',y') = (x + f^u (\mathbf{x}),y + f^v (\mathbf{x}))$  at  $t_1$ .
To verify the effectiveness of the proposed dynamic timing representation, we present two versions of USFlow. One is the PWC-like version. It adopts a variant of PWC-Net [31] which is also the backbone of SCFlow [12]. The other is the RAFT version which adopts an official small version of RAFT [32] as the backbone. More about these two backbones is included in the appendix. Considering one advantage of neuromorphic vision is low latency, our representation part should be lightweight and efficient. The two spike streams first pass into the shared dynamic timing representation module separately, whose outputs are sent into existing backbones. In addition, we design an unsupervised loss to break the dependency on labeled data. Note that all the components are trained end-to-end.
4.2 Dynamic Timing Representation
As stated in 3.1, one binary spike frame itself is sensitive to noise and meaningless without contextual connection. Given a bunch of spike data, selecting an appropriate data length is pivotal for subsequent processing. A too-long spike stream is not suitable for high-speed regions since time offset accumulation introduces more redundant information. A too-short spike stream won't work either, as it can not exhibit light intensity precisely with few binary data. To address this issue, we propose a dynamic timing representation for input spike streams.
The Dynamic Timing Representation consists of Temporal Multi-dilated Representation (TMR) module and a Layer Attention module (LA). The main ingredient of TMR is dilated convolution. By using dilated convolution, we can extend receptive fields to a larger range with just a few stacked layers, while preserving the input resolution throughout the network as well as computational efficiency. In more detail, we apply 1D dilated convolutions to the spike stream for each pixel and the parameters are shared across all the pixels. The TMR can be formulated as follow,
$$
\left\{F ^ {(i)} (\mathbf {x}) \right\} = \mathrm {D 1 C} ^ {(i)} \left(F ^ {(i - 1)} (\mathbf {x})\right) \quad i = 1, \dots , n, \tag {2}
$$
where  $\mathrm{D1C}(\cdot)$  represents the 1D dilated convolution operation,  $i$  and  $\mathbf{x}$  are layer index and spatial coordinate respectively. Note that  $F^{(0)}(\mathbf{x})$  is the input spike stream on position  $\mathbf{x}$ . Figure 2 depicts dilated convolutions for dilations 1, 2, 4, and 8. The higher level of the layer, the larger the receptive field is. The intuition behind this configuration is two-fold. First, a different layer can be regarded as summarizing spikes or extracting temporal correlation among spike frames with a different time window. Second, dilated convolution allows the network to operate on a coarser scale more effectively than with a normal convolution, which guarantees the model is lightweight. In Table 1, we show the parameter size of our USFlow and other methods.
Figure 2: The Temporal Multi-dilated Representation. 1D dilated convolutions are applied to all pixels with shared weights. The dilated convolutions are stacked for larger receptive fields. The higher level the layer is, the larger the receptive field is. Note that the illustration of the spike stream is shown in the top-left corner.
Multi-dilated architecture has already greatly expanded the input dimensions but not all layers provide equally useful information. Blindly fusing the output of all layers may impair the learning process. To address this issue, we propose the Layer Attention module to further flexibly emphasize which layer is meaningful or which time window is optimal. As illustrated in Figure 1, we average the output of  $n$  layers,  $\{F^{(i)}(\mathbf{x})\}_{i=1}^n$ , noted as  $F'(\mathbf{x})$  to generate one  $n$ -dimension layer context descriptor. The descriptor is then forwarded to a multi-layer perceptron(MLP) to produce our layer attention map. The layer attention values are broadcast along the layer dimension and the final concatenation,  $RF(\mathbf{x})$ , is computed as follows:
$$
R F (\mathbf {x}) = \sigma (\operatorname {M L P} (\operatorname {A v g P o o l} (\mathrm {F} ^ {\prime} (\mathbf {x}))) \otimes \mathrm {F} ^ {\prime} (\mathbf {x}), \tag {3}
$$
here  $\otimes$  denotes element-wise multiplication. We do this operation on all pixels by sharing weights.
4.3 Unsupervised Loss
Unsupervised learning is necessary for spike-based optical flow estimation due to the lack of labeled training data in high-speed real-world scenes. To this end, we sidestep the need for ground truth of optical flow as only spike data are required for training. More specifically, at training time, two spike streams centered on  $t_0$  and  $t_1$ ,  $S_{t_0}^L$  and  $S_{t_1}^L$ , are utilized to predict optical flow,  $\mathbf{f} = (f^u, f^v)$ , from  $t_0$  to  $t_1$ .
Different from traditional RGB frames, a binary spike frame can not accurately represent the light intensity at the current timestamp. Therefore, estimating precise current light intensity from binary spike streams is the key to constructing an unsupervised loss function.
For low-speed regions. Since the light intensity accumulation in low-speed regions is an approximately linear process, we count spikes to utilize longer-duration information to estimate the light intensity. In this way, we can improve the robustness of light intensity estimation. However, the data length of the spike stream used to estimate light intensity varies with different low-speed motions. For reducing computational complexity, we only set two different time windows. The light intensity estimation can be formulated as follow:
$$
\tilde {I} _ {T} (\mathbf {x}, \tau) = \frac {\omega_ {s} \cdot \theta}{2 D _ {s} + 1} \cdot \sum_ {t = \tau - D _ {s}} ^ {\tau + D _ {s}} s (\mathbf {x}, t) + \frac {\omega_ {l} \cdot \theta}{2 D _ {l} + 1} \cdot \sum_ {t = \tau - D _ {l}} ^ {\tau + D _ {l}} s (\mathbf {x}, t), \tag {4}
$$
here  $D_{s}$  and  $D_{l}$  are the half length of time windows. They are set to 40 and 100 respectively.  $\omega_{s}$  and  $\omega_{l}$  are the weight factors. The subscripts  $s$  and  $l$  refer to short and long time windows, respectively.
For high-speed regions. Selecting a large time span of spike streams to extract information would incur motion blur due to time offset accumulation. Hence, we estimate light intensity during a single spike interval which is typically on the order of microseconds. Since the spike streams have extremely high temporal resolution and the interval between adjacent spikes is ultra-short, we can safely assume that the light intensity remains constant during the interval [47]. Let us denote  $s(\mathbf{x},m)$  and  $s(\mathbf{x},n)$
as two adjacent spikes at position  $\mathbf{x}$ .  $m$  and  $n$  are the timestamps corresponding to these two spikes. According to the spike camera working mechanism in Equation 1, the constant light intensity can be approximated by:
$$
\hat {I} (\mathbf {x}) \approx \frac {\theta}{\alpha \cdot (n - m)} , m <   n \tag {5}
$$
In reality, however, the number of incoming photons in an ultra-short interval is a random variable subject to the Poisson distribution even under constant illumination conditions. Therefore, the light intensity calculated by Equation 5 includes errors due to random fluctuations. To address this issue, we extend it by multi-intervals and fuse light intensity information with different numbers of intervals,
$$
\tilde {I} _ {I} (\mathbf {x}, \tau) = \sum_ {k = 1} ^ {K} \left(\omega_ {k} \cdot \frac {(2 k - 1) \cdot \theta}{\alpha \cdot [ T (\mathbf {x} , N _ {\tau} (\mathbf {x}) + k - 1) - T (\mathbf {x} , M _ {\tau} (\mathbf {x}) - k + 1) ]}\right), \tag {6}
$$
$$
\text {w h e r e} \quad M _ {\tau} (\mathbf {x}) = \underset {z} {\arg \max } (T (\mathbf {x}, z) <   \tau), \quad N _ {\tau} (\mathbf {x}) = \underset {z} {\arg \min } (T (\mathbf {x}, z) \geq \tau). \tag {7}
$$
In Equation 6 and 7,  $T(\mathbf{x}, z)$  refers to the timestamp corresponding to the  $z$ th spike at position  $\mathbf{x}$ .  $[T(\mathbf{x}, N_{\tau}(\mathbf{x}) + k - 1) - T(\mathbf{x}, M_{\tau}(\mathbf{x}) - k + 1)]$  is the total time length of these  $(2k - 1)$  intervals.  $\omega_{k}$  is the weight factor of the light intensity calculated by using  $(2k - 1)$  intervals. Note that  $k$  is set to 1 and 2 in our experiments. This setting ensures that the data length of spike stream used for light intensity estimation in high-speed motion regions is much shorter than that used in low-speed regions. Further detailed discussion is presented in the appendix.
Learnable weights of estimated light intensity. Considering a scene may contain both high-speed regions and low-speed regions, it is necessary for our unsupervised loss to fuse light intensity estimation methods based on multiple intervals and multiple time windows. We use the estimated optical flow to reflect the motion speed, and choose the most appropriate light intensity estimation strategy for different regions. As shown in Figure 1, we learn the weights  $\omega_{s},\omega_{l},\omega_{k}$  from the estimated optical flow. Thence, we can fuse all the terms in Equation 4 and Equation 6 to achieve the final approximated light intensity  $\tilde{I}$ .
After achieving the  $\tilde{I}$ , we then derive the bidirectional photometric loss in a spike-based manner:
$$
\mathcal {L} _ {\text {p h o t o}} (\mathbf {f}, \mathbf {f} ^ {\prime}) = \sum_ {\mathbf {x}} \left(\rho \left(\tilde {I} (\mathbf {x}, t _ {0}) - \tilde {I} (\mathbf {x} + \mathbf {f}, t _ {1})\right) + \rho \left(\tilde {I} (\mathbf {x} + \mathbf {f} ^ {\prime}, t _ {0}) - \tilde {I} (\mathbf {x}, t _ {1})\right)\right), \tag {8}
$$
where  $\rho$  is the Charbonnier loss [3],  $\mathbf{f}$  is the flow from  $t_0$  to  $t_1$ ,  $\mathbf{f}'$  is the flow from  $t_1$  to  $t_0$ .
Furthermore, we use smoothness loss to regularize the predicted flow. It minimizes the flow difference between neighboring pixels, thus it can enhance the spatial consistency of neighboring flows and mitigate some other issues, e.g. the aperture problem. It can be written as:
$$
\mathcal {L} _ {\text {s m o o t h}} \left(f, f ^ {\prime}\right) = \frac {1}{H W} \sum_ {\mathbf {x}} | \nabla f (\mathbf {x}) | + | \nabla f ^ {\prime} (\mathbf {x}) |, \tag {9}
$$
where  $\nabla$  is the difference operator,  $H$  is the height and  $W$  is the width of the predicted flow. The total loss function consists of two loss terms above-mentioned, which can be written as  $\mathcal{L}_{\mathrm{total}} = \mathcal{L}_{\mathrm{photo}} + \lambda \mathcal{L}_{\mathrm{smooth}}$ , where  $\lambda$  is the weight factor.
In this work, we choose the SPIFT dataset [12] as the training dataset. The PHM dataset [12] and our proposed SSES dataset is used for evaluation. The SPIFT and PHM datasets provide two data settings, generating optical flow every 10 spike frames ( $\Delta t = 10$ ) and 20 spike frames ( $\Delta t = 20$ ) separately from the start to end of sampling. Therefore, we train models for the ( $\Delta t = 10$ ) and