# Estimating Optical Flow with Streaming Perception and Changing Trend Aiming to Complex Scenarios
## 2 Related Works  
Before RAFT emerged, the most compact and effective framework for estimating optical flow is PWC-Net [2], which contained five parts: Feature Pyramid [3], Warping Layer [4], Cost Volume Layer [5], Optical Flow Estimator [6] and Context Network. Almost all optical flow methods are implemented on this basis until 2020, when [PERSON] [7] introduced a brand-new architecture consisting of three components (feature extraction coding layer, 4D correlation layer, and recurrent update operator) into the optical flow estimating domain, improved the AEPE to a significantly low number, and became the state-of-the-art. RAFT mainly addresses three drawbacks of previous optical flow estimation frameworks. First, previous frameworks generally adopted a coarse-to-fine design, i.e., estimating optical flow on a low-resolution feature map, then sampling and adjusting it on a high-resolution one. In contrast, RAFT maintains and updates a single but fixed optical flow field directly at high resolution, reducing the prediction error rate due to low resolution, the probability of missing small but fast-moving targets, and the iterations of training. Second, previous frameworks included some form of iterative refinement but did not restrict the weights between iterations, which led to a limit on the number of iterations. For example, IRR [8] used FlowNet5 [9] or PWC-Net as iterative units. The former is limited by the size of the network (up to 38 M parameters), so it can only be valid for five iterations; the latter is limited by the levels of the pyramid [10; 11]. As a new architecture, the update cycle of RAFT is periodic and lightweight, with mere 2.7 M parameters being used to update the operator, and it can be iterated more than a hundred times. Finally, the fine-tuningmodule in the previous framework usually used a common convolution or correlation layer [12]. Compared to this, the update operator designed by RAFT consists of a GRU (Gate Recurrent Unit) [13] structure, which contains some convolutional operations, and performs better on 4D multiscale correlation volumes. Therefore, in this paper, the RAFT architecture is chosen as the baseline. The GMA (Global Motion Aggregation) [14] module solves the problem of estimating the optical flow for some occluded pixels. It is optimized and improved from the RAFT method, and inspired by the Transformer architecture [15], which is based on the Self-Attention mechanism [16; 17] and has been sweeping the field of deep learning tasks in recent years. The architecture is known for its remarkable ability to model long-range dependencies, so it is more suitable for global estimating [18] compared to CNN, which has local limitations. The GMA module uses the projection of context features as query and key features in the Attention mechanism and the projection of motion features as value features, thus modeling the motion pixels that are occluded in the image in a globally aggregated way. However, the Transformer architecture has the well-known problem that it requires significant additional overhead for training and prediction [10]. Therefore, using the idea of the GMA global estimation module and modeling motion pixel aggregation as a reference, we design a dual-stream input structure and continuous-momentary optical flow feature fusion module with the historical buffer. The details of the structure will be presented in the next section. Streaming perception is mainly used in object detection tasks to detect and track multiple continuously moving targets, pioneered by [PERSON], [PERSON], [PERSON] et al. [19], who introduced this concept and derived relevant baseline and evaluation metrics, where it is worthwhile to refer to a meta-detector that draws on the use of a Kalman filter [20], decision scheduling theory [21], and asynchronous tracking [22; 23] to achieve streaming perception. [PERSON], [PERSON], [PERSON] et al.