StreamFlow: Streamlined Multi-Frame Optical Flow Estimation for Video Sequences

Authors: shangkun  sun
jiaming  liu
huaxia  li
guoqing  liu
thomas h li
wei  gao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-06 04:46:41

Abstract: Prior multi-frame optical flow methods typically estimate flow repeatedly in a pair-wise manner, leading to significant computational redundancy. To mitigate this, we implement a Streamlined In-batch Multi-frame (SIM) pipeline, specifically tailored to video inputs to minimize redundant calculations. It enables the simultaneous prediction of successive unidirectional flows in a single forward pass, boosting processing speed by 44.43% and reaching efficiencies on par with two-frame networks. Moreover, we investigate various spatiotemporal modeling methods for optical flow estimation within this pipeline. Notably, we propose a simple yet highly effective parameter-efficient Integrative spatiotemporal Coherence (ISC) modeling method, alongside a lightweight Global Temporal Regressor (GTR) to harness temporal cues. The proposed ISC and GTR bring powerful spatiotemporal modeling capabilities and significantly enhance accuracy, including in occluded areas, while adding modest computations to the SIM pipeline. Compared to the baseline, our approach, StreamFlow, achieves performance enhancements of 15.45% and 11.37% on the Sintel clean and final test sets respectively, with gains of 15.53% and 10.77% on occluded regions and only a 1.11% rise in latency. Furthermore, StreamFlow exhibits state-of-the-art cross-dataset testing results on Sintel and KITTI, demonstrating its robust cross-domain generalization capabilities. The code is available [here](https://github.com/littlespray/StreamFlow).

Main paper: 
 introduction: 
 Optical flow estimation, which aims to model the per-pixel correspondence between two consecutive frames, is a fundamental task in computer vision. It has various downstream video applications, such as video generation [22, 33, 20] , video editing [6, 51] , and video compression [25, 19] . In video streams, there is frequently a need for multi-frame input and continuous optical flow estimation.Nevertheless, previous multi-frame methods usually perform spatiotemporal modeling in a pairwise way, which leads to redundant computation, as depicted in Fig. 1 and Alg. 1. Given input frames I t-1 , I t , I t+1 , previous pairwise methods such as VideoFlow [38] only output one forward flow F t→t+1 . To derive the consecutive flow F t-1→t , one additional forward pass with the input of I t-2 , I t-1 , I t is needed.(b) The proposed SIM pipeline (a) Pairwise multi-frame pipeline This gives rise to the design of what we refer to as in-batch estimation -a pipeline that simultaneously predicts all successive unidirectional flows within one forward pass, which greatly reduces the redundant computation. For instance, given input frames I t-1 , I t , I t+1 , all consecutive flows F t-1→t and F t→t+1 could be derived within one forward pass. It directly saves one forward pass time in the decoder. When T frames are input simultaneously in the batch, there is still only one forward pass needed to obtain T -1 flows, and more time is served on average as T increases.I t-1 I t I t+1 F t-1,t F t,t+1 I t-2 I t-1 I t I t+1 F t-1,t F t,t+1Nevertheless, despite the acceleration brought by the in-batch estimation, its alteration of the pipeline renders it unable to directly apply past pairwise spatiotemporal modeling methods [16, 4, 38] . Notably, an effective spatiotemporal modeling approach is the key to resolving problems such as occlusions for multi-frame methods. Thus, this raises a new key question: Under the constraint of in-batch estimation, how to perform effective spatiotemporal modeling while maintaining efficiency?In this work, we propose StreamFlow, a streamlined multi-frame optical flow estimation method tailored for video inputs. StreamFlow is made efficient through the Streamlined In-batch Multi-frame (SIM) pipeline, which avoids repetitive calculations when predicting unidirectional flows for videos. Furthermore, StreamFlow explores the challenge of effectively modeling spatiotemporal cues under the constraint of non-overlapping in-batch estimation. StreamFlow explores various spatiotemporal modeling methods and ultimately derives simple yet highly effective methods: a parameter-efficient Integrative spatiotemporal Coherence (ISC) module during encoding, and a lightweight Global Temporal Regressor (GTR) to decode all flows. Benefiting from these modules, StreamFlow achieves remarkable performance on Sintel and KITTI datasets without self-supervised pre-training and the aim of bidirectional flows. Moreover, StreamFlow attains state-of-the-art cross-dataset generalization results with comparable efficiency compared to two-frame methods, as illustrated in Figure 2 .In summary, our contributions are as follows: (1) We propose a Streamlined In-batch Multi-frame (SIM) pipeline for optical flow estimation, which eliminates the repetitive overlapping computation when computing forward flows for video inputs. (2) Under the constraint of a non-overlapping pipeline, we specifically designed the Integrative spatiotemporal Coherence (ISC) method to perform spatiotemporal modeling without additional parameters. (3) For the SIM pipeline, we devise a Global Temporal Regressor (GTR) during decoding to further exploit temporal cues with modest additional computation cost. (4) The proposed StreamFlow achieves superior performances on multiple benchmarks, particularly in occluded regions with comparable efficiency compared with two-frame methods, resulting in substantial improvements in optical flow estimation. 
related work: 
 Two-frame optical flow. Optical flow estimation in the form of a supervised learning task has been performed by FlowNet [8] using Convolutional Neural Networks (CNN). The encoder-decoder architecture of FlowNet predicts flow from coarse to fine using the hierarchy of the flow pyramid.Thereafter, a number of refined coarse-to-fine approaches [12, 42, 43, 10, 11, 50, 54, 13] emerged.The flow pyramid is constructed for the coarse-to-fine approach, which predicts the flow based on the flow guidance at a higher pyramid level. However, the flow guidance is often too coarse to capture small motions delicately and creates errors in later estimation. RAFT [45] recently introduced an iterative all-pairs flow transform technique, which enables the prediction of high-resolution flow and recurrent refinement of the residual flow estimation. It addresses the challenges of small motions and has consequently received high interest and performance in the field, inspiring numerous follow-up works [15, 28, 55, 21, 27, 56, 47] . To further address the occlusion issue, SKFlow [44] begins by expanding the spatial receptive field and designing effective large convolution kernel modules in the decoder of the flow network, with modest computational cost. Although StreamFlow is based on the SKFlow framework, it is designed to explore additional temporal cues and minimize redundant computations that are prevalent in earlier multi-frame methods. Their methodologies and foundational principles differ significantly. Multi-frame optical flow. Exploiting temporal cues in optical flow estimation is an effective way to recover the occluded motion. Previous works [37, 46, 32, 1, 16, 4, 38, 26] propose various approaches to fuse temporal cues, such as leveraging previously predicted motion feature, optical flow, or contextual information. For instance, ContinualFlow [32] uses previous flow priors to estimate the current occlusion map. STaRFlow [1] passes extracted features in multiple scales, jointly with occlusion maps. [45] proposes a warm-start strategy to initialize the original flow with the past flow before prediction. MFCFlow [4] and MFRFlow [16] propose to leverage previously estimated motion features during decoding via feed-forward CNNs and self-similarity modeling, respectively. Nevertheless, these methods obtain a pairwise strategy when handling video sequences, which divide the input sequence into lots of overlapping groups and take huge repeated computations. Splat-Flow [46] utilizes the differentiable splatting transformation to explore temporal cues. TransFlow [26] decodes all flows simultaneously and achieves impressive results. However, it needs self-supervised pre-training on the flow datasets to help the temporal modeling modules converge. Besides, its pure transformer architecture does not have advantages in time. VideoFlow [38] employs TROF and MOP modules to utilize multi-frame temporal cues and bidirectional optical flow to effectively mitigate occlusion issues. Nevertheless, it still follows the pairwise method to predict multiple unidirectional flows with the cost of predicting bidirectional flows. Differently, StreamFlow is proposed to avoid redundant, overlapping computation for consecutive unidirectional flow predictions while exploring efficient and effective temporal modules design under such a pipeline. It addresses the redundancy problems previous pairwise methods including VideoFlow encountered, and achieves excellent accuracy with latency similar to some two-frame methods. Initialize i = 1, stride= T -1. repeat F i , ..., F i+stride-1 = M odel(I i , ..., I i+stride ) i = i + stride until i + stride > N j = N -stride F j , F j+1 , ..., F N = M odel(I j , I j+1 , ..., I N ).In this Section, we introduce StreamFlow, an efficient and effective in-batch framework for multi-frame optical flow estimation. Extensive experiments guided our specific design of each module in Stream-Flow. The key components of StreamFlow consist of three parts: (1) The Streamlined In-batch Multi-frame (SIM) pipeline for efficient multi-frame estimation, which contributes to the speed improvement of StreamFlow. (2) Integrative spatiotem-poral Coherence (ISC) modeling, which is parameter-efficient and is specifically designed for spatiotemporal modeling in the encoder. (3) Global Temporal Regressor (GTR), which is quite lightweight and learns temporal relations during decoding. We will first give an overview of our methods in Section 3.1, and then introduce each module in Section 3.2, Section 3.3, and 3.4, respectively. In the end, we discuss the supervision in Section 3.5.Iterative GTR 0, 0, 0MotionUpdater Spa. EQUATIONm T i c k,k+1 f k i-1 e k , t k SplitQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Please refer to Section 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
overview: 
 The overall framework of StreamFlow is illustrated in Figure 3 . For the basic encoder and decoder, similar to VideoFlow [38] , StreamFlow adopts the Twins transformer [5] as the encoder and utilizes the motion encoder and updater in SKFlow [44] during decoding. The overall iterative-refinement design that adopts an iterative decoder is the paradigm proposed in IRR [12] and followed by a lot of subsequent works [44, 41, 15, 44, 9, 39] . Input frames are first passed to two feature encoders that share the same architecture to extract the correlation feature and contextual feature, respectively. Then, the multi-scale all-pairs correlation volume is calculated based on the correlation feature. Namely, given feature embeddings e 1 and e 2 from the target frame and the reference frame, respectively:EQUATIONwhere the derived c l (i, j, m, n) is the average over the correlation in the local 2 l × 2 l window. l denotes the l th correlation level. u and v are the horizontal and vertical pixel motions, respectively. ⟨, ⟩ refers to the dot product function. In summary, c l (i, j, m, n) means the cost volume vector of e 1 and e 2 pooled with the 2 l × 2 l kernel.Then, the iterative decoder refines the flows via several updates. As depicted in Figure 3 , flows are initialized to zeros. The derived multi-scale correlation volume, extracted context feature, and the initialized flows are passed to the decoder, and then the refinement is conducted.As illustrated in Figure 1 , previous pairwise multi-frame networks typically perform redundant computations for video inputs, resulting in substantial computational overlap. We have briefly discussed this issue in the first two paragraphs of Section 1. Specifically, given N frames {I In other words, only one forward pass is needed to derive T -1 flows. From this process, we could learn that the upper limit of the saved time for the SIM pipeline is approximately α T -1 , where α represents the computational time of the decoder. To approach the theoretical limit as closely as possible, a lightweight yet effective method of spatiotemporal modeling is crucial. This is precisely the key contribution that StreamFlow adds to the SIM pipeline, which will be discussed in detail in Section 3.3 and Section 3.4. Additionally, when directly comparing StreamFlow with other multi-frame methods in practice, we find that the speed improvements with StreamFlow far exceed the theoretical limits calculated above. This is because StreamFlow inherently includes a memory bank mechanism in its encoder, whereas previous multi-frame methods required special additional implementation to utilize Memory banks for caching features. Nevertheless, even when compared to methods that include a memory bank, StreamFlow still achieves significant speed advantages due to its efficiency in the decoder, as demonstrated in Figure 2 and Appendix A.1.During the encoding process, we propose an Integrative spatiotemporal Coherence (ISC) modeling method, especially for the SIM pipeline. Our design principles for temporal modeling modules in the decoder encompass two facets: firstly, adherence to the design criteria of the SIM pipeline, with a focus on minimizing pair-wise overlap operations, such as the computation of cross-frame attention between every pair of consecutive frames. Secondly, the modules should be efficient enough and not impede the overall speed of the network. Therefore, we design the ISC method, which introduces no additional parameters while learning spatiotemporal relations efficiently and effectively. The ISC method inherently takes the original modules in Twins. Specifically, after deriving patch embeddings from consecutive frames, patches from different frames are reorganized along the spatial dimension. Subsequently, it models the derived spatiotemporal graph using self-attention mechanisms and feed-forward layers in Twins, which can be formulated as,EQUATIONEQUATIONEQUATIONwhere f (•, •) is the attention function which conducts dot-product and softmax operation, x j t,d is the j th vector along spatial dimension at channel d of the t th frame. I denotes the integration operation, which integrates temporally contiguous multiple input embeddings into a large feature embedding along the spatial dimension. q, k and v is the derived query, key, and value vector. W proj is the projection matrix. By leveraging the derived spatiotemporal graph, the spatial and temporal relations are learned effectively, and no additional parameters are involved.As for the decoder, we propose a Global Temporal Regressor (GTR) to predict and refine flows. Compared with the previous widely used decoder [45, 15, 44, 28, 53, 27] , GTR introduces the temporal modeling module to exploit temporal cues from consecutive frames. Different from VideoFlow [38] that concatenates motion features along a temporal dimension and implicitly learns temporal relations or TransFlow [26] that applies a transformer symmetric to the encoder, the core of GTR is super convolution kernels [44] and a lightweight temporal transformer block. The input correlation volume, initialized flows, and contextual features are first passed into a motion encoder to derive motion features and then extracted for spatiotemporal features, which can be formulated as:EQUATIONEQUATIONEQUATIONEQUATIONEQUATIONEQUATION)where m k i is the derived motion feature of frame k at the i th update and f k i-1 denotes the flow of frame k after i -1 th refinement. c k,k+1 denotes the correlation volume between frame k and k + 1. M denotes the motion encoder which is the same as that in SKFlow [44] . r i denotes the temporal feature embedding extracted from the motion features of all frames. Notably, the caching mechanism of the MemoryBank is employed, thus necessitating the calculation of r i only once for different frames. F is a lightweight temporal-learning layer that consists of temporal attention and feed-forward layers. e k refers to the feature embedding of frame k. Note that e k and c k,k+1 are not updated during the refinement. A denotes the spatial cross attention inspired by [15] , but takes m k i and e k as the input. C represents the concatenation operation and U refers to the motion updater. t k i denotes the extracted contextual information, which will be updated during each refinement. In practice, the decoder estimates the residual of flow ∆f k i . And the final flow f k is updated via ∆f k i during each refinement.StreamFlow adopts the overall loss in the same group as the total loss function. For each flow, StreamFlow adopts the same loss function as successful two-frame networks. Namely, the weighted sum for the predicted flows at different refinements. During both the training and the fine-tuning process, the supervision can be formulated as follows:EQUATIONwhere f k i refers to the flow of frame k at the i th refinement. T and N are the number of frames and refinements, respectively. θ denotes the weights on corresponding estimated flows. f gt is the ground truth flow and || • || 1 means the l 1 distance between ground truth and our predicted flow. In practice, N is set to 12, θ is set to 0.8, the same as previous works [38, 45, 44, 15] for a fair comparison.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] Justification: The aim of this work is to enhance the utilization of machine learning to estimate more accurate inter-frame motion. We feel this work would not bring direct societal consequences.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experiments: 
 Experimental setup. In this study, we evaluate StreamFlow on the Sintel [3] , KITTI [31] , and Spring [30] datasets, following previous works [44, 9, 7] . In previous works, models are initially pre-trained on the FlyingChairs [8] and FlyingThings [29] datasets using the "C+T" schedule and then are subsequently fine-tuned using the "C+T+S+K+H" schedule on Sintel and KITTI datasets. After "C+T", the cross-dataset generalization tests are often performed, and the models from this stage are frequently used for various cross-domain video downstream tasks [18, 23] . In "+S+K+H", the evaluation typically focuses on intra-domain generalization capabilities. In specific, for Sintel, models are trained on a combination of FlyingThings, Sintel, KITTI, and HD1K [17] . Models are then trained on the KITTI dataset for KITTI evaluation and on the Spring dataset for Spring evaluation.Implementation details. Our StreamFlow method is built with PyTorch [34] library, and our experiments are conducted on the NVIDIA A100 GPUs. During training, we adopt the AdamW [24] optimizer and the one-cycle learning rate policy [40] , following previous works [45, 15, 44] . The number of refinements in the decoder is set to 12, following previous works. Given the absence of multi-frame data information in the Chairs dataset, we follow VideoFlow [38] to directly train on the FlyingThings in the first stage. For the Spring dataset, we follow the settings of MemFlow [7] and fine-tune the model for 180k steps. The remaining training configurations are consistent with prior works [38, 44, 15, 45] . The temporal and non-temporal modeling modules are concurrently trained. [45] that use the previous flow for initialization. Bold and underlined metrics denote the method that ranks 1st and 2nd, respectively.From Table 1 and Table 4 .1, we can learn that StreamFlow achieves advanced 0-shot performance on Sintel and KITTI. Compared to previous methods, StreamFlow reduces the 0-shot end-point error by 0.16 and 0.08 on the challenging Sintel clean and final pass, respectively. On KITTI, StreamFlow outperforms the previous state-of-the-art 0-shot results with 0.11 and 17.65% lower EPE and Fl-all metric. Besides, without self-supervised pre-training or bi-directional flows, StreamFlow attains commendable accuracy and efficiency on the challenging Sintel and KITTI test benchmarks using (C)+T+S+K+H schedule. On the challenging Spring [30] dataset, StreamFlow also achieves enhanced performance both before and after fine-tuning. A detailed analysis on its shortcomings is in Section 5.In this section, we validate if StreamFlow could help improve the performance on the occlusions. We compare StreamFlow with its base two-frame model Twins-SKFlow, which strengthens SKFlow [44] with the Twins [5] encoder. Evaluations are conducted on the matched and unmatched areas of the Sintel test dataset. The matched areas denote regions visible in adjacent frames and the unmatched areas refer to regions visible only in one of two adjacent frames. Our models are trained using the T+S+H+K schedule. We could learn that StreamFlow attains remarkable improvements on occluded areas, as shown in Table 3 . We also visualize the performance on occluded regions, which are shown in the supplements. 3 : Occlusion analysis on Sintel test set. Unm. and Mat. denote performance on unmatched and matched areas, respectively. "Baseline" denotes our baseline method Twins-SKFlow. regions, respectively. From the table, we can learn that StreamFlow improves not only the flow estimation in unmatched regions but also the estimation in matched regions.In this section, we verify the effectiveness of StreamFlow designs, as shown in Table 4 . For a fair comparison, all models are trained with the same settings on the FlyingThings dataset. Then we evaluate each method on Sintel and KITTI. Below we will introduce each experiment in detail.Baseline (Twins-SKFlow) StreamFlow (Ours) SIM pipeline. We test the efficiency of the vanilla pairwise pipeline and our SIM pipeline. Pairwise methods utilize multi-frames to predict the flow of the current two frames and bring substantial redundant computation, while the SIM pipeline estimates multiple flows concurrently and minimizes the overlapping calculation. As shown in Table 4 , the SIM pipeline brings great gain in efficiency.Notably, there might be information loss for the SIM pipeline due to different frame distances. For instance, given the frames I t-1 , I t , I t+1 , the estimated flow F t-1→t is typically less accurate than that derived from the sequence I t-2 , I t-1 , I t . However, this issue tends to be alleviated with longer sequences. When the number of frames increases to 4, the impact on accuracy is significantly reduced. We will discuss this issue in more detail in the appendix.Temporal modules. In this part, we explore the performance and efficiency of different temporal modeling methods in the flow encoder. Temporal attn refers to applying a temporal attention layer after each spatial self-attention modeling in Twins. Pseudo conv [36] denotes stacking 1D convolution layers in the temporal dimension to imitate 3D convolutions at minimal cost. We also apply 3D convolutions at the end of the flow encoder to learn temporal relations. As shown in Table 4 , our ISC module achieves a good trade-off between efficiency and effectiveness. The improvements achieved by other methods are not as pronounced. We hypothesize that the limited volume of optical flow data impedes the efficient training of the spatiotemporal module from scratch to accomplish good optimization. For comparison, VideoFlow does not apply temporal modeling modules in the encoder, and TransFlow [26] applies self-supervised pre-training for better optimization.Extra parameters. In this part, we aim to determine whether the performance gain is due to the additional parameters or the effective temporal modeling method. To this end, we introduce the additional parameters by widening the baseline network. Namely, we extract higher-dimension features along the spatial dimension and concatenate them with the original motion feature. All models in this section are equipped with the ISC module. "w/o" denotes the baseline Twins-SKFlow network. "w" means adding additional parameters. "Ours" denotes the method equipped with our temporal modeling modules. Results show the improvement achieved by simply adding more parameters is minor, and the performance gain is primarily attributed to the effectiveness of StreamFlow modules. GTR module. We also examined whether the GTR module could enhance flow predictions. "w/o" means applying vanilla SKFlow decoder while "w" denotes using GTR. All models in this part utilize the ISC module in the encoder. Table 4 demonstrates the necessity of incorporating the GTR, which could achieve stable improvement on multiple benchmarks. We could learn that GTR especially helps the estimation on the challenging final pass, with the performance gain of 0.14. In supplements, we give a more detailed discussion of its initialization, which is key to its training process.ISC module. We verify the effectiveness of the proposed ISC module. All models in this part adopt GTR as the flow decoder. From Table 4 , we could learn that the ISC module is efficient and effective in temporal modeling and greatly contributes to the improvement of the pipeline. It introduces no additional parameters and a modest increase in runtime, while significantly boosting the performance.# Frames. We delve into the influence of different numbers of frames, as illustrated in Table 4 .We set the number of frames to 4 due to limitations in GPU memory. From an efficiency standpoint, augmenting the number of frames results in a higher proportion of redundant computations eliminated by StreamFlow, consequently leading to a more substantial improvement in processing time. Although there is an increase in the parameter count for temporal modeling, the efficiency is further enhanced in the context of four input frames due to a reduced proportion of redundant computations, resulting in a shorter average prediction time per frame compared to the three-frame setting.We demonstrate visualization results on both synthetic (Sintel [3] ) and real-world scenes (KITTI [31] ), as shown in Figure 4 . In the supplements, we also show the visualizations on the real-world dataset DAVIS [35] . Our models are pre-trained using the T+H+S+K schedule. We could learn that StreamFlow could still achieve remarkable qualitative results when generalized to real-world scenes.We evaluate the efficiency of the StreamFlow in terms of runtime and parameter counts. Our experiments were conducted on NVIDIA A100 GPUs. Models are trained using the (C+)T schedule and evaluated on the Sintel dataset. The runtime is measured as the average inference time per frame of five runs on the Sintel training set. We could learn StreamFlow achieves comparable efficiency with state-of-the-art two-frame methods while achieving superior performance. The key to maintaining high efficiency is its SIM pipeline. StreamFlow does not perform pairwise redundant computation and predicts all flows simultaneously. Another reason for the high speed is its CNN-based decoder. We could learn that StreamFlow is much faster than the pure two-frame transformer architecture FlowFormer. Besides, the specially designed lightweight temporal modules also contribute to the performance, simultaneously aiding in better results compared to the 2-frame baseline Twins-SKFlow. 
limitations: 
 StreamFlow faces two primary challenges: (  
conclusion: 
 In this work, we proposed StreamFlow, a multi-frame optical flow estimation approach proficient in estimating optical flows across multiple video frames using efficient spatiotemporal relationship mining. StreamFlow aims to estimate consecutive unidirectional optical flows with less overlapping computation. It proposes to estimate multi-frame optical flows via the proposed SIM pipeline and introduces efficient and effective ISC and GTR methods for temporal modeling under such circumstances. Extensive experiments on multiple challenging benchmarks demonstrate the efficiency and effectiveness of the proposed StreamFlow method. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This work poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All datasets in this work are cited and used properly.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: As discussed in Section 3.2, to better demonstrate that the temporal advantage of StreamFlow is not solely due to the memory bank, this section explores the efficiency comparison between StreamFlow and other methods when using a memory bank. Given that the model's runtime is closely related to the coding implementation, this comparison prioritizes officially open-sourced multi-frame optical flow methods. However, as of the writing of this paper, the choices for leading open-source multi-frame methods are quite limited, and thus VideoFlow [38] was selected for comparison. The experimental setup and the machine are consistent with those described in Section 4. Within the SIM pipeline, the impact of different frame distance D indeed exists. For instance, the flow F t-1,t derived from I t-2 , I t-1 , I t tends to be more accurate than that from I t-1 , I t , I t+1 due to the different distance D t-1,t-2 and D t-1,t+1 . As the frame distance increases, the information provided may decrease, as confirmed by the results in Table 7 . However, this impact might not necessarily grow larger with more frames, which could be due to the affected frames mainly being distributed at the head or the tail of a group. We could define the longest frame distance that provides effective information as m. As the length of the group increases, there will be more frames in the middle of the group (i.e., frames I t within the interval [I t-m , I t+m ] all lying within the group), and fewer frames distributed on both ends. As shown in Table 7 , the impact was weakened with 4 frames. Maybe the future study on an appropriate choice of m is helpful for future multi-frame optical flow work.Table A .4 presents a more detailed comparison on the Sintel test set between StreamFlow and its two-frame baseline. The detailed metrics including all/matched/unmatched EPE, d0-10, d10-60, d60-140, s0-10, s10-40, s40+ are from the official Sintel website. We could learn that StreamFlow performs exceptionally well in unmatched areas, validating its effectiveness in addressing occlusion issues. Compared to the baseline method, it improved by 15.53% and 10.77% in the Matched areas on the clean pass and final pass, respectively, and by 15.56% and 11.83% in the Unmatched areas. Within different ranges of occlusion boundaries, it showed improvements of 15.13% and 10.30% at d0-10, 15.79% and 13.51% at d10-60, and 10.00% and 12.50% at d60-140 on the clean pass and final pass, respectively. For pixels moving at different speeds, it improved by 16.00% and 14.58% at s0-10, 10.39% and 10.14% at s10-40, and 14.41% and 10.49% at s40+ on the clean pass and final pass, respectively. It is shown that StreamFlow has outperformed its baseline on both the clean pass and final pass, with a more pronounced improvement on the clean pass. The enhancements are particularly noticeable in areas of occlusion and for objects moving at high speeds.In this section, we show the GPU usage of StreamFlow. During the inference, without the optimization in the third-party package, StreamFlow takes about 2.4 G and 3.3 G for 432 × 1024 inputs with the number of frame set to 3 and 4, respectively. With PyTorch 2.2 and flash-attention, using 12 refinements and 4 frames, the GPU memory usage for StreamFlow is shown in Table ??. Specifically, when the image size increases by 4 times, the GPU memory usage increases to nearly 4.4 times. When the image size is increased by 9 times, GPU memory usage grows to nearly 20 times. For most current scenarios, StreamFlow still maintains a relatively moderate use of GPU memory. The variation in GPU memory usage may be influenced by underlying optimizations in the framework. We believe the memory usage could be further optimized in the future.A.5 Qualitative analysis on real-world scenesIn this section, we facilitate our visualizations and evaluations using two prominent real-world datasets, namely DAVIS [35] . The DAVIS dataset, short for Densely Annotated VIdeo Segmentation, is a widely recognized benchmark in the field of computer vision. It comprises high-quality video sequences captured in diverse scenarios, encompassing a broad range of challenging visual conditions such as occlusions, motion blur, and dynamic object interactions. The dataset provides pixellevel annotations for every frame, facilitating precise evaluation and comparison of various video segmentation methods. The visualizations on the DAVIS dataset are shown in Figure 5 . Our model is pretrained using the "T" and "T+S+H+K" schedule and then fine-tuned on KITTI [31] . "T" denotes the FlyingThings [29] dataset and "T+S+H+K" refers to the combination of the FlyingThings, Sintel [3] , HD1K [17] , and KITTI datasets. Then we infer our models on the DAVIS dataset. The number of refinements is set to 12. The number of input frames for each non-overlapping group is 3. We could learn that StreamFlow demonstrates remarkable adaptability across real-world datasets, showing its robust performance in challenging scenes for optical flow estimation. This is particularly evident in scenarios such as the occlusion of the bear's hind legs in the first row, first column, and the small motion of the small tennis ball in the last column. Additionally, it can be observed that in the motion captured in the first row, second, and third columns, the hind legs of the camel and the leg movements of the dancer are also vividly delineated. These instances reaffirm its efficacy in diverse and demanding environments for optical flow estimation.In this section, we focus on the performance of the occluded regions. As discussed in previous works [15, 44] , here we term occlusions as areas where pixels appear in the current frame while disappearing in the next frame. We visualize the flow-error map on occluded regions of the Sintel dataset with the official occlusion masks. All models are trained using the (C+)T schedule. As shown in Figure 6 , significant occluded areas are highlighted using red boxes. A darker color in the flow-error map denotes a more significant error. We could learn that StreamFlow achieves better overall performance, and attains leading performance on the occluded regions. In this section, we investigate the impact of different GTR initialization methods. Previous works in spatio-temporal modeling such as [2] have suggested initializing the temporal modules with zero values. We employed two distinct initialization approaches, namely zero initialization and PyTorch's default initialization, and the corresponding results are presented in Justification: Please refer to the last paragraph in Section 1.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: The paper presents a method for optical flow estimation in videos that overcomes the need for repetitive overlapping computation in multi-frame optical flow. All the reviewers appreciate the results as state-of-the-art. Evaluations requested by zwXh on the Spring benchmark are provided in the rebuttal and support the effectiveness of the proposed method. The motivation of the proposed method is also clarified in the rebuttal noting that the cost volume computation avoids redundancy. The strength of results are also noted by PByw, whose request for discussion with respect to SKFlow and VideoFlow are addressed by the rebuttal. While ba6H is supportive of acceptance, additional results are requested and sufficiently provided by the rebuttal. Overall, the AC agrees with the majority opinion that the paper is well-motivated, proposes a novel approach and presents strong results. The authors are encouraged to release code and checkpoints as indicated by the rebuttal to zwXh, while also making the improvements to presentation listed there. In balance, the paper may be accepted for NeurIPS.

Reviews: review: paper_summary: paper_summary: The paper presents StreamFlow, a new optical flow estimation method tailored for video inputs. StreamFlow differentiates itself from earlier methods by incorporating a streamlined in-batch multi-frame pipeline that reduces duplicate computations across consecutive frames, thereby enhancing efficiency. Additionally, the introduction of an ISC module and a GTR decoder allows for effective leverage of spatio-temporal information without an increase in parameter count. Extensive experiments demonstrate that StreamFlow surpasses existing methods on the Sintel and KITTI datasets in both accuracy and efficiency.
strength_weakness: strength_weakness: - The literature review in Section 2 appears insufficient. It should be expanded to include more detailed discussions of related works such as SKFlow and Videoflow, upon which the architectural designs of this paper are based.
- As I understand it, the primary motivation of this research is to design a non-overlapping inference pipeline for multi-frame estimation. This raises a question about how the non-overlapping inference affects accuracy in terms of frame distance, a topic that seems to be omitted from the paper. For instance, when given three input frames [t-1, t, t+1], a pair-wise pipeline might estimate flow f_t,t+1, and then f_t+1,t+2 using [t, t+1, t+2]. In contrast, if the SIM pipeline computes f_t,t+1 using [t-1, t, t+1], it then predicts f_t+1,t+2 using [t+1, t+2, t+3], thereby missing the opportunity to utilize frame t in estimating the flow for f_t+1, t+2. I would expect this loss of information may become more severe with longer frame lengths. Such analysis should be included in the first row of Table 3 (ablation of SIM Pipeline), yet the ablation study was conducted naively, focusing solely on latency without considering factors that affect accuracy.
questions: questions: Please check the weaknesses above.
limitations: limitations: The authors described limitations adequately.
paper_summary: paper_summary: This work focuses on the task of multi-frame optical flow estimation. It challenges the traditional pair-wise flow estimation approach in multi-frame scenarios, which involves redundant calculations. To address this issue, a new framework is proposed that takes multiple frames as input and predicts successive unidirectional flows in a single forward pass.
strength_weakness: strength_weakness: - The paper is challenging to follow due to several presentation issues. 
Specifically, the `Integration’ operation used in Eq. 2 is not clearly defined. Additionally, from Eq. 5 to Eq. 9, using mathematical notations rather than component names to represent the model components in GTR would improve clarity and conciseness. Furthermore, the main figure requires more detailed captions to enhance comprehensibility.

- Lack of further evaluations. 
To enhance the validation of the proposed method, the author should also test their model’s generalizability on the widely used Spring [1] benchmark, as it is a standard in many recent optical flow estimation studies.

- Unclear Core Motivation. 
In my understanding, the core motivation of this paper is to enable the `simultaneous prediction of successive unidirectional flows in a single forward pass`, as opposed to making per-pair estimations in multi-frame flow estimation scenarios. However, I observed that there are numerous pair-wise operations among the inputs, such as cost volume calculations, in the `single forward pass`.
I suggest that the authors allocate less space to describing these pair-wise operations and instead focus more on discussing and analyzing their designs that enhance efficiency in the `single forward pass`. For example, providing a detailed analysis of the time complexity of the proposed method compared to other methods would be beneficial.
Additionally, it would be helpful for the authors to include comparisons and discussions regarding recent multi-frame flow estimation methods, such as MemFlow [2].


[1] Spring: A High-Resolution High-Detail Dataset and Benchmark for Scene Flow, Optical Flow and Stereo.

[2] MemFlow: Optical Flow Estimation and Prediction with Memory.
questions: questions: Please see the "Weaknesses" section for my questions and suggestions. If the author can address these concerns, I would be willing to consider raising my rating.
limitations: limitations: Yes.
paper_summary: paper_summary: The paper proposes a multi-frame optical flow estimation model with a novel flow decoder that estimates a flow output for all frames simultaneously. The paper describes this process as  a Stream-lined In-batch Multiframe (SIM) pipeline and argues that this leads to efficiency gains when processing video as each multi-frame batch needs to contain each video frame only once, i.e., the video processing can advance by multi-frame batch instead of frame-by-frame. The decoder is termed Global Temporal Regressor and uses an iterative design as common in many recent optical flow methods. In order to estimate multi-flow output the decoder uses a correlation volume for each pair of input images. The first step of the decoder is the output of a "MotionEncoder" which uses the current flow estimate and the correlation volume. These motion features form the input to a temporal feature encoder which is implemented with the Super Convolution Kernels by Sun et al., 2022. Spatial image features are than matched with a cross attention module and also input in the updater for the flow. The paper explains the stacking of feature over time and using the twins transformer for encoding temporal relationship in their Integrative Spatio-temporal Coherence (ISC) modeling. The paper reports experiments with the KITTI and MPI-Sintel datasets and two training schedules. The tables show that the proposed StreamFlow achieves either SOTA or close to SOTA performance. The paper reports that A100 GPU with 40GB of memory is enough to train either the method to work with 3 or 4 images as input to the batch. The paper reports a more detailed breakdown of Sintel results and an ablation study is provided.
strength_weakness: strength_weakness: The multiple input - multiple output processing does only partially address the high computation cost of recent optical flow methods. In particular the high memory costs limits the use of StreamFlow to non-edge devices. Considering that optical flow is a low-level visual cue, this prevents StreamFlow to be used as an optical flow sub-module in other tasks. 

The paper overstates the performance gains of the method. In particular, considering the full training schedule StreamFlow does not reach top performance on MPI-Sintel and in Kitti. While the table shows top performance on Kitti, the table does not include the on-line videoflow MOF method which achieves an F1-all score of 4.08 (Section 3 of VideoFlow supplemental material).

The paper does not conduct any experiments with data other than MPI-Sintel and Kitti. The recent large image Spring dataset by Mehl et al., 2023 would be a good choice.   

The authors did not include consideration of MemFlow, CVPR 2024 by Dong and Fu which has been available on arXiv since April 2024. As one may argue that this is parallel work, I think their superior results raise questions about the design choices for the proposed method.  

Unrolling time into space in the ISC module seems to be a fairly standard trick. It is hard to think of it as a novel contribution, even though this may be the first time that it has been used in optical flow.
questions: questions: Is there a relationship between image size and memory requirements and if so can the method be used to process large-scale images, e.g., in Spring?

The MPI-Sintel datasets makes numerous measures for detailed analysis of the flow results available. Can these measures be listed and discussed in the supplemental material?

The efficiency analysis makes claims about the efficiency of StreamFlow but no such data is provided. The only information included seems the number of parameters and the latency of the method. No comparison to other methods is given except with VideoFlow-BOF on latency in the Table 4 (Appendix A.1).

The training EPE is very low. Does this indicate overfitting?

The paper needs some careful proof-reading:
l.27 The consecutive flow goes backwards in time. Please check notation.
l.48 "the aim of bidirectional flows" cannot be understood.
l.83 "features from different in multiple scales" missing word.
l.122 "iterative decoder is the paradigm proposed in RAFT" has been proposed earlier by Hur and Roth in their IRR-PWC, 2019 paper.
l.177,184,200 "could" -> can
Eqn. 2 is very confusing by typing Integration and using superscript and subscript for the time range. Needs better notation.
l.173 jth should be j^{th} (superscript).
Table 2. TransFlow has lower EPE on Matched-clean.
l. 244 Abaltions -> Ablations
l. 306 "it significantly increases during training." fragment, please complete the sentence.
[33] should be updated to the peer-reviewed reference.
limitations: limitations: The paper contains a limitation section that raises two relevant limitations of the proposed method: the large memory usage for training and the lack of temporal connections between batches.

Comments: title: Response from Authors
Rebuttal
comment: I thank the authors for their rebuttal and the additional results provided. While there is no compelling reason to oppose publication, and I acknowledge the contributions of the paper, I do not find them to be particularly significant. Therefore, I will maintain my current rating.
Dear Reviewer,

Thank you once again for taking the time to review our manuscript. We have tried our best to address the questions you raised (please see our responses in the top-level comment and above) and have revised the paper according to the suggestions provided by all reviewers. 

Please kindly let us know if there are any additional questions requiring further clarification. Your feedback is highly valued, and we are more than willing to provide any further information that may be helpful.
Dear Reviewer,

Thank you once again for taking the time to review our manuscript. We have tried our best to address the questions you raised (please see our responses in the top-level comment and above) and have revised the paper according to the suggestions provided by all reviewers. 

Please kindly let us know if there are any additional questions requiring further clarification. Your feedback is highly valued, and we are more than willing to provide any further information that may be helpful.
Dear Reviewer,


Thank you very much for your valuable and prompt feedback. We sincerely appreciate the time and effort you have invested in reviewing our work and highlighting areas that require further clarification. We are eager to address your concerns as follows:

**Q1: Relationship between image size and memory usage.**

A: With PyTorch 2.2 and flash-attention, using 12 refinements and 4 frames, the GPU memory usage for StreamFlow is as follows. Specifically, when the image size increases by 4 times, the GPU memory usage increases to nearly 4.4 times. When the image size is increased by 9 times, GPU memory usage grows to nearly 20 times. For most current scenarios, StreamFlow still maintains a relatively moderate use of GPU memory. The variation in GPU memory usage may be influenced by underlying optimizations in the framework. We believe the memory usage could be further optimized in the future.



| Image Size | GPU Memory |
| ------------ | ------------ |
| 360x640    | 1.19 G     |
| 720x1280   | 5.20 G     |
| 1080x1920 | 24.11 G|

**Q2: All evaluation parameters for MPI-Sintel.**

A: We have collected all measures including ``all/matched/unmatched EPE, d0-10, d10-60, d60-140, s0-10, s10-40, s40+`` in Table 2 of the attched PDF. Compared to its two-frame baseline, StreamFlow performs exceptionally well in unmatched areas, validating its effectiveness in addressing occlusion issues. We would add a more detailed analysis for each metric in the supplements.

**Q3: Relative accuracy of StreamFlow.**

A: Thank you for your insightful comments and for pointing out concerns about the descriptions in this work. Our initial intention is to highlight its notable 0-shot cross-dataset generalization results, and we hope it did not come across as overstating its performance. We will refine the expression in the revised version to minimize potential misunderstandings. For instance, we will revise the content between Lines #223 and #230 to: ``From Table 1, we could learn that StreamFlow achieves excellent 0-shot cross-dataset performance on Sintel and KITTI. Compared to previous methods, StreamFlow reduces the 0-shot end-point error by 0.16 and 0.08 on the challenging Sintel clean and final passes, respectively. On KITTI, StreamFlow surpasses the previous 0-shot results with 0.11 and 17.65% lower EPE and Fl-all metrics. Besides, without self-supervised pre-training or bi-directional flows, StreamFlow attains commendable accuracy and efficiency on the challenging Sintel and KITTI test benchmarks using (C)+T+S+K+H schedule.``

Besides, StreamFlow indeed uses experimental settings different from others. However, this may also highlight StreamFlow’s advantages. For instance: (1) On the Spring dataset, StreamFlow was only trained for 180k iterations while outperforming MemFlow, which was trained for 400k iterations with the same batch size and learning rate. (2) For Sintel testing, StreamFlow was trained for 300k iterations on FlyingThings and 180k on T+S+H+K, while MemFlow was initially trained for 120k (FlyingChairs) and 150k (FlyingThings) in a 2-frame setting, followed by 600k iterations on FlyingThings and 600k on T+S+H+K. Despite this, StreamFlow still delivers good results, especially on Spring. (3) As for VideoFlow, it employs more frames and bidirectional flows for training, achieving excellent results on the KITTI dataset. We will update the result in Table 1 and add the related discussion. However, VideoFlow explores the accuracy and efficiency under bidirectional flow estimation, which differs from the focus of StreamFlow on a non-overlapping, continuous unidirectional flow pipeline. Additionally, its latency is significantly higher than that of StreamFlow. Therefore, A more relevant comparison for highlighting the issues that StreamFlow addresses is with the baseline method, Twins-SKFlow.

In the end, we greatly appreciate your valuable comments and the opportunity to clarify these points. Please kindly let us know if you have any further questions or require additional clarification. We highly value your insights and stand ready to provide any further information that could be helpful.
I thank the authors for their rebuttal and the additional results on the Spring dataset. I also would like to acknowledge the new graph giving an overview and comparison of the efficiency of various methods. But unfortunately, I find that the rebuttal only answers some of my questions.

My Q1 was about the relationship between image size and memory, while the authors provided helpful information, the question has not been answered.

My Q2 was a request to report all evaluation parameters for MPI-Sintel.

In weaknesses, I had pointed out that VideoFlow MOF outperforms StreamFlow for accuracy on Kitti, and that MemFlow has reported superior results. While I understand that different experimental setting lead to different results, the strong claims made in the paper about the relative accuracy of StreamFlow seem overstated.