Hybrid Cost Volume for Memory-Efficient Optical Flow

Authors: yang  zhao
gangwei  xu
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-30 16:22:35

Abstract: Current state-of-the-art flow methods are mostly based on dense all-pairs cost volumes. However, as image resolution increases, the computational and spatial complexity of constructing these cost volumes grows at a quartic rate, making these methods impractical for high-resolution images. In this paper, we propose a novel Hybrid Cost Volume for memory-efficient optical flow, named HCV. To construct HCV, we first propose a Top-k strategy to separate the 4D cost volume into two global 3D cost volumes. These volumes significantly reduce memory usage while retaining a substantial amount of matching information. We further introduce a local 4D cost volume with a local search space to supplement the local information for HCV. Based on HCV, we design a memory-efficient optical flow network, named HCVFlow. Compared to the recurrent flow methods based the all-pairs cost volumes, our HCVFlow significantly reduces memory consumption while ensuring high accuracy. We validate the effectiveness and efficiency of our method on the Sintel and KITTI datasets and real-world 4K (2160 × 3840) resolution images. Extensive experiments show that our HCVFlow has very low memory usage and outperforms other memory-efficient methods in terms of accuracy. The code is publicly available at https://github.com/gangweiX/HCVFlow.

Main paper: 
 introduction: 
 Optical flow, a fundamental aspect of computer vision, aims to estimate the two-dimensional motion of each pixel between two consecutive images. This task serves as a fundamental component providing dense correspondences as valuable clues for downstream applications, including object tracking [26, 39, 41] , high-quality video reconstruction [46, 51] , and autonomous driving [4, 14, 19] . With the emergence of deep learning, neural network-based methods [9, 15, 18, 31, 33-35, 37, 49] have become mainstream in optical flow algorithms, achieving superior results in accuracy. However, balancing the trade-off between memory consumption and high accuracy remains a challenging endeavor, which limits the application of optical flow algorithms in scenarios involving high-resolution images.Within the realm of deep learning methods for optical flow estimation, a key module known as the cost volume [9, 18, 34, 35, 37] , also referred to as the correlation volume, holds critical importance. This component captures the correlations between pixels across two images, effectively storing a measure of similarity or disparity between them. RAFT [37] represents a significant advancement in the field of optical flow research, constructing a global 4D cost volume by calculating correlations across all pairs of pixels. This cost volume, encompassing comprehensive matching information, has enabled RAFT [37] to achieve remarkable levels of accuracy. However, the approach comes with a drawback: the spatial complexity of building such a cost volume is 𝑂 (𝐻 × 𝑊 × 𝐻 × 𝑊 ). With increasing image resolution, the required memory for computation grows quadratically, limiting its application to high-resolutionFigure 2 : Comparisons with Flow1D [48] on high-resolution (1080 × 1920) images from DAVIS [30] dataset. We achieve better results than Flow1D [48] when consuming similar memory.images. To mitigate the memory overhead of constructing a full matching cost volume, some researchers have proposed memoryefficient methods for constructing cost volumes [21, 48, 56] . A representative example is Flow1D [48] , which constructs two threedimensional cost volumes along the horizontal and vertical directions, respectively. This approach reduces the overall complexity to 𝑂 (𝐻 × 𝑊 × (𝐻 + 𝑊 )). However, in the process of constructing the cost volume, Flow1D [48] utilizes global attention at each position to propagate and aggregate feature information orthogonal to the current row/column. This attention-based aggregation incorporates a substantial amount of noise and is strongly correlated with position, making it struggle to match large motions correctly.In this paper, we propose a novel Hybrid Cost Volume (HCV) for memory-efficient optical flow estimation. Leveraging this hybrid cost volume, we designed an end-to-end network, named HCVFlow, for optical flow estimation that achieves notable accuracy while requiring reduced memory resources.The Hybrid Cost Volume (HCV) is constructed in two primary stages: In the first stage, we build two 3D global cost volumes along both horizontal and vertical dimensions by calculating the correlation between the target and reference feature maps. Unlike Flow1D, which aggregates all information into a single value per row or column using attention techniques, our method employs a Top-k strategy. This strategy retains the k positions in each row or column with the highest relevance, ensuring that essential matching information is preserved. Additionally, we introduce a lightweight and efficient separable aggregation module. This module aggregates the 3D cost volumes along both dimensions, capturing more non-local information. This is particularly beneficial for addressing challenges such as occluded areas or large textureless/reflective surfaces. The aggregation module also provides a relatively accurate initial optical flow prediction, laying the groundwork for further optical flow regression. After aggregation, the result is two 3D global cost volumes that encompass a wider array of potential matching scenarios with minimal memory overhead, thereby enhancing the accuracy of subsequent optical flow predictions. In the second stage, a local 4D cost volume is constructed by calculating the correlation within a local 2D search space. This local 4D cost volume, with a comparatively small search domain, does not significantly increase memory consumption. Importantly, by preserving match information within a localized 2D domain, it complements the global 3D cost volume with critical local details that might be missing otherwise.By integrating these two global 3D cost volumes with the local 4D cost volume, we achieve the final Hybrid Cost Volume (HCV). This innovative structure effectively balances memory efficiency with the ability to capture detailed motion information, significantly improving both the precision and reliability of optical flow predictions across various challenging scenarios.The process of constructing two 3D cost volumes with the Top-k strategy incurs an overall complexity of 𝑂 (𝐻 × 𝑊 × (𝐷 + 𝐷) × 𝐾), where 𝐾 is substantially smaller than both 𝐻 (height) and𝑊 (width), especially for high-resolution images. We typically set 𝐾 to 8. D represents the maximum displacement in the horizontal/vertical direction. The complexity for building the local 4D cost volume is 𝑂 (𝐻 × 𝑊 × (2𝑅 + 1) 2 ), where 𝑅 is the search radius, and 𝑅 is much smaller than both 𝐻 and 𝑊 . As a result, the total complexity for constructing the Hybrid Cost Volume (HCV) is maintained at 𝑂 (𝐻 × 𝑊 × (𝐷 + 𝐷) × 𝐾). The 𝐷 is smaller than 𝐻 or 𝑊 for high-resolution images. In comparison to the 𝑂 (𝐻 × 𝑊 × 𝐻 × 𝑊 ) complexity associated with generating a cost volume in RAFT, our methodology significantly reduces memory requirements while capturing the essential matching information effectively.The experimental results demonstrate that our HCVFlow, constructed using our Hybrid Cost Volume (HCV), achieves remarkable accuracy and exceptionally low memory consumption. The experiments conducted on the KITTI [29] datasets showed that our method outperforms previous memory-efficient methods [21, 48] , such as Flow1D [48] , by more than 16%. The accuracy of our model are close to those of RAFT [37] , yet it only requires one-eighth of the memory used by RAFT. Furthermore, our benchmark tests on the Sintel [2] test dataset have surpassed RAFT, significantly exceeding other memory-efficient methods. Specifically, our method outperforms Flow1D by 26% on Sintel (Final) test dataset.Overall, our work makes the following key contributions: We obtain feature maps at 1/8 and 1/16 resolutions and construct the Hybrid Cost Volume (HCV) using these feature maps. Specifically, we compute initial cost volumes in both horizontal and vertical directions, followed by obtaining 3D cost volumes through a Top-k strategy. Subsequently, we aggregate these volumes using an aggregation module to obtain the final 3D global cost volume. Additionally, we construct a 4D local cost volume. Finally, we input the the hybrid cost volume and initial flow predictions generated by aggregation module into the ConvGRU module for iterative flow prediction.• We develop a memory-efficient technique for constructing cost volumes by implementing a Top-k strategy. This approach allows us to decompose the conventional 4D cost volume into two more manageable 3D cost volumes, significantly reducing memory requirements while preserving the most valuable matching information. Additionally, we designed a lightweight aggregation module that enables these 3D cost volumes to capture more non-local information, enhancing their capacity to account for complex motion scenarios. • Innovatively, we combine the 3D global and 4D local cost volumes to create the Hybrid Cost Volume (HCV). This novel structure not only minimizes memory consumption but also encodes a rich set of effective matching information capable of handling various motions. The integration of global and local cost volumes addresses the challenges of accurately predicting motion across a wide range of scenarios, making HCV a versatile and efficient solution. • Leveraging the Hybrid Cost Volume (HCV), we have constructed an end-to-end optical flow prediction network named HCVFlow. Experimental results demonstrate that HCVFlow surpasses representative memory-efficient method Flow1D by 16% in terms of accuracy on KITTI dataset, closely rivaling the performance of RAFT with only one-eighth of RAFT's memory consumption. On the Sintel test dataset, HCVFlow's benchmark results exceed Flow1D by more 26% and also surpass RAFT. 
related work 2.1 deep-flow method: 
 The research of optical flow has a long history, with traditional methods being explored for optical flow estimation decades ago. Among these, the Horn-Schunck [13] and Lucas-Kanade [24] methods stand out as seminal approaches. However, in recent years, the advent of deep learning has led to a surge of techniques based on this paradigm, which have significantly outperformed traditional methods in terms of accuracy. As a result, deep learning-based optical flow prediction methods [9, 11, 15, 18, 25, 28, 33, 36, 46] has become dominant in the field. FlowNetS [9] is pioneering work in the end-to-end prediction of optical flow using CNN technology, which follows a straightforward, end-to-end learning approach without any specialized layers or mechanisms specifically for optical flow beyond the standard convolutional layers. FlowNetC [9] introduces a correlation layer to better capture the relationship between two images by learning a similarity measure. FlowNet 2.0 [18] uses a stacked architecture that refines optical flow estimates through multiple training schedules to achieve high accuracy. PWC-Net [34] leverages pyramidal processing, warping, and cost volume layers which enables efficient handling of motions at different scales and has shown remarkable performance, especially in scenarios with rapid movement and occlusions.A recent notable work is RAFT [37] , which introduces an allpairs cost volume. This cost volume stores a wealth of matching information, allowing RAFT to achieve higher accuracy during flow regression with GRU block. RAFT achieves state-of-the-art accuracy in the optical flow, particularly outperforming other methods in challenging scenarios such as fast-moving objects and occluded areas. Following RAFT's success, several researchers have made improvements and innovations based on its fundamental structure. However, the construction cost of this all-pairs 4D cost volume, which is 𝑂 (𝐻 ×𝑊 × 𝐻 ×𝑊 ), results in high memory usage, making these methods challenging to apply to high-resolution images due to the significant GPU memory consumption.To facilitate the use of optical flow algorithms on lower-end consumer GPUs and with high-resolution images, recent research [21, 42, 48] has introduced a series of studies focused on developing memory-efficient optical flow networks. SCV [21] adopts a sparse cost volume to replace RAFT's all-pair cost volume, aiming to decrease memory usage. For each position, SCV utilizes a Top-k strategy, keeping only the k most relevant points for subsequent matching. However, in areas that lack distinctive features or are blurry, the inherent ambiguity can lead to a multitude of potential matches. In such cases, the Top-k approach may not encompass the correct match, potentially leading to erroneous motion predictions.Flow1D innovatively proposed replacing the all-pairs 4D cost volume with two separate 3D cost volumes [6, 7, 12, 32, [43] [44] [45] along the horizontal and vertical directions, significantly reducing the network's memory usage. However, in constructing the cost volume, Flow1D utilized the attention mechanism to aggregate information from each row/column into a singular value. This approach led to a loss of critical information in the resulting cost volume, making it susceptible to mismatches, particularly in long-distance, high-speed motions where Flow1D often predicts incorrectly. The limited local information also presents a challenge in achieving precise matching. Consequently, despite its advancements, the overall accuracy and generalization capabilities of Flow1D still fall short when compared to RAFT.In response to these challenges, particularly the lower accuracy and difficulties encountered in certain scenarios by memoryefficient methods [21, 48] , our research introduces a novel hybrid cost volume approach. This method combines the global 3D cost volumes with the local 4D cost volume to adeptly manage a variety of motions. The empirical results from testing our network on datasets such as KITTI [29] and Sintel [2] have shown notable enhancements in both generalization and accuracy over prior memory-efficient solutions. 
method 3.1 hybrid cost volume construction: 
 The construction of the Hybrid Cost Volume (HCV) consists of the following steps: initial 3D global cost volume construction via Top-k strategy; aggregation of the initial 3D global cost volume; and construction of the 4D local cost volume. 3D global cost volume construction. We provide a detailed description of the 3D global cost volume construction along the horizontal direction, while the construction for the vertical is similar.For the input feature mapsF ′ 1 ∈ R 𝐶 ×𝐻 ×𝑊 and F ′ 2 ∈ R 𝐶 ×𝐻 ×𝑊, we compute the correlation between feature points using the dot product operation, 𝐻 and 𝑊 represent the height and width of the 16× downsampled image respectively. Specifically, the correlation 𝐶 (𝑢, 𝑣, 𝑖, 𝑗) between the pixel at (𝑢, 𝑣) in F ′ 1 and the pixel at (𝑖, 𝑗) in F ′ 2 is formulated as:EQUATIONThe • symbol refers to the dot product operation, and 1√ 𝐶 acts as a normalization factor.For any given horizontal displacement 𝑑 (𝑑 ∈ {-𝐷, -(𝐷-1), -(𝐷-2), . . . , 0, . . . , 𝐷 -1}), we first utilize the following formula to calculate the horizontal correlation C ℎ𝑜𝑟𝑖𝑧𝑜𝑛𝑡𝑎𝑙 between the point (ℎ, 𝑤) in F ′ 1 and all corresponding points in F ′ 2 at a horizontal displacement of 𝑑:C ℎ𝑜𝑟𝑖𝑧𝑜𝑛𝑡𝑎𝑙 (ℎ, 𝑤, 𝑑) = {C 1 , C 2 , . . . , C 𝐻 }, C 𝑖 = C(ℎ, 𝑤, 𝑑, 𝑖).(EQUATIONThen we obtain a dense horizontal cost volume C ℎ𝑜𝑟𝑖𝑧𝑜𝑛𝑡𝑎𝑙 . To reduce memory consumption while preserving the majority of valuable matching information, we propose to use a Top-k strategy on C ℎ𝑜𝑟𝑖𝑧𝑜𝑛𝑡𝑎𝑙 , retaining only the top 𝑘 points with the highest correlation at a given horizontal displacement 𝑑. We haveEQUATIONwhere the notation TopK(•) denotes the operation of selecting the largest 𝐾 values from a given list. The spatial complexity of our sparse horizontal cost volume, C 𝑡𝑜𝑝𝑘 ℎ𝑜𝑟𝑖𝑧𝑜𝑛𝑡𝑎𝑙 , is 𝑂 (𝐻 × 𝑊 × 𝐷 × 𝐾). 𝐷 represents the maximum horizontal displacement on the left or right. 𝐾, with a default value of 8, is significantly smaller than both 𝐻 and 𝑊 in high-resolution images. Thus, we ultimately obtain a 3D global cost volume C 𝑡𝑜𝑝𝑘 ℎ𝑜𝑟𝑖𝑧𝑜𝑛𝑡𝑎𝑙 with a spatial complexity of 𝑂 (𝐻 × 𝑊 × 𝐷 × 𝐾).By applying this construction method to the vertical direction, we can easily obtain the vertical 3D global cost volume, C 𝑡𝑜𝑝𝑘 𝑣𝑒𝑟𝑡𝑖𝑐𝑎𝑙 . Aggregation of the 3D global cost volume. By employing the Top-k strategy, we have separately obtained the 3D cost volumes for both horizontal and vertical directions. The structure and properties of these unidirectional cost volumes resemble those found in stereo matching [5, 10, 12, 40, [43] [44] [45] 47] , inspiring us to adopt aggregation methods commonly used in stereo matching to optimize our initial 3D cost volumes.Therefore, we design a novel, lightweight aggregation module R to capture more non-local information, enhancing accuracy in handling complex scenarios such as occlusions and textureless regions. The cost aggregation is expressed as,EQUATIONThe C 𝐻 and C 𝑉 represent the aggregated horizontal 3D cost volume and vertical 3D cost volume respectively. To implement cost aggregation, we first employ a sequence of 3D convolutions with batch normalization and ReLU activations to downsample the feature maps while further extracting features. Then, we utilize a 3D transposed convolution layer, which enlarges the spatial dimensions of the feature maps, enriching them with spatial information. By developing these two 3D global cost volumes via Top-k strategy, not only can we capture the vast majority of valuable global match information, but we also substantially decrease the memory usage compared to the 4D global cost volume method employed by RAFT. Furthermore, we leverage the C 𝐻 and C 𝑉 for an initial optical flow estimation f 𝑖𝑛𝑖𝑡 :f 𝑖𝑛𝑖𝑡 _ℎ = 𝐷 -1 ∑︁ 𝑑=-𝐷 𝑑 × 𝑆𝑜 𝑓 𝑡𝑚𝑎𝑥 (C 𝐻 (𝑑)), f 𝑖𝑛𝑖𝑡 _𝑣 = 𝐷 -1 ∑︁ 𝑑=-𝐷 𝑑 × 𝑆𝑜 𝑓 𝑡𝑚𝑎𝑥 (C 𝑉 (𝑑)), f 𝑖𝑛𝑖𝑡 = Concat{f 𝑖𝑛𝑖𝑡 _ℎ , f 𝑖𝑛𝑖𝑡 _𝑣 }.(5) These initial flow estimation f 𝑖𝑛𝑖𝑡 provides a relatively accurate starting point for the subsequent flow regression process. 4D local cost volume construction. Our 3D global cost volume introduced before is constructed from 1D horizontal and vertical directions. This method, while beneficial for computational efficiency and memory conservation, may result in the loss of certain matching details. Consequently, this approximation can introduce minor inaccuracies in localized regions, potentially affecting the overall precision of the optical flow predictions. To address this limitation, we propose the construction of a local 4D cost volume. Unlike the 3D global cost volume which is constructed at 1/16 resolution, the 4D local cost volume is constructed at 1/8 resolution.Initially, the 1/8 resolution source feature map F 1 (F 1 ∈ R 𝐶 ×2𝐻 ×2𝑊 ) and target feature map F 2 (F 2 ∈ R 𝐶 ×2𝐻 ×2𝑊 ) are aligned and compared within a small local range (𝑙 𝑟 ) to compute a correlation score. This process begins by padding the second feature map to ensure that comparisons can be made across all valid positions, followed by Image RAFT GMA Ours Figure 4 : Qualitative comparisons with accuracy-oriented methods on the KITTI test set [29] . Our novel aggregation module aggregates contextual information to reduce mismatches, thus our method outperforms RAFT and GMA in real-world complex texture-less areas.Figure 5 : Qualitative comparisons with memory-efficiency method Flow1D on the KITTI test set [29] . Flow1D fails to accurately predict motion near object edges, while our method can precisely estimate local details.an unfolding operation that prepares the feature map for efficient local comparisons. The unfolding radius is 𝑙 𝑟 . The unfolded target feature map is denoted asEQUATION)This 4D local cost volume offers a detailed and rich representation of similarity scores across local regions between two feature maps. By integrating this 4D local cost volume, HCV is able to capture more local matching information and enable more precise matching of similar areas between images, thereby reducing the likelihood of mismatches.Following RAFT, we first utilize a feature extraction network to derive feature maps F 1 and F 2 from the original reference and target images, achieving an 8× downsampling. To further reduce the memory footprint for the 4D global cost volume construction, an additional downsampling stage by a factor of 2× is applied to obtain F ′ 1 and F ′ 2 . Additionally, we incorporate a context network to extract contextual information, which is instrumental for the flow regression. Subsequently, as described in Sec. 3.1, we construct two separate 3D global cost volumes and one 4D local cost volume. By concatenating these volumes, we obtain the core component of HCVFlow, the Hybrid Cost Volume (HCV).In the flow regression stage, we index correlation features from 3D global cost volumes, C 𝐻 and C 𝑉 , and 4D local cost volume C 𝐿 . The indexed hybrid correlation features, together with flow features and context features, are all inputted into the flow regression module. Here, we employ a Convolutional Gated Recurrent Unit (ConvGRU [37] ) to iteratively update the flow predictions, which leads to the output of the refined optical flow results.We first compute the L1 loss on the initial optical flow estimation obtained from the aggregation module, defining it as follows:EQUATIONFollowing RAFT [37] , we calculate the L1 loss between all predicted optical flow sequences and the ground truth. Similar to RAFT, we exponentially increase the weights, with 𝛾 set to 0.8 in our experiments. The loss for the predicted flow is calculated as follows:EQUATIONFinally, we add these two parts of the loss to obtain the total loss function:EQUATION4 Experiments 
experimental setup: 
 Datasets and evaluation setup. We conduct experiments on the KITTI [29] , Sintel [2] , and high-resolution DAVIS [3, 30] datasets to evaluate the effectiveness of our method. We first train our model on the FlyingChairs [9] and FlyingThings3D [27] datasets. Upon completing the training, we conduct extensive experiments on both the KITTI and Sintel datasets to verify the performance and generalization ability of our method. Subsequently, we fine-tune our trained model on a mixed dataset comprising HD1K [22] , KITTI, Sintel, and FlyingThings3D and submit it to the KITTI and Sintel websites for benchmark testing. We employ the End-Point Error (EPE) metric to evaluate the model's prediction accuracy on the Sintel dataset and use both EPE and F1-all metrics to evaluate the accuracy on the KITTI dataset. F1-all denotes percentage of outliers for all pixels. Finally, we validate the performance of our method on high-resolution images (1080P and 4K resolutions) using the DAVIS [3, 30] dataset.Implementation details. We implement our HCVFlow using the PyTorch framework, with Adam [23] serving as the optimizer. Our feature network implementation is followed by RAFT [37] , but we have added an additional downsampling layer, resulting in a feature map that is downsampled by a factor of 16. Similar to other optical flow methods [32, 37, 48] , we trained our model for 100K iterations on the FlyingChairs dataset with a batch size of 12. Then, we trained our model for another 100K iterations on the FlyingTh-ings3D dataset with a batch size of 6. We finally fine-tuned our model on a mixed dataset comprising FlyingThings3D, Sintel, KITTI, and HD1K. For the Sintel evaluation, the fine-tuning was carried out over 100K iterations, and for the KITTI evaluation, it was conducted over 50K iterations. The batch size was set to 6 for fine-tuning.During training, we employed 12 GRU-based iterations. For the evaluation phase, we used 32 GRU-based iterations for Sintel and 24 GRU-based iterations for KITTI, respectively. When building the 3D global cost volume, we utilize a Top-k approach with the k parameter set to 8.We carry out ablation studies to confirm the effectiveness and efficiency of HCV's key components. For these studies, models are trained on the FlyingChairs and FlyingThings3D datasets and subsequently evaluated on the Sintel and KITTI training sets. Across all experiments, memory consumption and inference time are measured using 12 GRU-based iterations on our RTX 3090. We initially verify the effectiveness of the two primary components of HCV: the 3D global cost volume and the 4D local cost volume. As shown in Table 1 , the network constructed solely with the global cost volume (GCV) performs well in handling large motions(s 40+ ), yet shows weak performance for small motions (s 0-40 ). Conversely, the network utilizing only the local cost volume (LCV) demonstrates good performance on short-distance movements but struggles with long-distance movements. Our proposed Hybrid Cost Volume (HCV), which concates both GCV and LCV, manages to integrate the advantages of both cost volumes, achieving a synergistic effect where the whole is greater than the sum of its parts. Whether dealing with large motions or small motions, our HCV consistently delivers optimal performance. Subsequently, we conduct experiments to validate the effectiveness of the Top-k strategy used in constructing the 3D global cost volume. In the construction process, we experiment with three different strategies for retaining the correlation values: the maximum correlation value, the average correlation value, and the Top-k correlation values (with 𝑘 = 8 in our experiments), while keeping all other parameters constant. As shown in Table 2 , models constructed using the Top-k strategy outperforms those built with either the maximum or average values across various accuracy metrics on both the Sintel and KITTI datasets. Experimental results demonstrate that our Top-k strategy, which selectively preserves matches with higher correlation, yields more precise estimations than straightforward aggregation approaches like averaging or attention mechanisms, the latter of which tend to incorporate noise. Comparison with existing representative cost volumes. The core of our approach lies in the construction of a hybrid cost volume. So we conduct extensive experimental comparisons with other existing representative cost volume construction methods. We test the accuracy and memory consumption of RAFT [37] , FlowNet2 [18] , PWC-Net [34] , Flow1D [48] , and our HCVFlow on the KITTI and Sintel datasets on RTX 3090. Our comparison involves models that employ various methods for constructing cost volumes. Specifically, RAFT constructs a 4D global cost volume, FlowNet2 generates a single-scale cost volume, PWC-Net develops a coarse-to-fine cost volume pyramid, and Flow1D uses an attention mechanism to build two 3D global cost volumes. As illustrated in Table 3 , our HCVFlow achieves suboptimal End-Point Error (EPE) on both the Sintel and KITTI datasets, slightly behind RAFT. However, it surpasses RAFT on the KITTI dataset in terms of the F1-all metric, achieving the best performance. Our method outperforms Flow1D by 26.8% and PWC-Net by 50.1% on the KITTI dataset in terms of the F1-all metric. Moreover, our method consumes significantly less memory compared to approaches like RAFT. For images at 1080P resolution, our memory usage is only one-fifth of RAFT's. Compared to Flow1D, which is also known for its memory efficiency, our method consumes only slightly more memory but significantly surpasses Flow1D in terms of accuracy. Comparison with memory-efficient methods. We further compare our method with other memory-efficient optical flow methods. As shown in Table 4 , our HCVFlow exhibits superior accuracy compared to Flow1D [48] and SCV [21] . While we are slightly behind DIP [56] on some accuracy metrics, our model significantly surpasses DIP in terms of memory usage and inference time. When processing 1080P images, our memory consumption is half that of DIP, and our inference time is only half as long.Comparisons with accuracy-oriented methods. We also compare HCVFlow with several accuracy-oriented methods. As shown in Table 5 , our accuracy on the KITTI and Sintel test datasets is close to these methods, with our EPE on the Sintel final dataset even surpassing some approaches like RAFT [49] and GMFlow [49] . Moreover, our method exhibits a clear advantage in terms of memory consumption and inference speed. During the inference process for 1080P images, our method's memory consumption is only one-seventh of SKFlow [36] 's and one-eighth of SepaFlow [54] 's.FlowFormer [15] consumes seven times more memory than our method for images of 448*1024 resolution and even fails to process images of 1080*1920 resolution due to OOM (given our RTX 3090's maximum memory capacity is 24G).In Table 6 , we showcase the benchmark results of our method on the KITTI and Sintel test datasets, along with comparisons to other methods. Our accuracy surpasses most methods, including PWC-Net+ [35] , SCV [21] , Flow1D [48] , and MaskFlowNet [55] , and is only slightly lower than methods designed with a focus on accuracy, such as SKFlow [36] and FlowFormer [15] . Compared to Flow1D [48] , a notable memory-efficient method, our performance on the Sintel (Final) test set improved by 26%, and we led by 12% on the KITTI test set. Figure 1 and Figure 5 display some predictive results of our method and Flow1D on the KITTI and Sintel datasets. It is evident that our approach preserves more fine structure and performs better in large textureless regions. Figure 4 demonstrates that our method performs better in handling textureless regions even when compared to some accuracy-focused approaches. We also conducted experiments on high-resolution images from the DAVIS dataset at both 1080P and 4K resolutions. As shown in Figure 2 and Figure 6 , our HCVFlow exhibits superior performance in handling fine details, achieving accuracy noticeably better than Flow1D while consuming a similar amount of memory. Furthermore, we only need 6GB of memory to process 4K images, whereas RAFT fails to handle them on 48G A6000 GPU due to OOM. 
conclusion: 
 RAFT and its successors achieve high accuracy in optical flow estimation using 4D global cost volumes. However, their memory consumption increases quadratically with image resolution, limiting their applicability to high-resolution images. In this paper, we propose a novel approach for constructing a hybrid cost volume (HCV) that achieves significantly lower memory consumption while maintaining high prediction accuracy. By decomposing the 4D cost volume into horizontal and vertical 3D cost volumes using the Top-k strategy and processing them with an aggregation module, we effectively reduce memory overhead while retaining the majority of matching information. Additionally, we construct a local 4D cost volume to supplement local information. By combining these cost volumes, our HCV not only drastically reduces memory usage but also achieves high prediction accuracy across various motion scenarios. We hope our research will advance the study and application of optical flow algorithms in high-resolution images and edge devices. 
: 
 Figure 1 : Qualitative comparisons on the Sintel test set [2] . Compared to the notable memory-efficient method Flow1D [48] , our approach achieves more accurate flow estimation in low-texture regions. 
 
 Appendix:  


Decision: This paper got one weak Accept, one Borderline Accept and two weak Reject. The studied problem is quite important for various multimedia applications, and it has been studied in previous works. The memory-efficient optical is achieved by building 3D global cost volumes, aggregating information via a Top-k strategy and a separable aggregation module, and constructing local 4D cost volume. The proposed idea is novel and the experimental results demonstrate its effectiveness. I recommend to accept to ACM MM. However, the authors should revise the paper according to the review comments.

Reviews: review: paper_summary: paper_summary: This paper presents a Hybrid Cost Volume (HCV) method to reduce memory consumption of optical flow estimation. Based on the observation that the global 4D cost volume used by RAFT has a spatial complexity growing quadratically with increasing image resolution, the proposed HCV mitigates the memory overhead by Top-k strategy. However, additional clarification along with experimental comparisons may be needed for better evaluations of the proposed Top-k strategy.
strength_weakness: strength_weakness: 1. This paper focuses on optical flow estimation, which is a critical component of motion analysis for various multimedia applications.
2. The proposed HCV method is devoted to the memory efficiency of optical flow algorithms by improving the cost volume. The improvement is achieved by building 3D global cost volumes, aggregating information via a Top-k strategy and a separable aggregation module, and constructing local 4D cost volume.
limitations: limitations: 1. Since Table 3 seems to be a replica of Table 4 and not be referred by the main content, the tables require an update.
2. Since the proposed HCVFlow architecture is trained through Adam that involves gradient descent methodologies, Section 3.1 only mentions the forward pass of TopK operation, missing the backward propagation regarding the TopK operation. Could the selection of the largest K values bring gradient loss? 
3. Since the Top-k strategy is a key contribution to the improvement of memory-efficiency, it may lead to a question of how to choose k. Though Table 2 provides results of k=8, additional results of different k-values may better present how the choice of k affect the trade-off between memory-efficiency and noise reduction.
4. HCV approach is described to have competitive accuracy compared with RAFT but the Figures only demonstrate comparisons with Flow1D, thus qualitative comparisons regarding HCV approach against RAFT and other SOTA methods may be added.
paper_summary: paper_summary: This work proposes a memory-efficient optical flow network, named HCVFlow. HCVflow adopts a novel Hybrid Cost Volume for memory-efficient optical flow, named HCV. HCV employs a Top-k strategy to construct 3D cost volumes, aggregates the initial 3D global cost volume, and constructs the 4D local cost volume. Experiments show that HCVFlow significantly reduces memory consumption while ensuring high accuracy.
strength_weakness: strength_weakness: 1. The writing is clear.
2. The Hybird Cost Volume combines the strength of global cost volume and local cost volume, which is validated to be effective.
limitations: limitations: 1. The main contribution of this work is the combination of global cost volume and local cost volume, which is limited. Moreover, in ablation study, the GCV+LCV retains the efficiency loss in trade-off accuracy improvement, which keeps a balance between efficiency and effectiveness. However, it is hard to conclude that it is the combination of these two cost volumes that introduces the efficiency improvement. 

2. The improvement is limited as compared to accuracy-oriented methods. Compared to RAFT, the accuracy performance on KITTI-test and Sintel-clean and efficiency performance (memory usage/ inference time) are inferior.

3. The contents in Table 3 and Table 4 are the same.
paper_summary: paper_summary: - This paper introduces an efficient Hybrid Cost Volume for estimating the optical flow.
- Different from previous flow methods introducing all-pairs cost volumes for high performance, the proposed HCVFlow proposes local 4D cost volume with a local search space, which reduces computational costs while retaining the performance.
strength_weakness: strength_weakness: - Employing local 4D cost volume scheme, reducing GPU memory usage by separating horizontal/vertical 3d global cost volume.
- Achieving comparable performance on multiple benchmark datasets with reduced computational costs.
limitations: limitations: - The crucial part of this paper is hard to follow. For instance, the process for generating cost volume, including calculating correlation and top-k strategy should be visualized with a description for better understanding. Like Figures in the Flow1D paper.
- For selecting horizontal/vertical Top-K pixels from the cost volumes, the spatial information of feature or cost volumes can be broken or no longer valid due to selection. Additional concerns about this problem need to be stated or properly described.
- Ablation study needs to be conducted more comprehensively. There are many hyper-parameters that can potentially impact performance in this work. Such as displacement radius D, number of K in Top-K, 4D local volume construction radius lr. 
- In Tab. 1, each GCV and LCV consume 0.32G for 448 $\times$ 1024, why does the GCV+LCV consume only 0.38, which is much lower than the sum of both settings? For GCV and LCV, the network structure needs to be described more comprehensively. 
- The visual results in Fig. 2,4,5, authors should display more candidate or competitive works to show superiority. 
- Overall, the quality of the writing and the contribution are insufficient. There are lack of notations and details in the overview (Fig. 3), which hinders understanding. Further, compared to previous flow estimation methods, authors achieve marginal improvement on only memory consumption. For instance, in Tab. 3, the RAFT shows the best performance on most metrics, including the parameters with slightly larger memory usage.
paper_summary: paper_summary: This paper proposed a memory-efficient optical flow method named HCV. The authors proposed a Top-K strategy that seperates the 4D cost volume into 3D, and it reduced memory consumption compared with the recurrent flow methods.
strength_weakness: strength_weakness: This paper is well-written and easy to follow, the main experimental result shows promising improvement.
limitations: limitations: It has not compared comprehensively with the recent advanced methods, e.g., FlowFormer++ (Shi et al., CVPR 23), 	DistractFlow (Jeong et al., CVPR 23)