Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose to improve optical flow estimation performance in occluded regions by increasing the receptive field size of the network. The idea itself is straightforward, but there are technical challenges in implementing it. In particular, optical flow datasets are small in size, thus simply optimizing a very deep network is not a viable option. Directly increasing the kernel size also does not work well due to especially high memory requirements during flow estimation and optimization difficulties associated with large kernel networks. To address these issues, the authors propose two architectural tricks - ones is using depth-wise-separable convolutions, and another is adding another 1x1 convolutional layer before the large kernel convolution to simplify optimization. Extensive experimental analysis demonstrates that the resulting approach outperforms the state-of-the-art by non-negligible margins with particularly large improvements in occluded regions. An extensive ablation analysis demonstrates the importance of the most of the architectural decisions for the final performance.
strengths_and_weaknesses: Strengths:

The proposed idea of increasing the performance of optical flow estimation in occluded regions by increasing the size of the receptive field is sound.

The proposed architectural tweaks to the GMA architecture to make training with large kernels viable are also sound and their effectiveness is experimentally demonstrated. 

The final approach outperforms the state-of-the-art in most scenarios, with improvements being especially significant in occluded regions. 

An extensive abalone analysis is provided.

The paper is readable. 


Weakness:

The novelty is mainly in the architectural tweaks to the GMA architecture to make training it with large kernels on small datasets viable. 

There are still quite a lot of grammatical mistakes in the text.

Although the ablation analysis is quite extensive, a few important points are not addressed. In particular, the switch from a GRU to a residual network is neither motivated nor ablated, and the choice of the hyper parameters is not explained (number of iterations during inference, lambda parameter in the loss).
questions: Please provide a motivation to for the switch from GRU to a residual network and ablate it.

Please report the number of iterations used during inference and explain the choice of lambda.
limitations: None
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a CNN-based method that deals with optical flow estimation, especially to tackle occluded areas. To that end, the proposed method studies the relationship between the receptive fields of the network and the effectiveness of recovering occluded motions. The proposed method designs the network using super kernel blocks, which are composed of small size depth-wise convolutions followed by a set of large-size convolutions. Comprehensive experiments demonstrate that the proposed method is quite effective and outperforms the state-of-the-art baseline methods. 
strengths_and_weaknesses: Strengths
1. The proposed method consistently outperforms the baseline methods on mainstream optical flow benchmarking datasets, including Sintel (synthetic), KITTI-15(real).  The proposed method is submitted to the Sintel benchmark platform and obtained the state-of-the-art ranking on that dataset. 

2. This paper is generally clearly written with a proper literature review. The introduction part has stated the motivation of the paper clearly and specifies the challenges of the existing methods as well as the key component to deal with the challenges.  The method section has delivered a clear architecture by using visual graphs and proper citation. The experiments part have gone through a systematic way to run benchmarks. 

Weaknesses
1. This paper's main problem lies in the theory part of the approach section. It does not specify the theory behind the super kernel to validate the reason for the effectiveness. Even though, we understand that the architecture outperforms other methods in the experiments. But it lacks the reasoning why the architecture is designed in that way in the first place. 

2. In the experiment section, the description says the models are trained on more challenging data (i.e. Sintel clean and final) but tested on the intrinsic components of the scene (i.e. Albedo), which usually cannot validate the effectiveness of the methods. But in Table 1, the results are conducted on the clean and final datasets.  

3. The authors are suggested to conduct one more ablation study to verify that the improved performance is not coming from more parameters compared with the baseline RAFT and GMA method by running a vanilla network with the same number of parameter networks. 

4. Based on the ablation study, it seems that SK modules ( motion encoder and updater ) affect the performance much more than the conical architecture compared with other architecture settings. But the paper claims that the key effective components should be the conical design of the network. Can the authors explain and validate more on this issue?




5. Many typos: 
i). All the equations (1) -(11) lack punctuation. 
ii). Line 173, 182, etc, the short headline should not come with full stop. 
iii). e

questions: 1. Usually KITTI-2015 dataset contains more large displacement / fast motion (than Sintel Dataset), which introduces considerably more occluded areas. But the performance of the network is not better than its predecessor network RAFT. Can the authors provide more discussion about it? 
limitations: The main limitation is that the paper does not describe clearly the reason why the Conical architecture is the key reason to improve the performance of the proposed network in dealing with occlusion areas. It is not straightforward/intuitive relationship between the receptive field and the occluded area. So it requires the paper to make a more sound analysis and insightful theoretical development.
review_title: null

final_score: rating: 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this work, the authors proposed a simple but effective way to handle occlusion in optical flow estimation. The main idea is to use a kernel with a large reception field. Unlike previous work that simply increased the depth of network to increase network, the authors proposed a new super-kernel block, which uses a super block that consists of a large and a small kernel. Experiments have demonstrated the proposed network can significantly boost the accuracy of optical flow, with minimal additional compute.
strengths_and_weaknesses: This is a very high quality work. The authors proposed a very simple but also very effective network to increase the receptive field of the flow estimation network. The authors achieved the best accuracy in the Sintel benchmark, and also did a very thorough experiment to demonstrate effectiveness of the proposed network on different setups and how different alternative designs may work. At last, although this work mainly focuses on motion estimation, the proposed architecture is very general, and I believe it could benefit many other image processing tasks and even beyond.

I don't find any particular weak point in this work.
questions: ### I only have two minor questions about this work.

Another way to increase the receptive field without introducing too many network parameters is dilated convolution. It would be great if the authors can discuss Pros and Cons of proposed approach compared to dilated convolution.

For efficiency evaluation, I think it is also important to report actual runtime, in addition to FLOPS/MACs, as due to cache locality and other memory issues, FLOPS/MACs do not always lead to a low execution time, particularly when a large kernel is used like this work. For completeness, it is better to report runtime on a certain device.

### I also have a few suggestions

The authors did not need to reply to this question in rebuttal, but would be great to answer them in the camera-ready version.

Funnel v.s. Conical. In theory, convolutional kernels are mostly interchangeable, so I would expect funnel v.s. Conical should achieve similar results. Experiments actually show that conical performs slightly better than funnel (Table 3). It would be great if authors could share some insights, even preliminarily, why funnel performs better, and why it performs better than parallel on the selected datasets (I understood datasets could make difference, but would still be interesting to the audience if the authors give more insights). 

The proposed super kernel is a very general design. It would be great if the authors could also discuss the potential usage of this structure in other image processing tasks in conclusions / future work.

limitations: No as far as I know.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work addresses the problem of optical flow estimation and is based on RAFT and GMA. The paper proposes to use a larger kernel in convolutional neural networks to solve the problem of occlusion occurring in optical flow. A super kernel block with depth-wise and channel-wise convolution is introduced to reduce the computational cost. The experimental result shows state-of-the-art performance.
strengths_and_weaknesses: Strengths:
1. The paper is clearly written with the baseline method and the contribution.
2. The proposed super kernel block is simple in structure and makes generalization to other networks possible.
3. The ablation study is quite comprehensive.
Weaknesses:
1. Using a larger kernel in optical flow has been proposed before, see line 40 and [7,11,24-26]. The extension to the proposed super kernel is rather incremental.
2. The proposed extension is only applied to the GMA architecture and it remains unclear whether the super kernel block will improve results for other architectures.
questions: 1. Line 127 is a little bit unclear about the computation cost. How is the cost obtained in more detail?
2. When is the computational cost of SKFlow smaller than the normal kernel convolutional method? Is the complexity related to the number of input channels and output channels and the size of the kernel? For the example in the paper (kernel size of 15), what is the ratio of cost_ours and cost_normal?
3. In line 276, it is said that the most effective auxiliary kernel size is 1*1. But in that case, isn’t depth-wise convolution just a multiplication? Where does the better performance come from?
4. Experiment: Why is GMA chosen as the baseline? Is there a specific reason? Have other baselines been explored using the super kernel block? If so, what results have been obtained?
5. Has the proposed approach been compared to a traditional large kernel approach? For example, using a kernel size of 15 to replace the super kernel block. Is this possible? How large is the computational cost in comparison to the super kernel block?

Additional questions of interest:

6. The occlusion analysis is only tested on the Albedo pass, how about testing on the Clean pass and the Final pass? How does it perform?

7. Have deeper layers been tried instead of a larger kernel to enlarge the receptive field? Does it perform badly or is it just a reasonable assumption that using a larger kernel will have a better result?
limitations: The paper did not mention any limitations or negative societal impacts. The latter might be difficult, but optical flow in general has its limitations regarding the estimation of the (projected) motion. Isn't it time to also consider learning to estimate 3D motion or the projected motion instead of optical flow?
review_title: null

Meta-review: metareview: The paper proposes an optical flow estimation network using very large convolution kernels. Three reviewers consider the paper above the bar, while one considers it below the bar. After consulting the paper, reviews, and rebuttal, the area chair decides to side with the positive reviewers due to the strong optical flow estimation results in the benchmark datasets. The AC believes the paper could provide the field some inspiration to look into large convolution kernels for other visual recognition and processing task.
confidence: Certain
Decision: Accept