SKFlow: Learning Optical Flow with Super Kernels
Authors: SHANGKUN SUN
Yuanqi Chen
Yu Zhu
Guodong Guo
Ge Li
Venue: NeurIPS 2022 Accept (2022)
Keywords: Optical flow
Computer vision

Abstract: Optical flow estimation is a classical yet challenging task in computer vision. One of the essential factors in accurately predicting optical flow is to alleviate occlusions between frames. However, it is still a thorny problem for current top-performing optical flow estimation methods due to insufficient local evidence to model occluded areas. In this paper, we propose the Super Kernel Flow Network (SKFlow), a CNN architecture to ameliorate the impacts of occlusions on optical flow estimation. SKFlow benefits from the super kernels which bring enlarged receptive fields to complement the absent matching information and recover the occluded motions. We present efficient super kernel designs by utilizing conical connections and hybrid depth-wise convolutions. Extensive experiments demonstrate the effectiveness of SKFlow on multiple benchmarks, especially in the occluded areas. Without pre-trained backbones on ImageNet and with a modest increase in computation, SKFlow achieves compelling performance and ranks $\textbf{1st}$ among currently published methods on the Sintel benchmark. On the challenging Sintel clean and final passes (test), SKFlow surpasses the best-published result in the unmatched areas ($7.96$ and $12.50$) by $9.09\%$ and $7.92\%$. The code is available at https://github.com/littlespray/SKFlow.

SKFlow: Learning Optical Flow with Super Kernels
Shangkun Sun
Yuanqi Chen
Yu Zhu Guodong Guo 2.3
Ge Li
SScoool of Electronic and Computer Engineering, Peking University Institute of Deep Learning, Baidu Research "West Virginia University
Abstract
Optical flow estimation is a classical yet challenging task in computer vision. One of the essential factors in accurately predicting optical flow is to alleviate occlusions between frames. However, is still a thorny problem for current top-performing optical flow estimation methods due insufficient local evidence model oc- cluded areas. In this paper, we propose the Super Kernel Flow Network (SKFlow), CNN architecture to ameliorate the impacts of occlusions on optical flow esti- mation. SKFlow benefits from the super kernels which bring enlarged receptive fields complement the absent matching information and recover the occluded motions. We present efficient super kernel designs by utilizing conical connec- tions and hybrid depth-wing wise convolutions. Extensive experiments demonstrate the effectiveness of SKFlow on multiple benchmarks, especially in the occluded areas. Without pre-trained backbones on ImageNet and with a modest increase in computation, SKFlow achieves compelling performance and ranks 1st among currently published methods on the Sintel benchmark. On the challenging Sintel clean and final passes (test), SKFlow surpasses the best-published result in the unmatched areas (7.96 and 12.50) by 9.09% and 7.92%. The code is available at https///gitbb.coommittsss///////
Introduction
Optical flow is the task of modeling per-pixel motion across a pair of frames with various down- stream applications, e.g., pedestrian re-identification, video segmentation, scene reconstruction, etc. Nowadays, optical flow estimation still faces tough challenges such as occlusions and motion blurs, etc. Among those challenges, occlusion is considered one of the most difficult problems which are still under-explored. Notably, the term occlusion in optical flow estimation can be extended to broader definition [16]: regions where pixels appear in the current frame while disappearing in the next frame.. Occlusions pose apparent difficulties in predicting optical flow since it directly violates the brightness constancy constraint [10], where the intensities of pixels are regarded as the same between consecutive frames. Therefore, the correspondence matching of occluded areas between frames could be extremely hard.
One of the potential solutions is to utilize the neighboring pixels to recover motions of the occluded regions, such as learning the neighboring relationship by CNNs 31
32 36), interpolating the However, both convolution and interpolation are restricted to the local information within the small operation window, which only focuses on learning the local evidence. As occlusions get worse, the local evidence would be insufficient recover the hidden motion and thus severely degrade the performance. Recent works [6]] 42] propose to model long-range dependencies between local descriptors via non-local methods to make up for the missing local evidence. These methods alleviate the issue to some extent but still tend to fail since the representative capabilities of local descriptors have been largely weakened when facing severe occlusions.
hidden motions via smoothness terms in Markov Random Fields 4
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

FPP: 2504
FPP:: 5..80
203
B2
6))
(b)
(c)
(6)
Figure I: Occlusions between frames and flow error map of different methods. From (a) to (b), the most of the blade moves out-of-frame (e.g. pixel B1 moves pixel B2). But the motion of the occluded B1 could still be recovered via the non-occluded motion of A₁ over long distances due to the large receptive fields. (c) and (d) denote the error between the predicted flow and ground truth from method with normal convolutions (e.g. GMA) and our SKFlow, respectively. The darker the color, the greater the error. SKFlow achieves lower end end-point error (EPE) and performs better in the occluded blade area.
Motivated by [21. 7 that larger kernels present an effective way compared to deeper layers for larger receptive fields, using large kernels in an optical flow estimation network is arguably a possible solution to handle the occlusion problem, as illustrated in Figure II However, directly applying large kernels is not practical due to: (1) The quadratic growth in computation. As the kernel size grows
from 3 x 3 to 15 x 15, the model would expand 25 times its original size. (2) The optimization issue. As shown in 7 [24 I1, even with careful design and training on a huge dataset, large kernel networks take effort to optimize and are even prone to performance drop. It is more challenging on optical flow networks, where the training data is often much smaller. For example, Sintel dataset has only 1, 041 training samples.
In this work, we propose Super Kernel Flow Network (SKFlow), where we introduce new architec- ture design that efficiently utilizes the conical connections and hybrid depth-wise convolutions, and accordingly develop an effective optical flow network to handle the occlusions. A detailed description of the approach is described in Section
B SKFlow is better at resolving the ambiguity caused by severe occlusions and obtains compelling performance on standard benchmarks (see Section 4 Besides, SKFlow attains good trade-off between accuracy and computation cost. On the challenging Sintel final pass test set, SKFlow ranks 1st among all published methods the time of submission, with the increase in MACs less than 8.42% compared with GMA [6]
Our contributions are summarized follows: (1) We introduce the super kernel schemes to the optical flow task for the first time. (2) We explore three new architecture designs for super kernel designs in the optical flow network and proposed a new network which we named SKFlow. (3) Our proposed SKFlow achieves state-of-thee performance on standard benchmarks and obtains strong cross-dataset generalization.
2 Related work
Optical flow estimation. 'rraditionall,, optical flow is formulated as an energy minimization prob- lem. Based brightness constancy and spatial smoothness, Horn and Schunck (10) pioneered the variational approach to computing optical flow. On that basis, there emerged subsequent improve- ments for visual similarity designs and regularization terms [10 30]. the deep lcarning era, CNNs have emerged as a powerful technique for optical flow estimation since FlowNet [8] Then the coarse-to- tone strategy widely adopted B1 32 12
methods are influential but tend to miss small motions due to the inaccurate flow guidance at the coarse level. Therefore, RAFT (36) presents an iterative refinement method with the all all-pairs field transform. It maintains a single fixed flow field at high resolutions and achieves significant improvements, inspiring lots of works such as Flow
4 Coarse-to-fi
FM-RAFT 17 Separable flow [42] AGFlow 20 and
GMA [16, etc.
Kernel sizes in convolution layers. Recently, there have emerged some explorations of applying large kernels to CNNs, which have not been widely used since early designs 34
35
33 27 The
ERF theory (2 indicates that larger kernels are more effective to obtain larger receptive fields than deeper layers. LRNet [TT proposes the local relation layer where 7 convolutions are widely used. However, there is performance drop when kernel size is increased to 9 x 9. GCN 244

---

LXL
Sx
LxL
LXL
Sx S
SxS
L L
PW
PW
PW
(a) Direct
(b) Paralell
(c) Funnel
(c) Conical
Figure 2: Architecture designs for super kernel blocks. and S denote the kernel size of large and the auxiliary depth-wise convolutions respectively. PW is the point-wise convolution. The underlined design used our SKFlow model.
further enlarges the kernel size in the segmentation task by dense-connected symmetric separable convolutions. It is observed that the performance tends to decrease when transferred to other tasks. CKCNN 26) and FlexConv (25] adopt larger kernel sizes, while the model size and computation cost grow correspondingly. RepLKNet [2] develops the re- pe-aaammtteriztiion technique to help optimize large kernels and scales the kernel size 31 31, achieving state-of-the-mee performance on classification and downstream tasks such as detection and segmentation. Despite its competitive results on these high-level vision tasks, our experiments show that it could not be directly applied for dense prediction tasks such as optical flow estimation (see Section (.4)). Therefore, we explore the large kernel designs in the optical flow network, aiming at alleviating the occlusions and improving the estimation performance.
3 Approach
We propose to utilize the large receptive field to handle the local ambiguity caused by occlusions. Since the motion of the occluded pixels is hard to estimate with local ambiguity, our SKFlow method introduces super kernels to help complement local evidence, and an efficient architecture design is proposed to reduce the computation burden. Section 3.1 The overall network architecture is then presented in Section 3.2 Finally, the learning process of SKFlow is demonstrated in Section 3.3
describe the details of super kernel block in
3.1 Super kernel block designs
We enlarge the kernel size instead of deepening layers attain a large receptive field given the following considerations. (1) Deep layers lead to the optimization issue. Although ResNet [9] has resolved the optimization problems for deeper networks 7 9
to some extent, recent works [
still suggest that it might behave like shallow networks and obtain the limited receptive fields with deeper layers. (2) The Effective Receptive Field (ERF) grows sub-linearly with the number of layers and linearly with the kernel size, which demonstrates that the large kernel size is more effective.
Super kernel block components. In order reduce the massive computation cost caused by large kernel sizes, we propose the super kernel block that contains three components: (1) Hybrid depth-wise convolution kernels. Inspired by separable convolutions [5], we first split convolution into couple of depth-wie -wie convolutions. Namely, a large depth-wiss wise kernel with size L x L and an auxiliary small depth- hepth-wise kernel with size S S. With the input of shape N C. X H W, the computation cost for this hybrid depth-wise convolutions is N x Cin x H X W x (I² + S2) ) x 1. The auxiliary kernel is designed to help capture small-scale patterns the frame. (2) Residual connections. Residuals (9] are used (a) to combine the large kernel and the auxiliary kernel; (b) connect depth-wise and point-wise convolutions. (3) Point-wise convolution. An point-wise convolution applied after hybrid depth-wise convolutions to help the information flow across channels. The point-wise convolution

---

does not change the input dimension and the computation cost is: N x Cin H X W x Cin with the input shape N Cin H W.
Super kernel block architecture. We propose three architecture designs for our super kernel blocks. As shown in Figure 2 our designs (b, and d columns) are derived from the Direct design in column a. Specifically, (1) Parallel (Figure (b)). is block of layers that adopting parallel small kernel in the large depth-wise convolution layer. A point-wise convolution is followed by parallel VGG-style convolutions combined with skip connections.
(2) Funnel (Figure 2 (c),, is ResNet-style block of layers where the kernel sizes are decreased from large kernel I. to auxiliary kernel S. (3) Conical (Figure (c)), is similar the Funnel design but applies the hybrid convolutions in the opposite order. Experimental results in Section 4.44 that all three large kernel designs outperform normal convolution layers with normal small kernels and the computation increase is very limited. Among all these architectures, the conical block obtains the best performance and is adopted as our final design, which can formulated as:
h (xx Conu due
sx
(1)
d (h + ConeExI (h))
du
(2)
p Coneixi (d)
d (p))
(4)
where x and o denote the input and output feature map, respectively. Conv Convixx r. refer to the depth-wise convolution with large and small kemels, respectively. Comeixi is the point-wise convolution and is the activation function.
To match the dimension and increase more non-linear transforms, two pairs of x convolutions performed in each super kernel block. Each pair of convolution layers will first transform the input feature map from dimension Cin to aCin. and then from aCin to C². Namely, the computation cost is N H. x W x Cin aCin + N H W aCin C², where C" Cin for the first pair of convolution layers and C" = Cout for the second pair. Therefore, the computation is N H x W aC X (Cin + Cout + Cin + C.n). In summary, the total computation of a block is N x H x W x Cin x [L2 + S2 + Cin + a33 + Cout)]. We can then compute the computation cost ratio of our super kernel block and the normal convolution as:
Costours
N Hx
x Cin x [L2 + S2 + Cin + aa33 + Cout)]
Costnormal
Nx x H W x Cin X Cout x L2
1
S2
3a Oi
a
+
(5)
Cont
Cout
L
L2
Cout
2
where a, Cin and Cout constants, and Cout >> in practice. Notably, the computation cost of our super kernel is reduced to O(1/I,²) compared with normal large kernels. Besides, our super kernel also achieves compelling performance (see Section 4 for results).
3.2 Super kernel flow network
Our Super Kernel Flow (SKFlow) network follows similar framework to GMA [16] except the super kernel modules, which are made up of super kernel blocks. We present the overall network architecture in the following.
All All-pairs correlation cost volume. The all-pairs correlation cost volume proposed by 36 is used to model correlations for all possible displacements. Features from two frames perform a dot-product and then the matching correlations across different levels can be built, namely,
2!
2
d' (i.jmm, m.
x (ix))
2'n +
(6)
221
where x, and X2 are feature maps extracted from input frames, and refers to the correlation level. (.) denotes the inner product function.

---

Frame
Siamest Fealure Encoder
Cost Volume
Frame
Siamese Feature Encoder
Motion Feature
Global Motion Feature
GMA Module
SK MoE.
Frame
Upd.
Context
Encoder
Figure 3: Overview of our proposed SKFlow. SK MoE. and SK Upd. denote our proposed super kernel motion encoder and updater respectively. GMA module the non-local method proposed in [16].
Global motion aggregation (GMA) module. The GMA module adopts the self-attention mecha- nism to model long-range connections between local descriptors. Following GMA [16), we apply the module in the flow decoder. Given the input motion feature with height / and width W, the output is defined as:
H W
o(i,j) x
Σ
(i)) (y, v))
Σ
(i.j),
where gy and ky denote the query and key vector derived from the context feature map. Vi, is the value vector derived from x, f the dot-product attention function and y is learnable coefficient.
Super kernel modules. Super kernel modules consist of (1) Super kernel motion encoder, which shares similar architectures with GMA but applies super kernel blocks. It derives motion features from cost volume vectors с and the predicted flow f, namely,
c' SK Block SK (lock(c))
(8)
f' SK Blocki SK (lock(())
(9)
Concat S Block((Concaattee
(10)
where SK Block denotes proposed super kernel block, and Concat is the concatenation operation. (2) Super kernel updater. Different from the ConvGRU used in RAFT and GMA, we directly adopt our super kernel block the updater to refine the predicted residual flow Af, which could be formulated as:
Af SK Block Block Concall(Xm Xex Xg))
(11)
where Xm x, denote the motion feature and context feature. Xg refers the global motion feature from the GMA module.
The whole architecture of our SKFlow is shown in Figure It is notable that the super kernel blocks are applied in the decoder only based on the following consideration: Applying in the decoder is more efficient due its relatively small size. Given the lack of efficient implementation like normal 3 x 3 kernel for large depth-wise kernels in most libraries, super kernels in those two large encoders can take lots of additional time in inference and training. Although it may be solved when faster implementations occur in the future, currently applying super kernels in the smaller decoder part brings a modest time increase and achieves significant performance improvements.
3.3 Supervision
Following previous works (36 16, we adopt the following loss function to supervise the parameter update. The 4 loss of the predicted flow after every refinement is weighted with the exponentially

---

Table 1: Quantitative results on Sintel and KITTI. We report average End-Point Error(EPE) unless specified. C+T refers the FlyingChairs Sintel, KITTI, and HDIK training sets are combined when finetuning on Sintel. denotes using warm- strategy RAFT 36. SKFlow achieves the best generalization performance on C+T and C+T+S+K+H.
FlyingThings schedule. + K + H denotes that
Sintel (train)
KITTI-15 (train)
Sintel( test)
KITTI-15 (test)
Training Data
Method
Clean
Final
Fl-epe
Fi-all
Clean
Final
Fl-all
4
3.84
8.77
13.17
24.0
PWC-N Net
3
2.55
3.93
10.35
33.7
VCN
40
2.21
3.68
8.36
25.1
MaskFlow
4
2.25
3.61
23.1
FlowNet2
15
2.02
3.54
10,08
30.0
3.96
6.02
C+T
DICL-Flow
38
1.94
3.77
8.70
23.6
RAFT 36
1.43
2.71
5.04
17.4
AGFlow
20
1.31
2.69
4.82
17.0
Separable Flow
42
1.30
2.59
4.60
15.9
GMA 116
1.30
2.74
4.69
17.1
SKFlow (Ours)
1.22
2,46
4.27
15.5
LiteFlowNee1 13
(1.30)
(1.62)
(1.47)
(4.8)
3.48
4.69
7.74
PWC-Net+ 32
(1.71)
(2.34)
(1.50)
(5.3)
3.45
4.60
7.72
MaskFlowNer 43
2.52
4.17
6.10
RAFT 3G
(0.76)
(1,22)
(0.63)
(1.5)
1.61*
2.86
5.10
C+T+S+K+H
AGFlow 20
(0,65)
(1.07)
(0.58)
(1.2)
1.43
2.47
4.89
Separahle Flow
42
(0.69)
(1.10)
(0.69)
(1.6)
1.50
2.67
4.64
GMA 16
(0.62)
(1.06)
(0.57)
(1.2)
1.39
2.47
5.15
FKF (Ours)
(0.52)
(0.78)
(0.51)
(0.94)
1.28
2.27
4.84
increasing coefficients:
N
L
IIʃ: fg
(12)
where ||- 1/1 denotes the l distance between flow ground-truth for and output flow, and represents the predicted flow field at the ith refinement, denotes weights on different predicted flows and set to 0.8 in experiments.
4 Experiments
Experimental setup. Our model is evaluated Sintel [3] and KITTI [22 datasets. We adopt the average Fnnd-Poitt-rrror (EPH) as the evaluation metric, which denotes the mean flow emror over all pixels. KITTI also adopts the FI-All (%) metric, which computes the percentage of pixels with EPE larger than 3 pixels or over 5% of ground truth. Sintel consists of different passes rendered with different levels of diffieulty. Specifically, the Albedo pass is rendered without illumination effects and has approximately piecewise constant colors. The clean pass introduces reflection properties and various illumination. The final pass adds motion blur, atmospheric effects, and other artistic embellishments to the lighting, which is the most challenging. We use the clean and final passes for training and evaluate our methods on all three passes. Notably, our SKFlow design scheme works both on GMA and RAFT networks. Detailed discussion on the RAFT-based SKFlow method is shown in the supplementary materials.
we first pre-train our SKFlow using the FlyingChairs - FlyingThings schedule, and then fine-tune for Sintel with the combined dataset from Sintel, FlyingThinge KITTI and HDIK [18). Finally, we finetune our model on KITTI. We adopt the AdamW [19] optimizer and one-cycle policy (28]. Our SKFlow is built with PyTorch (23] library and trained using two Tesla V100 GPUs.
Implementation details. Following previous works B6 16 42
20

---

3.0
1.40
RAFT
2,8
1.35
2.6
1.30
GMA
1.25
2.4
. Parallel
2.3
1.20
Furnal
2.2
SKFlow (Ours)
Coniral
211
1.15
5.0
6.5
3 Kernel
15
31
Model size (milllonns)
Figure 4: (a) Comparisons between MACs, parameters, and performance. A larger bubble represents higher MACs. (b) The comparison between the three designs with different kernel sizes. The Conical style achieves the lowest error with a 15 x 15 kernel size.
4.1 Quantitative results
20 we evaluate our SKFlow on Sintel and KITTI-2015 dataset. We first evaluate the generalization ability models by pretraining on FlyingChairs and FlyingThings datasets and evaluate on Sintel and KITTI datasets. Next, we compare the performance of each dataset. Models are finetuned on the training combined from Sintel. HDIK, and KITTI datasets, and evaluated on the Sintel and KITTI datasets. The experimental results are shown in Table 1
Following previous works 36
16
Results for cross-dataset generalization. Our SKFlow is first pre-trained on FlyingChairs and FlyingThings datasets and then evaluate on the Sintel and KITTI-2015 datasets. The pross-datase results can be seen in Tables "C+T" rows. From the results, we can see the generalization capability of different methods. Specifically, on the challenging Sintel final pass, our SKFlow obtains the best performance among all published methods, outperforming GMA by 10.21%. On KITTI and clean pass, our method achieves the Fl-epe of 4.27 and Fl-all of 15.5, and the EPE of 1.22, outperforning GMA by 8.96%, 9.36% and 6. 15%, respectively.
Results on Sintel benchmark. The experimental results are shown T From the results, we can see that our SKFlow achieves significant improvements in performance on both Sintel training and test sets among the listed methods. Compared with GMA, our SKFlow achieves the EPE reduction of 7. 7.91% and 8.10% on the clean and final pass of the Sintel test set. On the training clean and final pass, SKFlow outperforms GMA by 16. 13% and 26.42%.
rows.
Results KITTI benchmark. The results are shown in Table N "C+T+S+K+H" rows. Our SKFlow achieves 4.84, achieving competitive results. On the KITTI training set, our method achieves the improvement on GMA with the EPE of 0.51% and of Fl-epe of 0.94%. On the KITTI test set, our method outperforms GMA by 6.02%.
4.2 Model efficiency
We then compare the efficiency of our SKFlow with current state-of-the-e methods. We use MACs. runtime, and model size to measure the model efficiency. Specifically, MACs are measured using PTFLOPS 29 library. The runtime is calculated by the average inference time per frame on the KITTI-15 dataset and the reported value is the average of three runs. Every model is applied to 10 updates during inference on a GTX TITAN GPU. And the model size is measured in the number of parameters. All models are fine-tuned using the combined dataset from Thyingghhnng3D,, HDIK, KITTI, and Sintel training sets. The evaluation is conducted on the Sintel test set. illustrated in Figure 4 our proposed SKFlow shows better accuracy compared with GMA with similar MACs (8.42%) and limited increase in model size (6.63%). The runtime comparison is shown in the supplemental materials. Compared with other architectures that widely adopt the well-optimized

---

3 x 3 convolution, the runtime of SKFlow is a little bit longer in practice, due to the lack of efficient implementation for large depth wise kernels in the current PyTorch library.
4.3 Occlusion analysis
We split pixels into occluded (Occ) and non non-occluded (Noc) with the help of the occlusion maps provided in the Sintel dataset. Following the previous work [16], the model is pre-trained using the C+T schedule and then fine-tuned using the C+T+S+H+K schedule for 150K iterations. The Albedo pass of the Sintel dataset is used verify the effectiveness of estimating occluded motions and is not used training. We choose Albedo pass as the test set because rendered without illumination effects and thus adheres the brightness constancy constraint except in the occluded areas, which highlights the performance on occlusions. The results are shown in table 2 where "Noc" denotes non-occluded pixels, "Oce" denotes the occluded pixels and "All" denotes the overall pixels. We also test their performance on the unmatched areas on the Sintel test set. Detailed results are shown in the supplements. Results on the occluded and non-oceluded areas in the training and test demonstrate that our SKFlow method can better resolve the ambiguity caused by occlusion areas, compared with RAFT and GMA models.
-RAFT
GMA
Data
Type
RAFT
GMA
Ours
13
Noc
0.29
0.27
0.26
12
Clean (train)
Occ
5.00
4.44
3.96
All
0.63
0.58
0.53
1.1
Noc
0.60
0.52
0.46
1
Final (train)
Occ
6.90
6.32
5.39
All
1.06
0.95
0.82
Noc
0.31
0.31
0.27
Albedo (test)
Occ
6.18
5.96
4.75
All
0.74
0.72
0.60
110k
140k
170k
Trainingsteps
Figure 5: EPE curve Sintel during training. All models are trained under the same settings.
Table 2: Performance occluded and non- occluded areas on the Sintel dataset.
4.4 Ablation study
We perform set of ablation experiments show the importance of each component in the network design. The results are shown in Figure (b) and Table 5 respectivell. All ablated versions are trained using the C+T schedule. The settings which used in our final model are underlined. In the following, we describe each of the experiments in more detail.
Training schedules. In particular, we find that the performance of our proposed SKFlow keeps improving when training steps are increasing. Specifically, train the model with additional 30K steps on the PlyingTThing dataset and additional 60K steps for fine-tuning on the Sintel dataset. For fair comparison, we then train GMA and RAFT using the same settings. As illustrated in Figure(5) the EPE of SKFlow keeps decreasing while the training steps are increasing, and an obvious improvement in performance can be obtained compared the RAFT and GMA. We argue that large receptive fields are inherently helpful to capture more motion information and raise the upper bound of the learning ability of networks, although it tends be hard to optimize.
Kernel sizes. We explore the impact of different kernel sizes. Specifically, the kernel size is increased from to 31, growing 2x each time. As shown in Table 5 we can see that kernel size 15 x achieves the best overall performance. Although obtains 0.89% and 0.92% lower performance than 7 x 7 kernel on KITTI, it achieves 6.98% and 5.56% higher in performance on Sintel clean and final pass. The comparison with other designs shown in F (b). We also observe much lower performance when the kemel size is 31, especially small dataset such as

---

Table 3: Results on the FlyingChairs validation set.
Table 4: Results on the Sintel training set.
Model
FlyingChairs
Parameters
Model
Clean
Final
RAFT
0.85
5.26M
GMAMoE GMAGRU
1.30
2.74
RAFT-Deep
0.85
6.00M
(GMAMoE GMAGRU)+
1.36
2.72
3KFlow-RAFE
0.83
5.59M
SKM6E GMAGRU
1.28
2.56
GMA
0.80
5.88M
SKMoF GMA-LargeGRRU
1.32
2.54
GMA-Deep
0.83
6,25M
SKMoF + SKBlock-Smal
1.25
2.56
SKFlow- MMA (Ours)
0.78
6.27M
SKMoE + SKBlock (Ours)
1.22
2.46
KITTI, which indicates the optimization of the extremely large kernel is still challenge given the limited number samples in optical flow datasets.
Super kernel block components. We explore the effect of each component in Section 3.1 Specif- ically, "Res" denotes the residual connections and "Aux" denotes the Auxiliary small kernel. From 3 "Componenss"" we can see that by using both the residual connections and the auxiliary small kernel, the overall performance improves significantly. (comparable result 2.46 vs 2.45 on Sintel Final). Nevertheless, without the auxiliary kernel, there will be significant drop for small dataset like KITTI, indicating the importance of the auxiliary kernel on optimization.
Super kernel block architecture. We explore the performance of three hybrid architecture designs in Section 3.J as shown in Table "rchittctturrr We can see that the conical connection design achieves the best performance, showing strong capability to optimize optical flow networks with large kernels. different from the existing work (7] showing that the parallel re-parameterization structure performs well in high-level tasks such as image classification. On optical flow estimation, our conical structure obtains better performance. There might be two reasons: (1) Differences the distribution and size of datasets. Compared with large datasets such as ImageNet where samples are from real life, most optical flow datasets are synthetic and contain much smaller samples, which makes it harder for large kernels to optimize. (2) Different focus of tasks. Compared with optical flow tasks, high-level vision tasks pay more attention to semantic information instead per-pixel correspondence between images.
Auxiliary kernel size. We also explore different sizes for the auxiliary kernel. According to the results shown in Table kernel size", the most effective auxiliary kernel size is 1. Although 3 x 3 achieves slightly better performance on Sintel, 1 x 1 kernel outperforms x 3 kernel on Fl-epe and Fl-all of the KITTI dataset with more improvement and less computation.
Super kernel modules. We compare the performance of using super kernels in update block and motion encoder and the results are shown in Table "SK modules". Upd. refers to the update block and MoE. denotes the motion encoder. We can see that by using super kernels in the update block the performance 2.65 the Sintel final pass, while by using the super kernels in the motion encoder, the performance is 2.56. When super kernels are applied on both the motion encoder and update block the SKFlow network, the performance on both Sintel and KITTI datasets is further improved significantly.
Deep layers vs. Large kernels. As shown in Table 3 adopting deeper layers brings little im- provement compared with using large kernels. We argue that the deeper layers may lead to harder optimization, and then the receptive field will behave like that in shallow networks, as mentioned in the previous work [7]. All models are trained on the FlyingChairs training set with the same setting. We stack deeper layers in RAFT and GMA decoders so that they have model sizes similar to SKFlow. It also ablates the effect of additional parameters in SKFlow.
Switch from GRU SKBlock. directly increasing the kernel size of GRU architecture tends not to bring performance gain. As shown in Table from GMAGRU to GMA-LargeGRU, the improvement on Sintel (final) tends to be saturated and the performance on Sintel (clean) degrades to a small extent. Nevertheless, from SK (BBloc-Smaall SKBlock, the improvement is quite stable. All models are trained using the C + T schedule under the same settings. Note that GMAGRU and SKBlock-Sma share the same
The motivation for replacing GRU with SKBlock is that we find

---

Table 5: Ablations on super kernel block designs. Models are trained using the C+T schedule.
(trin)
KITTI-I5(traii)
Experiment
Method
Parameters
Clean
Final
Fl-epe
Fl-all
31
1.27
2.65
4.73
17.40
7.08M
15
1.20
2.55
4.51
16.26
6,27M
Large kernel size
7
1.29
2.70
4.47
16.11
6,08M
3
1.30
2.77
4.83
17.31
6,04M
1
1.41
2.83
7.12
21.58
6.03M
No Res
1.35
2.56
4.76
17.33
6.27M
No Aux
1.25
2.45
4.53
16.55
6,27M
Components
No Res & Aux
1.33
2,61
5.04
17.38
6.27M
With Res & Aux
1.22
2.46
4.27
15.47
6,27M
Parallel
1.24
2.53
4.71
17.82
6.27M
Architecture
Funnel
1.19
2.54
4.55
16.97
6.27M
Conical
1.22
2,46
4.27
15.47
6.27M
3
1.21
2.44
4.33
15.56
6.29M
Aux kernel size
1
1.22
2.46
4.27
15.47
6.27M
Upd.
1.37
2.65
4.89
16.89
5.30M
SK modules
MoE.
1.28
2.56
4.60
16.86
6.67M
Upd.+MoE.
1.22
2.46
4.27
15.47
6.27M
kernel size, and GMA-LargeGRU obtains the same kernel size as SKBlock. GMAMoE + GMAGRU denotes the result from the original GMA paper and (GMAMoE + GMAGRU>* our reproduced result for GMA.
4.5 Visualiztions
As shown in Figure O we visualize the predicted optical flows on the sample image from the Sintel test set. Results are highlighted using red circles in the figure, which demonstrates that our proposed SKFlow predicts more accurate flow and captures more details compared with other methods.
o
Ground Truth
RAFT
GMA
SKFlow (Ours)
Figure 6: Visualizations of predicted optical flow among different methods on Sintel test set.
5
Conclusions
In this work, we propose SKFlow, which utilizes large receptive fields brought by super kernels to recover the occluded motion in optical flow estimation. To the best of our knowledge, SKFlow is the first apply the kernel size up to 15 15 to optical flow networks with significant performance gain and modest computation increase. To leverage the computation cost for the large kernels and attains improved accuracy, SKFlow explores and provides effective and efficient design schemes for applying super kernels, such as conical connections and hybrid depth-wiee wise convolutions. Comprehensive experiments have demonstrated the effectiveness of SKFlow. With less than 8.42% increase in MACs, SKFlow outperforms the previous state-of-thee the-art method on the Sintel final pass, ranking 1st among all published methods.
Acknowledgements. This work was supported by National Natural Science Foundation of China (No. 621720..
10

---

References
[1] Michael Black and Padmanabhan Anandan. A framework for the robust estimation of optical flow. In 1993 (4th) International Conference on Computer Vision, pages 231-236. IEFE, 1993.
[2] Andrés Bruhn, Joachim Weickert, and Christoph Schnörr. Lucas/kanade meets horn/schunck: Combining local and global optic flow methods. International journal of computer vision, 61(3):211-331, 2005.
[3] Daniel J Butler, Jonas Wulff, Garrett B Stanley, and Michael J Black. naturalistic open source movie for optical flow evaluation. In European conference computer vision, pages 61 1-65.. Springer, 2012.
[4] Qifeng Chen and Vladlen Koltun. Full flow: Optical flow estimation by global optimization over regular grids. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4706-4714, 2016.
[5] François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 1251-1258, 2017.
[6] Soham De and Sam Smith. Batch normalization biases residual blocks towards the identity function in deep networks. Advances in Neural Information Processing Systems, 33:19964- 19975, 2020.
[7] Xiaohan Ding, Xiangyu Zhang, Yizhuang Zhou, Jungong Han, Guiguang Ding, and Jian Sun. Scaling up your kernels to 31x31: Revisiting large kernel design in cnns. arXiv preprint arXiv:2203.06717, 2022.
[8] Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015.
[9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[10] Berthold KP Horn and Brian G Schunck. Determining optical flow. Artificial intelligence, 17(1-3):185-203, 1981.
[11] Han Hu, Zheng Zhang, Zhenda Xic, and Stephen Lin. Local relation networks for image recognition. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3464-3473, 2019.
[12] Tak Wai Hui, Xiaoou Tang, and Chen Change Loy. Liteflownet: A lightweight convolutional neural network for optical flow estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8981-8989, 2018.
[13] Tak-Wai Hui, Xiaoou Tang, and Chen Change Loy. A lightweight optical flow cnn-revisiting data fidelity and regularization. IEEE transactions on puttern analysis and machine intelligence, 43(8):2555-235 2569, 2020.
[14] Junhwa Hur and Stefan Roth. Iterative residual refinement for joint optical flow and occlusion estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5754-5763, 2019.
[15] Eddy 1lg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
[16] Shihao Jiang, Dylan Campbell, Yao Lu, Hongdong Li, and Richard Hartley. Learning to estimate hidden motions with global motion aggregation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9772-9781, 2021.
[17] Shihao Jiang, Yao Lu, Hongdong Li, and Richard Hartley. Learning optical flow from few matches. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 16592-16600, 2021.

---

[18] Daniel Kondermann, Rahul Nair, Katrin Honauer, Karsten Krispin, Jonas Andrulis, Alexander Brock, Burkhard Gussefeld, Mohsen Rahimimoghaddam, Sabine Hofmann, Claus Brenner, et al. The hei benchmark suite: Stereo and flow ground truth with uncertainties for urban autonomous driving. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 19-28, 2016.
[19] Nya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018.
[20] Ao Luo, Fan Yang, Kunming Luo, Xin Li, Haoqiang Fan, and Shuaicheng Liu. Learning optical flow with adaptive graph reasoning. arXiv preprint arXivv2202.03887, 2022.
[21] Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks, Advances in neural information processing systems,
29, 2016.
[22] Moritz Menze, Christian Heipke, and Andreas Geiger. Joint 3d estimation of vehicles and scene flow. In ISPRS Workshop on Image Sequence Analysis (ISA), 2015.
[23] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De Vitoo, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[24] Chao Peng, Xiangyu Zhang, Gang Yu, Guiming Luo, and Jian Sun. Large kernel matters- improve semantic segmentation by global convolutional network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4353-4361, 2017.
[25] David W Romero, Robert-Jan Bruintjes, Jakub M Tomezak, Erik J Bekkers, Mark Hoogendoorn, and Jan C van Gemert. Flexconv: Continuous kernel convolutions with differentiable kernel sizes. arXiv preprini 1rXxv:2110008000 2021.
[26] David W Romero, Anna Kuzina, Erik J Bekkers, Jakub M Tomezak, and Mark Hoogendoorn. Ckconv: Continuous kernel convolution for sequential data. arXiv preprint (tti::2022.02611 2021.
[27] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
[28] Leslie N Smith and Nicholay Topin. Super-convergenee: Very fast training of neural networks using large learning rates. In Artificial intelligence and machine learning for multi- domain operations applications, volume 1 11006, page 1100612. International Society for Optics and Photonics, 2019.
[29] Vladislav Sovrasov. Flops counter for convolutional networks in pytorch framework, 2019.
[30] Deqing Sun, Stefan Roth, and Michael J Black. A quantitative analysis of current practices in optical flow estimation and the principles behind them. International Journal of Computer Vision, 106(2):115-177, 2014.
[31] Deqing Sun, Xiaodong Yang, Ming- Yu Liu, and Jan Kautz. Pwe-net: Cnns for optical flow using pyramid, warping, and cost volume. Proceedings of the IEEE conference on computer vision and patlern recognition, pages 9943, 2018.
[32] Deqing Sun, Xiaodong Yang. Ming- Yu Liu, and Jan Kautz. Models matter, does training: An empirical study of cnns for optical flow estimation. IEEE transactions on pattern analysis and machine intelligence, (2(6))1408-8 1423, 2019.
[33] Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In Thirty-first AAAI conference on artificial intelligence, 2017.
[34] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9, 2015.
[35] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2826, 2016.
12

---

[36] Zachary Teed and Jia Deng. Raft: Recurrent all-pain field transforms for optical flow. In European conference on computer vision, pages 402-419. Springer, 2020.
[37] Andreas Veit, Michael Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks. Advances in neural information processing systems, 29, 2016.
[38] Jianyuan Wang, Yiran Zhong, Yuchao Dai, Kaihao Zhang, Pan Ji, and Hongdong Li. Displacemenn-invaaiann matching cost learning for accurate optical flow estimation. Advances in Neural Information Processing Systems, 33:15220-15231 2020.
[39] Haofei Xu, Jiaolong Yang. Jianfei Cai, Juyong Zhang, and Xin Tong. High-resolution opti- cal flow from 1d attention and correlation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10498-10507, 2021.
[40] Gengshan Yang and Deva Ramanan. Volumetric correspondence networks for optical flow. Advances in neural information processing systems, 32, 2019.
[41] Zhichao Yin, Trevor Darrell, and Fisher Yu. Hierarchical discrete distribution decomposition for match density estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6044-6053, 2019.
[42] Feihu Zhang, Oliver Woodford, Victor Adrian Prisacariu, and Philip HS Torr. Separable flow: Learning motion cost volumes for optical flow estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10807-10817, 2021.
[43] Shengyu Zhao, Yilun Sheng, Yue Dong, Eric Chang, Yan Xu, al. Maskflownet: Asymmetric feature matching with learnable occlusion mask. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6278-6287 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [No]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions nceded to reproduce the main experi- mental results (cither in the supplemental material or URL)? [Yes] See supplemen-
tal materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're
using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content? [No]
13

---

5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "rbbhJQsyQrP", "reviewer": "Reviewer_Qxdd", "summary": "This paper proposes a CNN-based method that deals with optical flow estimation, especially to tackle occluded areas. To that end, the proposed method studies the relationship between the receptive fields of the network and the effectiveness of recovering occluded motions. The proposed method designs the network using super kernel blocks, which are composed of small size depth-wise convolutions followed by a set of large-size convolutions. Comprehensive experiments demonstrate that the proposed method is quite effective and outperforms the state-of-the-art baseline methods. ", "questions": "1. Usually KITTI-2015 dataset contains more large displacement / fast motion (than Sintel Dataset), which introduces considerably more occluded areas. But the performance of the network is not better than its predecessor network RAFT. Can the authors provide more discussion about it? ", "limitations": "The main limitation is that the paper does not describe clearly the reason why the Conical architecture is the key reason to improve the performance of the proposed network in dealing with occlusion areas. It is not straightforward/intuitive relationship between the receptive field and the occluded area. So it requires the paper to make a more sound analysis and insightful theoretical development. ", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The proposed method consistently outperforms the baseline methods on mainstream optical flow benchmarking datasets, including Sintel (synthetic), KITTI-15(real).  The proposed method is submitted to the Sintel benchmark platform and obtained the state-of-the-art ranking on that dataset. \n\n2. This paper is generally clearly written with a proper literature review. The introduction part has stated the motivation of the paper clearly and specifies the challenges of the existing methods as well as the key component to deal with the challenges.  The method section has delivered a clear architecture by using visual graphs and proper citation. The experiments part have gone through a systematic way to run benchmarks. \n\nWeaknesses\n1. This paper's main problem lies in the theory part of the approach section. It does not specify the theory behind the super kernel to validate the reason for the effectiveness. Even though, we understand that the architecture outperforms other methods in the experiments. But it lacks the reasoning why the architecture is designed in that way in the first place. \n\n2. In the experiment section, the description says the models are trained on more challenging data (i.e. Sintel clean and final) but tested on the intrinsic components of the scene (i.e. Albedo), which usually cannot validate the effectiveness of the methods. But in Table 1, the results are conducted on the clean and final datasets.  \n\n3. The authors are suggested to conduct one more ablation study to verify that the improved performance is not coming from more parameters compared with the baseline RAFT and GMA method by running a vanilla network with the same number of parameter networks. \n\n4. Based on the ablation study, it seems that SK modules ( motion encoder and updater ) affect the performance much more than the conical architecture compared with other architecture settings. But the paper claims that the key effective components should be the conical design of the network. Can the authors explain and validate more on this issue?\n\n\n\n\n5. Many typos: \ni). All the equations (1) -(11) lack punctuation. \nii). Line 173, 182, etc, the short headline should not come with full stop. \niii). e\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "eMZgaCbctpr", "reviewer": "Reviewer_vu4D", "summary": "The authors propose to improve optical flow estimation performance in occluded regions by increasing the receptive field size of the network. The idea itself is straightforward, but there are technical challenges in implementing it. In particular, optical flow datasets are small in size, thus simply optimizing a very deep network is not a viable option. Directly increasing the kernel size also does not work well due to especially high memory requirements during flow estimation and optimization difficulties associated with large kernel networks. To address these issues, the authors propose two architectural tricks - ones is using depth-wise-separable convolutions, and another is adding another 1x1 convolutional layer before the large kernel convolution to simplify optimization. Extensive experimental analysis demonstrates that the resulting approach outperforms the state-of-the-art by non-negligible margins with particularly large improvements in occluded regions. An extensive ablation analysis demonstrates the importance of the most of the architectural decisions for the final performance.", "questions": "Please provide a motivation to for the switch from GRU to a residual network and ablate it.\n\nPlease report the number of iterations used during inference and explain the choice of lambda.", "limitations": "None", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe proposed idea of increasing the performance of optical flow estimation in occluded regions by increasing the size of the receptive field is sound.\n\nThe proposed architectural tweaks to the GMA architecture to make training with large kernels viable are also sound and their effectiveness is experimentally demonstrated. \n\nThe final approach outperforms the state-of-the-art in most scenarios, with improvements being especially significant in occluded regions. \n\nAn extensive abalone analysis is provided.\n\nThe paper is readable. \n\n\nWeakness:\n\nThe novelty is mainly in the architectural tweaks to the GMA architecture to make training it with large kernels on small datasets viable. \n\nThere are still quite a lot of grammatical mistakes in the text.\n\nAlthough the ablation analysis is quite extensive, a few important points are not addressed. In particular, the switch from a GRU to a residual network is neither motivated nor ablated, and the choice of the hyper parameters is not explained (number of iterations during inference, lambda parameter in the loss).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "TQOXHKQKWJd", "reviewer": "Reviewer_dtXq", "summary": "This work addresses the problem of optical flow estimation and is based on RAFT and GMA. The paper proposes to use a larger kernel in convolutional neural networks to solve the problem of occlusion occurring in optical flow. A super kernel block with depth-wise and channel-wise convolution is introduced to reduce the computational cost. The experimental result shows state-of-the-art performance.", "questions": "1. Line 127 is a little bit unclear about the computation cost. How is the cost obtained in more detail?\n2. When is the computational cost of SKFlow smaller than the normal kernel convolutional method? Is the complexity related to the number of input channels and output channels and the size of the kernel? For the example in the paper (kernel size of 15), what is the ratio of cost_ours and cost_normal?\n3. In line 276, it is said that the most effective auxiliary kernel size is 1*1. But in that case, isn’t depth-wise convolution just a multiplication? Where does the better performance come from?\n4. Experiment: Why is GMA chosen as the baseline? Is there a specific reason? Have other baselines been explored using the super kernel block? If so, what results have been obtained?\n5. Has the proposed approach been compared to a traditional large kernel approach? For example, using a kernel size of 15 to replace the super kernel block. Is this possible? How large is the computational cost in comparison to the super kernel block?\n\nAdditional questions of interest:\n\n6. The occlusion analysis is only tested on the Albedo pass, how about testing on the Clean pass and the Final pass? How does it perform?\n\n7. Have deeper layers been tried instead of a larger kernel to enlarge the receptive field? Does it perform badly or is it just a reasonable assumption that using a larger kernel will have a better result?", "limitations": "The paper did not mention any limitations or negative societal impacts. The latter might be difficult, but optical flow in general has its limitations regarding the estimation of the (projected) motion. Isn't it time to also consider learning to estimate 3D motion or the projected motion instead of optical flow?", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is clearly written with the baseline method and the contribution.\n2. The proposed super kernel block is simple in structure and makes generalization to other networks possible.\n3. The ablation study is quite comprehensive.\nWeaknesses:\n1. Using a larger kernel in optical flow has been proposed before, see line 40 and [7,11,24-26]. The extension to the proposed super kernel is rather incremental.\n2. The proposed extension is only applied to the GMA architecture and it remains unclear whether the super kernel block will improve results for other architectures.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8miE5slXDVc", "reviewer": "Reviewer_64b1", "summary": "In this work, the authors proposed a simple but effective way to handle occlusion in optical flow estimation. The main idea is to use a kernel with a large reception field. Unlike previous work that simply increased the depth of network to increase network, the authors proposed a new super-kernel block, which uses a super block that consists of a large and a small kernel. Experiments have demonstrated the proposed network can significantly boost the accuracy of optical flow, with minimal additional compute.", "questions": "### I only have two minor questions about this work.\n\nAnother way to increase the receptive field without introducing too many network parameters is dilated convolution. It would be great if the authors can discuss Pros and Cons of proposed approach compared to dilated convolution.\n\nFor efficiency evaluation, I think it is also important to report actual runtime, in addition to FLOPS/MACs, as due to cache locality and other memory issues, FLOPS/MACs do not always lead to a low execution time, particularly when a large kernel is used like this work. For completeness, it is better to report runtime on a certain device.\n\n### I also have a few suggestions\n\nThe authors did not need to reply to this question in rebuttal, but would be great to answer them in the camera-ready version.\n\nFunnel v.s. Conical. In theory, convolutional kernels are mostly interchangeable, so I would expect funnel v.s. Conical should achieve similar results. Experiments actually show that conical performs slightly better than funnel (Table 3). It would be great if authors could share some insights, even preliminarily, why funnel performs better, and why it performs better than parallel on the selected datasets (I understood datasets could make difference, but would still be interesting to the audience if the authors give more insights). \n\nThe proposed super kernel is a very general design. It would be great if the authors could also discuss the potential usage of this structure in other image processing tasks in conclusions / future work.\n", "limitations": "No as far as I know.", "rating": 9, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is a very high quality work. The authors proposed a very simple but also very effective network to increase the receptive field of the flow estimation network. The authors achieved the best accuracy in the Sintel benchmark, and also did a very thorough experiment to demonstrate effectiveness of the proposed network on different setups and how different alternative designs may work. At last, although this work mainly focuses on motion estimation, the proposed architecture is very general, and I believe it could benefit many other image processing tasks and even beyond.\n\nI don't find any particular weak point in this work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept