DFlow: Learning to Synthesize Better Optical Flow Datasets via a Differentiable Pipeline
Authors: Kwon Byung-Ki
Nam Hyeon-Woo
Ji-Yun Kim
Tae-Hyun Oh
Venue: ICLR 2023 poster (2023)
Keywords: Synthetic data
Optical flow

Abstract: Comprehensive studies of synthetic optical flow datasets have attempted to reveal what properties lead to accuracy improvement in learning-based optical flow estimation. However, manually identifying and verifying the properties that contribute to accurate optical flow estimation require large-scale trial-and-error experiments with iteratively generating whole synthetic datasets and training on them, \ie, impractical. To address this challenge, we propose a differentiable optical flow data generation pipeline and a loss function to drive the pipeline, called DFlow. DFlow efficiently synthesizes a dataset effective for a target domain without the need for cumbersome try-and-errors.  This favorable property is achieved by proposing an efficient dataset comparison method that uses neural networks to approximately encode each dataset and compares the proxy networks instead of explicitly comparing datasets in a pairwise way. Our experiments show the competitive performance of our DFlow against the prior arts in pre-training. Furthermore, compared to competing datasets, DFlow achieves the best fine-tuning performance on the Sintel public benchmark with RAFT.

Published as a conference paper at ICLR 2023
DFLOW: LEARNING TO SYNTHESIZE BETTER OPTICAL FLOW DATASETS VIA A DIFFERENTIABLE PIPELINE
Kwon Byung-Ki', Nam Hyeon-Woo', Ji- Jun Kim', Tae-Hyun Oh!,2, 12,3
'Department of Electrical Engineering, POSTECH Graduate School of AI, POSTECH Institute for Convergence Research and Education Advanced Technology, Yonsei University pruungiikwonn hyeonw... aemonw.na,, junekim, (ehyyun@epsseech...
ABSTRACT
Comprehensive studies of synthetic optical flow datasets have attempted to reveal what properties lead to accuracy improvement in learning- based optical flow esti- mation. However, manually identifying and verifying the properties that contribute accurate optical flow estimation require large-scale trial-and-eree experiments with iteratively generating whole synthetic datasets and training on them, ie., impractical. To address this challenge, we propose differentiable optical flow data generation pipeline and loss function to drive the pipeline, called DFlow. DFlow efficiently synthesizes a dataset effective for target domain without the need for cumbersome try-and-eroors This favorable property is achieved by proposing an efficient dataset comparison method that uses neural networks to approximately encode each dataset and compares the proxy networks instead of explicitly compar- ing datasets in pairwise way. Our experiments show the competitive performance of our DFlow against the prior arts in pre-training. Furthermore, compared to competing datasets, DFlow achieves the best fine-tuning performance on the Sintel public benchmark with RAFT.
INTRODUCTION
Optical flow is fundamental computer vision problem to find dense pixel-wise correspondences between two subsequent frames in a video. Optical flow is indeed a key building block in many practical applications. including video understanding, action analysis, video enhancement, editing, 3D vision, etc. Recently, optical flow has been significantly advanced by learning-based approaches with deep neural networks (Fischer et al., 2015; Ilg al., 2017; Ranjan & Black, 2017; Hui et al., 2018; Sun et al., 2018; Teed & Deng, 2020) in terms of accuracy and efficiency.
A driving force of these prior arts is large-scat scale supervised datasets. However, it is difficult to collect a reasonable amount of real-world optical flow labels. Thus, they exploited large-scale synthetic datasets, e.g., Fischer et al. (2015); Mayer et al. (2016), which has become the standard in optical flow, e.g., training on FlyingChairs (Fischer et al., 2015) followed by Flying TriingThing3DD (Mayer et al., 2016). After the seminal studies, there have been various efforts to build different synthetic datasets (Gaidon et al., 6; Richter et al., 2017; Lv et al., 2018 8; Oh et al., 2018; Aleotti et al., 2021). Despite the vast efforts of these studies, remains unclear which factors are important for an effective synthetic dataset construction against the target domain.
Instead manually identifying important design criteria, AutoFlow (Sun al., 2021) pioneers the first learning-bant based approach to go beyond being heuristic by posing data generation as a hyperparameter optimization problem maximizing validation performance on target dataset. AutoFlow generates data samples by composing simple 2D layers with non-differentiable hyperparameters, which are optimized by sampling- based evolutionary search. The use of evolutionary search requires large resources, which burdensome because each target scenario requires to re-generate different datasets.
To address this challenge, we propose DFlow, which is an efficient synthetic optical flow dataset generation method. We compose each data sample by simple differentiable graphic operations, such as warping layer and real- world effects, so that each sample can be parameterized a learnable manner. This allows to exploit efficient gradient descent methods to generate each sample, and

---

Published as a conference paper at ICLR 2023
thereby DFlow is more than order of magnitude efficient than AutoFlow in GPU hours when constructing the same amount of training data.
We also introduce a new loss function that learns the data parameters by contrasting target dataset from a base dataset, e.g., FlyingChairs. Since directly using large datasets the contrastive learning process is cumbersome, on respective datasets as proxies. This approximation allows an end-to-end differentiable pipeline from the data parameters the loss function.
approximate the base and target datasets by two neural networks trained
Through comprehensive experiments, we show that DFlow is effective in both pre-training and fine- tuning. The DFlow data has size of 512 x 384, which is the same FlyingChairs, but the RAFT network (Teed & Deng, 2020) pre-trained on DFlow achieves comparable performance compared to the high-resolution competing datasets (Sun et al., 2021; Mayer et al., 2016). In addition, compared to competing datasets, the RAFT model initially pre-trained on DFlow achieves the best fine-tuning performance on the Sintel public benchmark. We summarize our contributions as follows:
A simple and efficient differentiable data generation pipeline for optical flow (refer to Table 1);
A contrastive-style learning scheme and its loss function by approximating expensive dataset-to- dataset comparison by leveraging proxy neural networks (refer to Sec 3);
Compared to competing datasets, DFlow achieves the best fine-tuning performance the Sintel public benchmark with RAFT. (refer to Table 4).
2 RELATED WORK
Optical Flow. Dense optical flow estimation is to find pixel-wise correspondences from the bright- ness patterns of images (Gibson, 1950; Gibson & Carmichael, 1966; Horn & Schunck, 1981). After conventional optimization algorithms (Black Anandan, 1993; Zach et al., 2007), deep-learning algorithms (Fischer et al., 2015; 1lg et al.. 2017) become dominant due to their computational effi- ciency and reasonable performance. Prior arts (Xu al., 2017; Bailer et al., 2017: Wulff et al., 2017; Sun et al., 2018) have attempted to implement explicit neural modules that are suitable for optical flow estimation. Recently, RAFT (Teed & Deng, 2020) adopts recurrent architectures and achieves a notable performance improvement, which is represented as state-of-the-art.
Those recent advances learning- -based approaches require large-scale data with ground-truth, but labeling dense optical flow is a highly undetermined task, i.e., challenging (Fischer et al., 2015). The previous real-world datasets have been built under sophisticated labeling conditions, including the special sensor hardware, controlled environment, limited objects (Scharstein & Szeliski, 2002; Scharstein & Pal, 2007; Geiger et al., 2012; Kondermann et al., 2014). It leads to the limitation the size of datasets. To relieve this issue, synthetic datasets (Fischer et al., 2015; Mayer et al., 2016) have been proposed and achieved remarkable accuracy despite the gap between real and synthetic datasets. After that, previous
endeavor construct more realistic synthetic datasets (Gaidon al., 2016; Richter et al., 2017: Lv et al., 2018). The prior arts (Alcotti et al., 2021; Han et al., 2022) generate the subsequent frames and ground-truth optical flow by warping the previous frame. These do not handle the photometric inconsistency that is common in real-world scenes. In this work, we propose a differentiable synthetic data generation pipeline with the target knowledge so that the generated dataset could improve its performance more.
cearning-based Optical Flow Dataset. AutoFlow (Sun et al., 2021) is the first learning- data generation approach in optical flow, but with the sampling-buus based evolutionary search for non- differentiable optimization. It is our closest related work in the sense that they Iearn the data generation parameters for performance improvement on the specific target dataset. However, distinctively, our method is the first differentiable method to learn data generation parameters, which leads to more efficient pipeline than AutoFlow in terms of computation cost and GPU hours for data generation. We list other differences in Table 1. Recently, RealFlow (Han et al., 2022) proposes iterative learning framework that learns enhanced flow estimation and pseudo ground -tuutt generation, alternatively. Different from our work, this work suggests framework including iterative model training and dataset generation, and does not suggest a goodness measure of a resulting dataset, which we address. Other than optical flow, there are also interesting attempts generate synthetic data in learnable
2

---

Published as a conference paper at ICLR 2023
AutoFlow (Sun ct al., 2021)
DFlow (Ours)
Learning method
Evolutionary search P100 GPUs
Gradient descent
Computational resource
single V100 GPU
GPU hours for constructing dataset
336 days / P100
9.3 days V100 512 384
Data resolution
1280 x 720
Table 1: Comparison of dataset generation methods of AutoFlow and DFlow (ours).
ways (Sixt et al., 2018; Yang & Deng, 2020; Kaspar et al., 2019; Zhao et al., 2021) for other computer vision applications.
3 DIFFERENTIABLE DATA GENERATION PIPELINE
Our data generation pipeline synthesizes subsequent frames L, and [[+1, and a flow (motion) map GT between those two frames. We concisely denote pair of subsequent frames and a flow map as data sample. We parameterize each data sample with a learnable parameter vector 0 that characterizes a composition of elemental graphic operations, such as color perturbation and geometric warping. We refer to the composition process as data generator, which is designed to be differentiable.
With this differentiable data generator part, we specifically seek to synthesize optical flow data improving the accuracy on the target dataset. We define loss function Liccal Lask + Lreg to effectively update the learnable parameters, which drives the data generation an improving direction. With this loss, we efficiently learn the parameters {0} in fully-differentiable way.
Differentiable data generation pipeline (DFlow)
Data parameter
Differentiable data generator
Loss
(O
Leask Leng
Update (0')
Figure 1: An overview of DFlow. We parameter-
The overview of our data generation pipeline is illustrated in Fig. 1, called DFlow. In Sec. 3.1, we first describe the design of the loss func- tion. In Sec. 3.2, we describe the data generator method from the data parameters {0}. Further detailed descriptions and implementation details can be found in Appendix A.
ize data samples to be synthesized. The learnable data parameters {0} I are updated by our proposed loss function.
3.1 TASK LOSS FUNCTION
Given our differentiable data generator, a sample parameter is rendered to a data sample, i.e., a pair of subsequent frames I, (0) and I (0) and those corresponding flow map FoT By rendering a collection of data samples with the data generator and a set of {0). we can construct a dataset. To find an updated parameter 0* that improves the accuracy of the optical flow model, need an effective criterion to drive the parameter update.
As tractable criterion efficiently find such a dataset, we are motivated by learning-to-augee and dataset condensation approaches (Sixt et al., 2018; Kaspar et al., 2019; Zhao et al., 2021). They synthesize data have characteristics similar target data under the motivation that learning with two similar datasets is likely to yield accuracy improvement (Ben-David et al., 2010; Kaspar al., 2019), which is typically achieved by distribution matching. However, directly comparing combinatorial pairs of samples in the target dataset and synthesized one is computationally expensive, and even intractable during iteratively updating a synthesized dataset.
To efficiently deal with this issue, we propose to encode the target dataset into proxy neural network, called target network. With an optical flow neural network furget (I,, I.+1) - F, we pre-train the network with given small target optical flow dataset, which yields the trained target network. We deem the target network as a differentiable proxy of the target dataset that encodes the knowledge of target data. Compared to using the target data differentiable proxy allows the whole procedure to be tractable. With the target network, we define target loss Lurger (Furgea (l), I++ . (o)( that measures flow errors for data sample associated with @ in the view of the target domain. We use the same loss used for pre-training the target network, e.g., the sequence loss for RAFT (Teed & Deng, 2020), for the target loss Laarge (-)-
set in set-to-set comparison, leveraging this
3

---

Published as a conference paper at ICLR 2023
With the target network and loss, could first attempt to update the parameters {0) by
{0") }- arg min (0) Σ, Lrangr (Surges (I,)),,, I++1 (0.)). Fcr((:))
(I)
If the target loss is small for a data sample, means that the target network is familiar with the given data sample, and the sample is close to one of the target data, ie., similar characteristics to the target data. In this sense, Eq. (1) distills the target knowledge into generated data. However, in our empirical preliminary study, we observed instability and under-fitting issues with Eq. (1), where optimizing the loss is stuck at high value. We assume that target networks trained on the popular benchmarks (Butler et al., 2012; Menze & Geiger, 2015) may not produce sufficiently rich training signals for synthesizing data, which might be due to limited scales of those real-worke benchmarks.
To complement more informative training signals, we employ the contrastive-style learning scheme by using another comparator network, called base network, which is trained with a common large-scale synthetic dataset, e.g., (Fischer et al., 2015; Mayer et al., 2016), base dataset. Similar to the target network, we pre-train the base network but on large-scale dataset, FlyingChairs (Fischer et al., 2015), which is randomly generated. We use the base network similarly to the target network except for maximizing flow errors Lhuse to implement a contrastive behavior, so that a resulting data sample should be closer to the target dataset while being different from the base (common) dataset. This contrastive behavior can generate combinatorial diversity of gradient signals by pairwise contrasting, which may more strongly drive the data generation to an improving direction in favor of target domain than using the target network alone. To implement the contrastive behavior, we define a wrapping loss function, Lunk (Cuarget, Lruse) ): R 2 RR,R, called task loss.
What Function Type is Suitable for Lusk There are many candidates for Lunk satisfying the aforemen- tioned criteria. As a function form, we are first mo- tivated by sample-wise weighting distillation (Zhou et al., 2021), where the loss function is dynamically weighted according to each data sample. The loss function weighted by the target and base losses in multiplication form. We additionally examine addition forms to select the best one. We consider tanh, sigmoid, and exponential functions to imple- ment bounded contrasting behaviors.' Table 2 shows the accuracy comparison of different forms of loss functions using 1, 000 data samples generated with the Sintel dataset (Butler al., 2012) as target. The addition form with the exponential function shows promising results in terms of performance, which corresponds
Dataset
Last
Sintel clean
Sintel final
Exponential
4.26
4.49
Multiplication
Sigmoid
4.12
4.35
Tanh
2.02
3.35
Exponential
1.85
3.09
Addition
Sigmoid
2,00
3.13
Tanh
1.95
3.21
'Bold denotes the best.
Table 2: AEPE of different forms of Ltask. We train the RAFT network with different loss forms. The addition form with exponential shows promising results. Refer to Appendix A.2 for details.
Lusk (Lungett Lowse) = Lunget + a exp((33 Creee + +7),
Lugar
where we set the balance parameters (B,7) to (1,0), and especially set to 20 for the experiments in Table 2. We found that, depending on a, generated data characteristics are distinctive. When generating our final dataset, we first generate subsets of data with different a values, and ensemble the subsets into a single dataset by taking union, denoted as {a} combination. This is found to be notably effective. More details of the total loss be found in Appendix.
3.2 DATA GENERATOR
In this section, we describe the differentiable data generator that synthesizes a data sample from a parameter 0. As shown in Fig. 2, the data generator is composed of geometric warping, flow field translation, layer composition, color perturbation, and real-world effects. The data generator takes N pairs of layer images and masks, [L J1=1 and (M) J,N, Similar to Oh et al. (2018), We randomly sample the layer images from public image datasets (Kuznetsova et al., 2020; Perazzi et al., 2016; Mayer et al., 2016; Lin et al., 2014) and layer masks from segmentation dataset (Everingham et al.,
We observe that contrasting without bounds may lead to instability and divergence. We list the specific equations of the used loss functions in Appendix A.2.
4

---

Published as a conference paper at ICLR 2023
Color perturbation
Geometric warping
Geometric warpiing
MY
MA.
M
Flow field trunslation
Whise halance g²
Grld guin Sice
M
B gaik
Tranclation
B BlA-k)na
Pereective I24
warping lil
Real-world effects
Layer composition
conmining learnable
Texture noise
Fog
Step nit containing leamable panameter
leez
Noise Ny
Aniostropic Goussia kemel RX*
Leamable paraeetess (dimetsioo)
For
Nise Neaa g
Figure 2: Schematic of data generator. The data generator composed of color perturbation, two steps of geometric warpings, flow field translation, layer composition, and real-world effects. The whole pipeline is differentiably parameterized.
Algorithm 1: PyTorch-style pseudo-code for data generator.
#1, 72: geonetria a warping parameter
Ds: franslation parameters
white balance parameters
Real-world effect parameters
imageso, masks0:
initial layer images and masks
Applying ng color perturbation images0 ColorPertuzbation (images0,
Warping images masks
imagesl, masks1 GeometricMarping (images0, masks0, w2s Flowfieldlranslatiiii (W2, Ds)
images2, masks2 GeometricWarping (imagesl, maskal, W2s)
Superimposing the images, masks, and flow fields imagel LayerComposition (imagesl, masksl) image2 LayerCompssition (images2, masks2)
flow LayerComposition (w2s, masksl)
Applying real-world effects the subsequent images imagel, image2 kealxorldfffect (imagel, image2, R)
2010). The data generator then outputs synthesized subsequent frames I, and I441 with ground-truth optical flow FGT. All the following pipeline is parameterized by 0.
Geometric Warping. Geometric warping consists of rigid transformation, perspective warping, and grid warping. The geometric warping generates a warping field. In the first geometric warping, we 1 apply the same warping field Wo-141 [L and {M] and generate the layer images and masks the frame t 1, [ W.V and 1
addition this globally shared geometric warping, we model the local movement of each layer, i.e., segmented objects, in the second geometric warping step. To generate complex optical flows of the frame we can apply independent warping fields to each layer image and mask. However, we observe poor optimization behaviors when we use independent warping fields on each layer. Thus, we propose to use decomposed warping parameters for time / to reduce the number of parameters. We use anchor geometric warping, W Wi+1++4, shared across all layers and flow field translation (Ar, Ay), (D' R2 R each layer, and construct each warping of layers by w! W = (D' Wo+1-1),, where o denotes the warping operation. This strategy also beneficial in terms of optimization stability. Finally, the operations of geometric warping and flow field translation are as follows:
1'4 Wo L
L w/
W L'4 (D's W L/41
(3)
M 41 Wo o M M w/
W/ M2 (D' W M
(4)
5

---

Published as a conference paper at ICLR 2023
Figure 3: Generated data samples. (top) current frames; (bottom) optical flow visualizations.
Layer Composition. After the geometric warping, we have N pairs of layer images, masks, and warping fields in order of depth. We superimpose each stack of layers to generate subsequent frames I and 4++1 with ground-truth optical flow FGT. We can leverage alpha blending and softmax splatting (Niklaus & Liu, 2020). Both strategies show comparable performance, but mainly use the softmax splatring strategy in our experiments. The comparison of alpha blending and softmax splatting can be found in Sec. 4.
Color Perturbation and Real-world Effects. Prior studies (Mayer et al., 2018; Sun et al., 2021) have shown that synthetic data containing real-world effects, such texture noises, fog. and motion blur, often brings the performance improvement of optical flow networks in generalization. Inspired introduce color perturbation and real-world effects into our data generator as
by the observations, well. Color perturbation adjusts the white balance of each layer image, and our real-world effects apply texture noises, fog, and motion blur, which are all parameterized to be controlled and updated.
We synthesize optical flow data by applying the above components: {L))" {Mi)" + {I,, I++1, F F Note that we apply regularizations to the grid warping and texture noise, i.e. grid and noise regularizations. The detail of each component, including regularizations, can be found in Appendix A.3.A.6. The overall differentiable data generator pipeline is summarized in Algorithm as pseudo-code. Each component has its own parameters, which are updated by our task loss and regularizations. See generated samples in Fig. 3.
Summarization. We provide the summarization of DFlow below:
1. We train base and target networks on base and target datasets, respectively.
2. We fix both networks and update the data parameter 0 using our loss function.
3. We train optical flow networks with the generated dataset.
4 RESULTS
In this section, we analyze the effects of our method in pre-training and fine-tuning perspectives with different optical flow models. We report the average end-point error (AEPE) for Sintel (Butler et al., 2012) and ABPE& & Fl for KITTI 2015 (Menze & Geiger, 2015). From this section, we refer to our dataset generated by targeting Sintel with RAFT proxy models as DFlow, unless specified otherwise. Other details of experiment setups and implementation can be found Appendix
Pre-training RAFT Results. Pre-training performance is one of the key factors in evaluating the
applicability of optical flow datasets (Fischer et al., 2015; Mayer et al., 2016; 2018; Aleotti et al., 2021; Sun al., 2021) by testing benchmarks, i.e., Sintel and KITTI 2015 datasets. We train the RAFT networks (Teed& & Deng, 2020) from scratch on respective competing datasets, including our DFlow. Table 3-(a) shows the pre- pra-training results of each dataset. Compared to FlyingChairs which has the same data resolution as DFlow, the model trained on DFlow outperforms the one trained on FlyingChairs in both Sintel and KITTI 2015 datasets. Despite the fact that DFlow is around resolution of AutoFlow, DFlow achieves the best performance on the KITTI 2015 and Sintel clean datasets. We postulate that the real-world textures used in DFlow led to performance improvement on KITTI 2015. Also, we use both Sintel clean and final as the target datasets to generate DFlow, unlike AutoFlow using Sintel final only. It might affect performance on the Sintel clean and final datasets. Figure 4 shows the qualitative results obtained from FlyingChairs and DFlow. These results show that DFlow is effective for learning an accurate model in challenging scenes, such as shaded, foggy, and motion-blurred scenes.

---

Published as a conference paper at ICLR 2023
Table Pre-training results. We train the (a) RAFT (Teed & Deng, 2020) and (b) FlowNet (Fischer et al., 2015) (c) GMA (Jiang et al., 2021) networks on respective pre-training datasets, and evaluate on the evaluation datasets, i.e., Sintel and KITTI 2015. Things denotes the heterogeneous dataset experiment pre-training on FlyingChairs followed by FlyingThingsee
Evaluation dataset
Model
Pre- tre-training dataset
Sintel clean
Sintel final
KITTI 2015
AEPE
AEPE
AEPE FI
Flying
2.28
4.51
9.85 37.56
AutoFlow
2.08
2.75
4.66/
(a) RAFT
DFlow (Ours)
1.81
2.93
4.59/ 15.03
Thaigs-+Things
1.43
2.71
5,04 17.40
FlyingChairs
4.71
6.22
20.36
(b) FlowNet
AutoPlow
5.43
19.64 43.95
DFlow (Ours)
3.45
4.73
12.94 38.95
Chairs- >Things
4.69 / 17.10
(c) GMA
DFlow-GMA (Ours)
4.47 13.10
*Blld denotes the best.
Iuput Image &
Ground Truth
FlytayCheirs
DFlow
Ground truth
RhingCheirs
DFlow
Figure 4: Qualitative results. Top: pre-training results on the Sintel final pass. Bottom: pre-training results on KITTI 2015. The RAFT network pre-trained on DFlow shows robust results challenging shaded, foggy, and motion-blurred scenes.
Model Generalization Property. Since DFlow is generated with RAFT proxy networks, a question naturally arises about the effectiveness of the generated dataset and data generation pipeline on other model architectures. To evaluate the generalization ability of the generated dataset, i.e., DFlow, we pre-train the Flownet network on the DFlow dataset generated with the RAFT network. As shown in Table 3-(b), FlowNet pre-trained on DFlow shows noticeable improvement over the ones pre-trained on the competing datasets. We further investigate the generalization ability of the data generation pipeline. this experiment, we select the KITTI 2015 dataset as the target dataset and generate DF1ow-GMA
the DFlow-GMA dataset using the GMA network (Jiang et al., 2021). Using dataset, we train the GMA network and list the performance on KITTI 2015 in Table 3-(c). The GMA network trained
FlyingChairs followed by DFlow-GMA achieves higher performance than the one trained on From these observations, we postulate that our generated data and generation pipeline agnostic to the deep network structure.
7

---

Published as a conference paper at ICLR 2023
Table 4: Fine-tuning results on public benchmarks. We report Fl scores for KITTI 2015 and AEPE for Sintel public benchmarks. C, K. S, T, and H denote FlyingChairs, KITTI 2015, Sintel, FlyingThings3D, and HDIK (Kondermann al., 2014). The results of Sintel public benchmarks are evaluated without the warm-start initialization (Teed & Deng, 2020).
Dataset schedule
Sintel clean
Sintel final
KIT'I 2015
C-T TSKH/K (Teed & Deng, 2020)
1.94
3.18
5.10
AutoFlow + TSKHV (Sun et al., 2021)
2.01
3.14
DHlow (Ours) - TSKH/K
1.62
3.07
5.03
*Bold denotes the best. Table 5: Analysis of data generation components. We analyze the effects of each generation component. The experiments are conducted by adding or removing one of the generation components. The default setting is with all components and the number of foregrounds parameter in the range of 8 to 12 except the {ar combination, which is annotated with underlines.
Evaluation dataset
Sintel clean
Sintel final
KITTI 2015
Experiments
AEPE
AEPE
AEPE/F F1
1.86
3.04
5,21 16.27 17.31 / 46.84
Data update
9.70
9.67
Off On
1.86
3.04
5,21 16.27 4.90 15.78
{a} combination
1.90
3.02
softmax splatting alpha blending
1.86
3.04
5,21 16.27
Layer composition
1.95
2.83
5.23 / 16.06
4.10
4.89
7,56 23.76
Number of foregrounds
2.15
3.32
5.73 17.60
2.00
3.15
5,41 16.33
8-12
1.86
3.04
5.21 / 16.27
1.86
3.04
5,21 16.27
Color perturbation
2.07
3.20
4,87 16.09
On
1.86
3.04
5,21 16.27
Motion blur
Off
1.95
3.57
5.06 16.24
On
1.86
3.04
5.21 / 16.27
Fog
Off
1.98
3.20
5.04 15.68
1.86
5,21 16.27
Texture noise
Off
2.02
3.15
5.47 / 16.56
On
1.86
3.04
5,21 / 16.27
Noise regularization
Off
1.91
3.09
4.97 15.87
1.86
3.04
5.21 16.27
Grid regularization
1.88
3.10
5.09 16.35
Sintel
1.86
3.04
5.21 / 16.27
Target dataset
KITTI 2015
2.14
3.79
4.31 14.29
Fine-tuning Results on Public Benchmarks. Table 4 shows the fine-tuning results on the public benchmark test sets with corresponding dataset schedules. DFlow improves the original RAFT recipe (C-:T-TSKH/S on both benchmarks by replacing the conventional initial dataset schedule, i.e., ChyingChairs followed by Flying Things3D (C+T), with our DFlow. particular, compared to competing datasets, DFlow achieves the best fine-tuning performance the Sintel with RAFT. This result validates that DFlow as a pre-training dataset effectively affects fine-tuning results.
Analysis of Components. We analyze the effects of each generation pipeline component: {a} combination, the number of foregrounds, color perturbation, real-world effects, regularizations, and target dataset. For the fair and quick experiments, we generate 2k training data and train RAFT networks with the same training details. As shown in Table 5, we add or remove each of the components from a default setting to measure their effects with the pre-training experiment. The default setting indicates the dataset with all components and the number of foregrounds parameter in the range to 12 except the {a} combination. We observe that applying each component brings consistent performance improvement on the target dataset, i.e., Sintel, which is used for generating
8

---

Published as a conference paper at ICLR 2023
DFlow; whereas the results on KITTI 2015, which is not target data of DFlow, are inconsistent with the results of Sintel.
Data update. This analysis compares DFlow with randomly initialized data without any optimiza- tion, ie., randomly generated by our pipeline. We train and compare the RAFT networks with those datasets, which show the effectiveness gap between the two datasets.
{a} Combination. We collect the same amount of data by combining subsets of data, which are obtained from diverse a values. This balances the performance between evaluation datasets because of the regularization effect.
Layer composition. The softmax splatting and alpha blending show comparable performance on Sintel. The softmax splatting improves the performance the Sintel clean pass, but shows the performance drop in the Sintel final pass.
Number of foregrounds. Without any foreground, we observe the sigmificant performance drop on Sintel and KITTI 2015. Adding only 2 foregrounds brings notable performance improvement.
Color perturbation and real-world effects. Without the color perturbation, the performance on Sintel drops moderately. Removing the motion blur significantly affects the accuracy on Sintel, especially in the Sintel final pass. The effects of fog and texture noise are moderate.
Regularizations. Removing the regularizations shows a slight performance drop in Sintel.
Target dataset. The generated dataset optimized to KITTI 2015 shows notable performance gain on KITTI 2015, while the performance on Sintel significantly drops. This may be caused by the distribution gap between the two datasets.
Robustness to Corrupted Data (Whether Arti- fact). The photometric inconsistency degrades the robustness optical flow estimation. For example, the rainy scene occurs photometric inconsistency. We evaluate the performance of RAFT on rainy scenes of Virtual KITTI (Gaidon al., 2016). We choose FlyingChairs and RealFlow (Han et al., 2022) as base- lines. Using the rainy scenes of Virtual KITTI a target dataset, we generate an additional dataset, DFlow-V. Note that the target of DFlow the Sin- tel dataset. Table 6 lists the performance of rainy scenes of virtual KITTI. As Han al. (2022) have mentioned their limitation of discontinuous illumi- nation, the performance RealFlow lower than others. DFlow outperforms the baselines, and DFlow- V, which of the target Virtual KITTI, shows much higher performance rather than others. From these results, assume that method can distill the characteristics of photometric inconsistency.
Rainy scene AEPE/
Dataset
FhyingChairs
7.22 / 24.95
RealFlow
7.75 27.57
DFlow
5.87 / 22.89
DFlow-V
3.11 / 12.61
"Bold note the best.
Table 6: Analysis of photometric inconsis- tency scenario.
5 CONCLUSION AND DISCUSSION
We propose a new data generation pipeline for training optical flow networks. Our pipeline consists of geometric warping, real- world effects, etc., which are all parameterized differentiably. We propose new objective function that drives our data optimization by leveraging the compressed knowledge of the proxy networks pre-trained on target and base datasets, respectively. Optical flow models trained on our datasets achieve favorable or superior performance against the competing datasets on pre-tre trainiiing and fine-tuning experiments. We conclude our paper with a discussion section.
Discussion. We use the pre-defined elementary data generation operations, e.g., fog, geometric warping, etc. While DFlow shows effectiveness in the real-world dataset, iee., KITTI 2015, the pre-defined and restricted operations might not span all the real-world effects. Thus, diverse and complementary operations would further improve expressiveness and may lead to additional perfor- mance improvement. Our method also aims at a specific target dataset. For that, we need a target network trained the target dataset, which requires least some amount of optical flow annotations. To mitigate the requirement of the supervised data, it would be an interesting future direction to investigate the way to train the target network in unsupervised method.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
This paper develops an optical flow data generation method and uses the public datasets (Kuznetsova al., 2020; Perazzi al., 2016; Mayer al., 2016; Lin et al., 2014; Everingham al., 2010) for data generation. Therefore, the author does not expect any potential ethical issues related to sensitive information of the dataset.
REPRODUCIBILITY STATEMENT
We introduce the whole data generation pipeline in Sec. In Sec. 3.1, we present the loss function for data optimization. In Sec. 3.2, we describe the data generator method from the parameters. We list more details of the loss function and data generation pipeline in Appendix A.
ACKNOWLEDGEMEET
This work was partially supported by the National Research Foundation of Korea (NRF) grant (No. NRF-2021R CIC1006799), and Institute of Information communications Technology Planning Evaluation (IITP) grant NN..2022-0-011 Self-Improving Competency-Awant Learning Capabilities; No.2021 No.2021-0-02066 -0-02068, Artificial Intelligence Innovation Hub) funded by the Korea government (MSIT). This project is the result of "HPC Support" Project supported by the "Ministry of Science and ICT" and NIPA.
Development of Artificial Intelligence Technology for
REFERENCES
Filippo Aleotti, Matteo Poggi, and Stefano Mattoccia. Learning optical flow from still images. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Christian Bailer, Kiran Varanasi, and Didier Stricker. Cnn-based patch matching for optical flow with thresholded hinge embedding loss. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Shai Ben-David John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79(1):151-175, 2010.
Michael Black and Padmanabhan Anandan. A framework for the robust estimation of optical flow. In IEEE International Conference on Computer Vision (ICCV), 1993.
D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black. A naturalistic open source movie for optical flow evaluation. In European Conference on Computer Vision (ECCV), pp. 611-625, 2012.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International joteraaaionaouuualll of computer vision, 88(2): 303-338, 2010.
Philipp Fischer, Alexey Dosovitskiy, Eddy Ilg, Philip Häusser, Caner Hazirbas, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks, In JEEE International Conference on Computer Vision (ICCV) 2015.
A Gaidon, Q Wang, Y Cabon, and E Vig. Virtual worlds as proxy for multi-object tracking analysis. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
James Gibson. The perception of the visual world. 1950.
James Jerome Gibson and Leonard Carmichael. The senses considered as perceptual systems. Houghton Mifflin Boston, 1966.
10

---

Published as a conference paper at ICLR 2023
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. volume 13, pp. 723-773, 2012
Yunhui Han, Kunming Luo, Ao Luo, Jiangyu Liu, Haoqiang Fan, Guiming Luo, and Shuaicheng Liu. Realflow: Em-based realistic optical flow dataset generation from videos. In European Conference on Computer Vision, pp. 288-305. Springer, 2022.
Berthold KP Horn and Brian G Schunck. Determining optical flow. Artificial intelligence, 17(1-3):
185-203, 1981.
Tak-Wai Hui, Xiaoou Tang, and Chen Change Loy. Liteflownet: A lightweight convolutional neural network for optical flow estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018
Eddy 1lg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2017.
Shihao Jiang, Dylan Campbell, Yao Lu, Hongdong Li, and Richard Hartley. Learning to estimate hidden motions with global motion aggregation. In IEEE International Conference on Computer Vision (ICCV). pp. 9772-9781, 2021.
Alexandre Kaspar, Tae-Hyun Oh, Liane Makatura, Petr Kellnhofer, and Wojeiech Matusik. Neural inverse knitting: From images manufacturing instructions. In International Conference Machine Learning (ICML), 2019.
Daniel Kondermann, Rahul Nair, Stephan Meister, Wolfgang Mischler, Burkhard Güssefeld, Sabine Hofmann, Claus Brenner, and Bernd Jähne. Stereo ground truth with error bars. In Asia Conference on Computer Vision (ACCV), 2014.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper R. R. Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Tom Duerig, and Vittorio Ferrari. The open images dataset V4: unified image classification, object detection, and visual relationship detection at scale. In International Journal of Computer Vision (IJC)) 2020.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV), 2014.
Zhaoyang Lv, Kihwan Kim, Alejandro Troccoli, Deqing Sun, James Rehg, and Jan Kautz. Learning rigidity in dynamic scenes with a moving camera for 3d motion field estimation. In European Conference on Computer Vision (ECCV), pp. 468-486466 484,484444 2018.
Nikolaus Mayer, Eddy 1lg, Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Nikolaus Mayer, Eddy Ilg, Philipp Fischer, Caner Hazirbas, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. What makes good synthetic training data for learning disparity and optical flow estimation? International Journal of Computer Vision, 126(9):942-90) 960, 2018.
Moritz Menze and Andreas Geiger. Object scene flow for autonomous vehicles. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3061-3070, 2015.
Simon Niklaus and Feng Liu. Sofumax splatting for video frame interpolation. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 5437-5446, 2020.
Tae-Hyun Oh, Ronnachai Jaroensri, Changil Kim, Mohamed Elgharib, Fr'edo Durand, William T Freeman, and Wojciech Matusik. Learning-baser video motion magnification. In European Conference Computer Vision (ECCV), pp. 633-648, 2018.
Federico Perazzi, Jordi Font-Tuset, Brian Mc MeWiliiams, Lue Van Gool, Markus Gross, and Alexander Sorkine- Hornung. A benchmark dataset and evaluation methodology for video object segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.

---

Published as a conference paper at ICLR 2023
Anurag Ranjan and Michael J Black. Optical flow estimation using spatial pyramid network. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Stephan R Richter, Zeeshan Hayder, and Vladlen Koltun. Playing for benchmarks. In IEEE International Conference Computer Vision (ICCV), pp. 2213-2222 2017.
Daniel Scharstein and Chris Pal. Learning conditional random fields for stereo. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2007.
Daniel Scharstein and Richard Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International journal of computer vision, 2002.
Leon Sixt, Benjamin Wild, and Tim Landgraf. RenderGAN: Generating realistic labeled data.
Frontiers in Robotics and AI, 5:66, 2018.
Deqing Sun, Xiaodong Yang, Ming- Yu Liu, and Jan Kautz. Pwe-net: Cnns for optical flow using pyramid, warping, and cost volume. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Deging Sun, Daniel Vlasic, Charles Herrmann, Varun Jampani, Michael Krainin, Huiwen Chang, Ramin Zabih, William T Freeman, and Ce Liu. Autoflow: Learning better training set for optical flow. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In European Conference on Computer Vision (ECCV), 2020.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, (866:2557-22605 2008.
Jonas Wulff, Laura Sevilla-Lara, and Michael Black. Optical flow in mostly rigid scenes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Jia Xu, René Ranftl, and Vladlen Koltun. Accurate optical flow via direct cost volume processing, In
IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 5807-5815, 2017.
Dawei Yang and Jia Deng. Learning to generate training data through hybrid gradient. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 779-789, 2020.
Christopher Zach, Thomas Pock, and Horst Bischof. A duality based approach for realtime tv-1 1 optical flow. In Joint pattern recognition symposium, 2007.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with distribution matching. International Conference Learning Representations (ICLR), 2021.
Helong Zhou, Liangchen Song. Jiajie Chen, Ye Zhou, Guoli Wang, Junsong Yuan, and Qian Zhang Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective. In Interna- tional Conference on Learning Representations (ICLR), 2021.
12

---

Published as a conference paper at ICLR 2023
APPENDIX
We present implementation details and additional experimental results. The contents are listed as follows:
CONTENTS
Details Differentiable Data Generation Pipeline
A.1 Base and Target Networks
A.2 The Form of Task Loss Functions
A.3 Geometric Warping and Flow Field Translation
A.4 Layer Composition
A.5 Color Perturbation and Real-world Effects
A.6 Regularization Loss
A.7 Details of Learnable Parameter 0
Details Experiment Setup
B.1 Generation Details of the DFlow Dataset
B.2 Details of Pre-training Results
B.3 Details of Model Generalization Property Results
B.4 Details of Fine-tuning Results
B.5 Details of Robustness against Corrupted Data
Additional Experiments
C.I Contrasting Effect of Base and Target Networks
C.2 Validation Results
C.3 Dataset Size
C.4 Pre-training Performance according to Motion Magnitudes
C.5 Analysis of Multi-target Datasets
C.6 Additional Qualitative Results
Additional Generated Data Samples
13

---

Published as a conference paper at ICLR 2023
A DETAILS OF DIFFERENTIABLE DATA GENERATION PIPELINE
In this section, we describe the implementation details of the differentiable data generation pipeline.
A. BASE AND TARGET NETWORKS
Base Network. For the architecture, we use the same architecture for both base and target networks. We use the published weight of RAFT (Teed & Deng, 2020) base network", which is pre-trained on FlyingChairs (Fischer et al., 2015) for 100k iterations with a batch size of 12, 496 x 368 image size, and learning rate 4 x 10-4.
Target Network. We fine-tune the base network on the Sintel datasets (Butler et al., 2012) for the target network. During fine-tuning, we only use the image crop as data augmentation and fine-tune the base network for 20k iterations with batch size of 6, 768 x 368 image size, and learning rate 1.25 x 10
A.2 THE FORM OF TASK LOSS FUNCTIONS
We introduce our task loss function Lrask Crask(Curgot, Lasse) for contrastive-style learning and evaluate the candidate of the loss function in Sec. 3.1. Table shows the used task loss functions in experiments of Table 2. We the sequence loss of flow estimates of RAFT (Teed & Deng, 2020) for [targett Algorithm 2 is the pseudo-code for updating the parameters of each data sample.
Table 7: The forms of task loss function. We list the used forms of task loss function Sec. 3.1. The form is the combination of [Multiplication, Addition} and (Exponential, Sigmoid, Tanh}. The task loss functions have the hyperparameters, a, B. and 7.
Task loss function Cusk(Luurgee Lruse)
Multiplication & Exponential
(1 a exp( Crase/ (Erange + E) + 7) Lrarget
Multiplication & Sigmoid
(1 asi aesigmiidl (ooood((Caa) Lurge/ (Louse + c) +u)
Multiplication & Tanh
(1 +
(froee + c) + 7) Liaggt
Addition & Exponential
Lurget + a exp(- BL (Lurget + c) 7)
Addition & Sigmoid
Lunget + ors ossigmo id (Layua (Lrave + e) +
Addition & Tanh
Lunget + ortanh(3 Gaaage (Lhuse E) +7)
A.3 GEOMETRIC WARPING AND FLOW FIELD TRANSLATION.
Given N pairs of layer images and masks, (L)) and (M) NN we generate subsequent frames, I, and I1 with ground-truth optical flow FCT.
(5) We apply two steps of geometric warping with warping fields Wo-++1 and W 1-41-1 which computed from combinations of rigid transformation, perspective warping, and grid warping, First, we warp the layer images and masks, {L and [M] to be the layer images 1141)/ and masks { IN at the t + frame by applying the same warping field Wo-r+1 to [L (M))
[L (M) {L, I,1,
1N
1 and
(6) where o denotes the geometric warping operation according to a given warping field. In the next step, each layer image L' and mask M; at the t frame generated from the ones at the + 1 frame, i.e., L and M1+1. To simulate complex optical flows with minimal parameters, we introduce the flow field translation {D') N Jim which translates the warping field W Wi+1-r and obtains each warping of layers: W (+ (D' W++1-))) Using each warping of layers W! (+1-11 we warp the layer
L Wo-r-1o L. M HLi:
Wo-+41 o M
images and masks at the f + time.
L = ( W441-)) L M = (D's W.41-1) M
14

---

Published as a conference paper at ICLR 2023
Algorithm 2: Pylorch-style pseudo-code for DFlow.
G: data generator
B: base network
T: target network
L: our task loss function
Lt: target loss function
Lb: base loss function
A: data parameters
for 1, in loader:
Generate optical flow data
1: layer images, layer masks
imagel, image2, label = (11 m, 0)
loss
Lt (B (imagel, image2), , label)
lb = Lb (T (imagel, image2), label)
loss L(lt, 1b)
Update
loss backward ()
optimizer step ()
The above parameterization is efficient for approximating multi-layer geometric warpings because it only introduces two additional parameters per layer to generate the local movement of each layer from single warping field W --t- As aforementioned, we propose to use the two steps of warping with Wo+t++1 and W++1++1 This is distinctive from the existing works, including Fischer et al. (2015); Sun et al. (2021), where they only use single step of warping from randomly generated image to a synthetic next frame. This only parameterizes the next frames, not the reference frames randomly generated. This could not update the textures and mask shapes of the reference frames, whereas our method parameterizes both frames with two warping fields, Wo-+41 and W++1-1- It allows us jointly update subsequent frames.
A.4 LAYER COMPOSITION
From the geometric warping, we have N pairs of layer images, masks, and warping fields in order of depth. We adopt the softmax splatting (Niklaus & Liu, 2020) compose the layer images subsequent frames and warping fields with flow field translation, i.e., L; L and W! get the importance image Z and weight image K of each layer and frame,
1+1-- We
z Maa' max; (M,)
(8)
exp( Z2,)
Ki
Σ exp( Z2)
where a' cl and is constant value which set 6 for layer composition. As the value of increases, sharper boundaries can be obtained. The subsequent frames and ground-truth optical flow follows:
N I
Σ Kinn C L
Σ
K
L
Wo W -
(L41, Fer), (10)
1=1
A.5 COLOR PERTURBATION AND REAL-WORKH EFFECTS
Color Perturbation. Color perturbation consists of N white balances adjust the color intensity of layer images. Each white balance has three-channel values from zero to one and has half of the chance be applied. Color perturbation is the same for subsequent frames and optimized.
15

---

Published as a conference paper at ICLR 2023
(a)
(b)
(c)
(d)
Figure 5: Color perturbation and real-world effects. Real-world effects are composed of texture noises, fog, and motion blur. From left to right: (a) color perturbation, (b) texture noises, (c) fog, (d) motion blur. (a) an overlaid image before (upper right) and after color perturbation (lower left).
Texture Noises. We simulate the shot noise to mimic real-world photos. We apply three-channel texture noises to all of the pixels while masking the area where the noises actually applied. Two different texture noises, N, and N are applied to subsequent frames and optimized.
Fog and Motion Blur. Inspired by AutoFlow (Sun al., 2021), we introduce two real-world effects; fog and motion blur. To generate random fog, we follow the implementation of AutoFlow and superimpose the fog on I, and 1++1. However, one difference that we adjust the three-channel values of fog, which yields the colored fog. The fog is the same for the subsequent frames and is optimized as well. For the motion blur, we randomly sample object masks from PASCAL VOC (Everingham al., 2010) and combine them generate motion blur mask. We the 2D gaussian blur to approximate the motion blur kernel, and the motion blurred frames are alpha-blended blended to I, and L The standard deviation of each axis and angle of rotation are parameters be updated.
Figure 5 shows the color perturbation and real-world effects including texture noise, fog, and motion blur.
A.6 REGULARIZATION Loss
The grid warping and texture noise may take shortcuts only to minimize task loss. These behaviors are undesirable, and the generated data might not properly train the optical flow network. To handle this potential issue, we propose a regularization term Leg consisting of grid and noise regularization losses, i.e., Lreg Cgrid + Cnoise- The grid regularization Egrid prevents from producing infeasible 2D motion. We define the grid regularization loss Lgri as follows:
or h-1 EIC C. + 1)]+ k=1j=1
W-11
Carid max
EI k=
C.(k+1,1 1,5)), (1I)
where C, is the warped coordinate by the geometric warping W Legrd gives penalty when the warped coordinates of the previous grid exceed those of the next grid. The noise regularization Croise prevents from producing too noisy images. We define the noise regularization loss Cnoise as follows:
Lusis = IIN
(12)
where N, and No+1 are texture noises of the current and next frames, respectively. Finally, we obtain our total loss by adding the regularization loss to the task loss as:
Listal
Lunk (Lurgets Lraee) freg.
(13)
With the total loss Liotal, we update the data parameters {0}.
A.7 DETAILS OF LEARNABLE PARAMETER 0
Learning Rate of O. We set the learning rate of real-world effects as 3 x 10-2, and the others as {1 x 10". 1 x 10-1, 1,2 x 10 10-2) depending on whether they are pixel-unit operations (e.g.. translation parameters) not. We decay the learning rates linearly over the update iterations; the decay factor is (1 - iteration /80), where 80 is the maximum iteration of updates. We distinguish the pixel-unit operations and the others
16

---

Published as a conference paper at ICLR 2023
Table 8: Generation details of DFlow. Data augmentation used in generating DFlow.
Color jitter
Random resize and crop
Data augmentation
brightness 0.1
contrast
saturation
huc
min scale
max scule
image crop
0.1
0.1
0.04
0.93
2.30
496 x 368
Pixel-unit operation (learning rate of 1 x 10% translation of the rigid transformation, grid warping, perspective warping, and flow field translation
Non-pixel-un operation (learning rates {1 x 10 1011100100000 1.2 10 ²):: rotation and scaling of the rigid transformation
Initialization of 0. Following the implementation of AutoFlow (Sun al., 2021), sample the initial data parameters 0 from the uniform distributions ranging from a to b, i.e, [a, bj.
White balance: [0, 1]
Rigid transformation (translation, rotation, scaling): ([-80, 80), [-5, 5], [0.75, 1.13))
Grid warping: [0, 0]
Perspective warping: [-25, 25]
Flow field translation: [-50, 50]
Texture noise: [-0.01, 0.01]
Fog color: [0, 1]
Motion blur (std of axis std of axis y, angle of rotation): ([1, [3, 11], 10, 90])
B DETAILS OF EXPERIMENT SETUP
In this section, present the generation details of the DFlow dataset. Then, present the details of pre-training and fine-tuning results in Sec. of the main paper.
B. GENERATION DETAILS OF THE DFLOW DATASET
To generate the DFlow dataset, the data augmentation is included in the proposed differentiable data generation pipeline. We randomly augment each data sample before inputting them into the base and target networks. For stability of data generation, we stack randomly augmented data samples and input them into networks. Each data sample of DFlow is updated up to 80 times and saved depending on the threshold 25 of the target network loss. The DFlow data has a size of 512 384, which the same as FlyingChairs. Table 8 summarizes the data augmentation applied to each data sample.
B.2 DETAILS OF PRE-TRAINING RESULTS
Except for the data augmentation, we set the training parameter settings by following the training details of RAFT (Teed & Deng, 2020) on FlyingChairs (Fischer et al., 2015), and pre-train the RAFT network on DFlow with a data size of 15k. We use the same data augmentation applied for generating the DFlow dataset. We early stop the training at 95k iteration.
B.3 DETAILS MODEL GENERALIZATION PROPERTY RESULTS
For the FlowNet results, we use the Pytorch implementation of FlowNet'. We pre-train the FlowNet model for 2700 epochs with batch size Following the Pytorch Implementation of FlowNet, we use the proposed data augmentation including translation, rotation, image crop, vertical flip, and horizontal flip. For the AutoFlow result, we use the published AutoFlow dataset (Sun al., 2021). 1) To generate the DFlow-GMA dataset, replace the RAFT proxy networks with GMA networks (Jiang et al., 2021) while selecting the KITTI 2015 dataset as the target dataset. The set a to 13 for the DFlow-GMA dataset and treit the GMA network DFlow-GM. with data size of 2k.
17

---

Published as a conference paper at ICLR 2023
Figure 6: Data samples of the rainy scene of Virtual KITTI.
Table 9: Contrasting analysis in pre-training. The dataset generated with the contrasting effect achieves comparable results at the target dataset, KITTI 2015. The contrasting effect leads to a notable performance improvement on Sintel, which used for data generation.
Evaluation dataset
Contrasting effect
Sintel clean
Sintel final
KITTI 2015
AEPE
AEPE
AEPE /I
X
4.38
5.03
4.61 13.86
V
2.14
3.79
4.31 / 14.29
"Bold denotes the hest.
B.4 DETAILS OF FINE-TUNING RESULTS
To achieve fine-tuning results on the public benchmarks, fine-tune the RAFT model pre-trained on DFlow. Except for the initial data schedule, i.e., FlyingChairs followed by FlyingTTingsst we follow the same dataset schedule and training details of the original implementation (Teed & Deng, 2020).
B.5 DETAILS OF ROBUSTNESS AGAINST CORRUPTED DATA
The photometric inconsistency is the main cause of the error in optical flow estimation. In real- worldworld images, real-world effects, such as blur, fog, and illumination change, cause photometric inconsistency, and these effects frequently occur. Since DFlow generates dataset using the compressed knowledge of the target dataset, verify DFlow's ability generate the dataset for training a robust optical flow network against photometric inconsistency. We use rainy scenes of Virtual KITTI (Gaidon et al., 2016) because the photometric inconsistency dominant, shown in Fig. To obtain the target network, we further train the base network on the rainy scenes for 20k iterations and generate an additional dataset, DFlow-V by targeting the rainy scenes. We use the published weight of RAFT trained on the RF-AB dataset for the result of RealFlow (Han et al., 2022).
C ADDITIONAL EXPERIMENTS
We analyze the contrasting effect of base and target networks in Sec. C.I, and evaluate the validation performance with DFlow in Sec. C.2.
C.1 CONTRASTING EFFECT OF BASE AND TARGET NETWORKS
To evaluate the contrasting effect of base and target networks, we measure the pre-training perfor- mance depending on the contrasting effect. The details will be presented the following paragraphs.
Contrasting Analysis Pre-training. For the contrasting analysis, we generate two types of dataset with sample size of 2k. For the first dataset, we use KITTI 2015 (Menze & Geiger, 2015) as the target dataset, i.e., the target network is obtained by fine-tuning the base network on the KITTI 2015 dataset. To synthesize the first dataset with contrasting effect, we minimize and maximizes the loss of the target and base networks, respectively. For the other dataset, we generate a new target network that pre- mr--triied the KITTI 2015 dataset. With the target network, generate new pre-tree are-training dataset that is synthesized to minimize the loss of the new target data network only, Le., the contrasting effect is not applied.
18

---

Published as a conference paper at ICLR 2023
contrasting) purs (wo, contrasting)
(initial)
base
Figure 7: t-SNE plot (van der Maaten & Hinton, 2008) of different datasets. We use the context encoder of the base network trained FlyingChairs feature extractor and extract the features of several datasets. (Red): target datasets, e.g., KITTI 2015, (Blue): base dataset, e.g. g., FlyingChairs (Sky): our initial dataset before optimization, (green): updated our dataset without the contrasting effect, (Yellow): updated our dataset with the contrasting effect. The features of our dataset with the contrasting effect are getting closer the features of the target dataset through the optimization process than the features of other datasets.
Table 9 shows the pre-training results depending on the contrasting effect. With the contrasting effect, the RAFT network achieves comparable results on the target dataset, i.e., KITTI 2015, compared to the other network trained on the dataset generated with the new target network only. However, without the contrasting effect, the network shows notable performance degradation on Sintel (Butler et al., 2012), which is not used for data generation.
We also visualize data features to identify the effect of the contrasting effeect. We embed the features of several datasets, such as FlyingChairs, KITTI 2015, our datasets with and without the contrasting effect, and our dataset before optimization. We use the context encoder of the base network feature extractor. As shown in Fig. 7, ours with the contrasting effect is closer to the target than one without the contrasting effect. We additionally com- pute distances of feature mean between ours and KITTI 2015. As shown in Table 10. the maximum mean discrepancy (MMD) (Gretton al.. 2012) of ours with the contrasting effect is lower than ours without the contrasting effect. These results show that the contrasting effect drives the data generation toward improving direction target domain.
Table 10: Distance of feature mean be- tween ours and KITTI 2015.
Datasets
MMD
Ours wo. contrasting
6.6892
Ours w. contrasting
6.1454
C.2 VALIDATION RESULTS
The advantage of the proposed pipeline is that the generated data can be used to improve the validation performance on the unseen target dataset. We split the KITTI 2015 dataset into 50 training samples and 150 validation samples; KITTI-50 and KITTI-150. Using the target network fine-tuned on KITTI-50, we synthesize dataset; OursKITH so- Therefore, any data information on KITTI-150 not included in the data generation process. We fine-tune the base network on several combinations of datasets with 20k iterations and evaluate the validation performance on KITTI-150, As shown in the combination of Sintel and KITTI-50 shows comparable performance
Table I1, fine-tuning with the one of fine-tuning on KITTI-50 alone. In contrast, we observe notable improvement in validation performance when adding the same number of Ourskirn-s0 the training dataset. As data generation pipeline is not limited to the number samples that can generate, we analyze the effect of our data size and observe the improvement of validation accuracy.
19

---

Published as a conference paper at ICLR 2023
Table 11: Comparison of the validation performance networks fine-tuned on different dataset combinations. We fine-tune the base network on the combination of KITTI-50 and another dataset. Ourskrrn-so notably improves the validation accuracy on the unseen KITTI dataset, KITTI-150.
Validation dataset
Fine-tune dataset
Data size
KITTI-150
AEPE / Fl
KITTI-50
3,41 11.47
KITTI-50+Sine
50+2082
3.39 11.21
KITTI-50+OuSSann
50+2082
3.07 10.99
KITTII-0++OrS<TTTT.S
50+12003
2.98 10.70
"Bold denotes the best, and underline denotes the second best.
Table 12: Pre-training results depending on the amounts of DFlow data. We train the RAFT networks FlyingChairs (Fischer et al.. 2015) and various amounts DFlow dataset. We evaluate the performance on the Sintel and KITTI 2015 datasets.
Evaluation dataset
Dataset
Data size
Sintel clean
Sintel final
KITTI 2015
AΣPE
AEPE
AEPE / FI
FlyingChairs
22873
2.28
4.51
9.85 / 37.56
DFlow
100
2.72
4.03
9.15 / 23.09
DFlow
500
2.20
3.48
5,40 17.54
DFlow
1000
2.02
3.24
5.22 17.06
DFlow
2000
1.90
3.02
4.90 15.78
DHlow
4000
1.87
2.92
4.78 / 15.64
DFlow
8000
1.81
2.91
4.88 / 15.51
DFlow
15000
1.81
2.93
4.59 / 15.03
C.3 DATASET SIZE
The number of the dataset is key factor for training accurate optical flow networks, and synthetic optical flow data have an advantage because of the annotation efficiency compared to the world data. We analyze the effect of data size in the optical flow network, RAFT (Teed & Deng, 2020) We train the RAFT network on the various amount of DFlow data and evaluate the performance on both Sintel and KITTI 2015 datasets. We determine FlyingChairs (Fischer et al., 2015) competing dataset because FlyingChairs and DFlow have the same data resolution. As shown in Table 12, the RAFT network trained on the 500 DFlow dataset outperforms the model trained on the full FlyingChairs dataset. This result shows that DFlow data has more information with the same data resolution. The overall performance on Sintel and KITTI 2015 datasets is also improved as the data size increase, which shows that diverse texture and motion is effective for training the optical flow network.
C.4 PRE-TRAINING PERFORMANCE ACCORDING TO MOTION MAGNITUDES
Table 13 summarizes ABPES of RAFT pre-trained on respective FlyingChairs (Fischer et al., 2015) and our DFlow according different motion ranges. The model trained DFlow shows higher accuracy than that of FlyingChairs except for the minimal motion range. Performance gaps between the models tend to be enlarged in larger motion. These results may be explained by human intuition that the small motion hardly contributes to the error of optical flow networks. For the same reason, most motion magnitudes of DFlow are in the middle and high motion ranges, as shown in Fig. 8.
20

---

Published as a conference paper at ICLR 2023
Table 13: Motion accuracy of the RAFT networks in different magnitude ranges. We evaluate the AEPE score of models pre-trained on FlyingChairs and DFlow in different motion ranges. As the motion magnitude increases, the model trained on DFlow achieves more considerable performance improvements than that trained on FlyingChairs.
Pre-training dataset
Evalutaion dataset
Motion magnitude ranges
1
[1. 10
(10, 20]
(20, 30]
30
FlyingChairs DFlow (Ours)
0.57
1.15
3.31
6,47
24.08
Sintel Final
0.69
0.80
3,59
15.18
FlyingChairs DFlow (Ours)
0.74
1.67
2.65
3.86
21.32
KITTI 2015
1.16
0.81
1.31
2.10
9.72
*Bold denotes the best.
100.0%
Percattageof Pixdis
PhinggChiis Sirtel DHow
Tingse KiTli 2015
10.0%
1.0%
0.1%
0.0%
50
100
Motion Magnitude (pixels)
Figure 8: Motion magnitude histograms. DFlow focuses more the mid-high ranges of motion than the other datasets, because small motion hardly contributes to errors of networks.
Table 14: Analysis multi-target datasets. The target of DFlow is Sintel, DFlow-V rainy scenes of Virtual KITTI. We denote DFlow+DFlow-V the multi-target datasets. Multi-target datasets show the trade-off in general but achieve the highest performance on KITTI 2015.
Evaluation dataset
Dataset
Sintel clean
Sintel final
KITTI 2015
Rainy scene
AEPE
AEPE
AEPE FI
AEPE /
DFlow
1.81
2.93
4.59 / 15.03
5.87 / 22.89
DFlow-V
2.57
3.86
5.98 15.16
3.11 / 12.61
DFlow DFlow-V
1.99
3.14
4.39
14.84
4.30/ / 16.91
*Bold and underline the best and second best, respectively.
C.5 ANALYSIS OF MULTI-TARGET DATASETS
We focus on generating the dataset close the target dataset. questionable whether talti-target datasets can be used to generate the DFlow dataset consisting of diverse data samples. The targets of DFlow and DFlow-V are Sintel and rainy scenes of Virtual KITTI. Table 14 lists the optical flow performance on Sintel clean, Sintel final, KITTI 2015, and the rainy scene of Virtual KITTI. As expected, DFlow-V achieves higher performance than DFlow on the rainy scene; but DFlow-V has inferior performance there is a trade-off: the performance DFlow with DFlow- is usually between the performance of DFlow and DFlow-V, However, the mixture of DFlow and DFlow-V V outperforms in KITTI 2015. It indicates that exploiting multi-target datasets is a promising research direction.
the other datasets. When combining DFlow and DFlow-V, we observe that
21

---

Published as a conference paper at ICLR 2023
C.6
ADDITIONAL QUALITATIVE RESULTS
Ground Inuth
HyingCheins
DFoww
Ground truth
FlyingChairs
DFlow
Input Image
Ground truth
FlyingChairs
DFlow
Figure 9: Additional qualitative results. Top: pre-training results on the KITTI 2015 dataset. Bottom: pre-training results on Sintel Final pass.
22

---

Published as a conference paper at ICLR 2023
D ADDITIONAL GENERATED DATA SAMPLES
Figure 10 shows additional data samples. Color perturbation, texture noise, fog, and motion blur can be found in the data samples.
Figure 10: Additional generated data samples. Top: current frames; Middle: next frames; Bottom: visualizations of ground truth optical flow.
23


Reviews:
[{"review_id": "tNsE7UFSv_", "reviewer": "Reviewer_uY3h", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a differentiable optical flow data generation pipeline and a loss function to drive the pipeline. The proposed modules enable automatic and efficient synthesis of a dataset effectively to a target domain, given a snippet of target data. This distinctiveness is achieved by proposing an efficient data comparison method. Experiments show the competitive performance of DFlow against the prior arts in pre-training.", "summary_of_the_review": "The paper proposed a differentiable optical flow data generation pipeline (DFlow) and a loss function to drive the pipeline. This is a simple and efficient differentiable data generation pipeline for optical flow. It also proposed a contrastive-style learning scheme and its loss function by approximating expensive dataset-todataset comparison to leverage proxy neural networks.", "strength_and_weaknesses": "Strength:\n\n+ A simple and efficient differentiable data generation pipeline for optical flow.\n\n+ A contrastive-style learning scheme and its loss function by approximating expensive dataset-todataset comparison to leverage proxy neural networks.\n\nWeaknesses:\n\n- The claim that \"the RAFT model pre-trained with DFlow achieves state-of-the-art performance on the Sintel public benchmark in fine-tuning\" is questionable. According to the benchmark in Sintel. The performance of \"DF-RAFT\" is even not as good as the orignal RAFT.\n\n- The paper did not discuss the limitations of the proposed approach.\n\n- There are some recent work generating optical flow dataset in a rather cheap way such as :\nRealFlow: EM-Based Realistic Optical Flow Dataset Generation from Videos, ECCV 2022.\nYunhui Han, Kunming Luo, Ao Luo, Jiangyu Liu, Haoqiang Fan, Guiming Luo, Shuaicheng Liu", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Vb9EMOiJ5z", "reviewer": "Reviewer_dkLC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a differentiable data generation pipeline for optical flow which depends on a target optical flow dataset, a common base dataset and prior data generation constraints on grid warping and texture noise. The data generation process is much more efficient than existing work of AutoFlow and the method achieves state-of-the-art performance on public Sintel benchmark.", "summary_of_the_review": "The idea of differentiable data generation is interesting but the experiment result is not impressive and the reproducibility is not good.", "strength_and_weaknesses": "Strength:\n1. The fully differentiable data generation process is interesting and efficient which includes various components including color perturbation, geometric warping, flow field translation, layer composition and real-world effects. \n2. The experiment on the analysis of each data generation component is clear and convincing.\n\nWeakness:\n1. The data generation process of DFlow needs a target dataset and a common base dataset on which the proxy network models are trained in fully supervised way, this makes the generated dataset somewhat task-dependent. \n2. The declaration of SOTA performance on the Sintel public benchmark is not correct: recently there are lots of published optical flow algorithms achieved better performance than RAFT, please ref to: http://sintel.is.tue.mpg.de/results for more details. Table 5 shows comparison with RAFT but does not show SOTA performance, is it possible to apply DFlow to more recent deep network structures?\n3. It is not clear whether the dataset size of DFlow affects the optical flow estimation result and whether the data generation process can use multiple target datasets for better generalization on pre-train or better fine-tune performance.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NIW0kgKe5Py", "reviewer": "Reviewer_iPLs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new data generation pipeline for training optical flow networks. The pipeline consists of parameterized differentiably geometric warping, flow field translation, color perturbation, and real-world effects. It also proposes a new objective function that drives the data optimization by leveraging the compressed knowledge of the proxy networks pre-trained on target and base datasets, respectively. Optical flow models trained on the datasets achieve favorable or superior performance against the competing datasets on pre-training and fine-tuning experiments.  ", "summary_of_the_review": "My overall rating for this work is positive. The solid and comprehensive experimental results show the method and the generated datasets would benefits the community.  ", "strength_and_weaknesses": "Strength: the problem this work focuses on is meaningful and useful. The authors do propose a solid and effective solution for optical flow data generation. \n\nWeakness: Limiation and future work should be discussed.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "HdK4-ELBBtm", "reviewer": "Reviewer_tPfm", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a differentiable pipeline for learning an optical flow dataset. Given a pre-defined set of dataset generation parameters and network, the method optimizes the variable using an optical flow loss in an end-to-end manner. Comparing to the previous work, the method achieves competitive accuracy but mostly better accuracy. Especially in terms of computational cost (e.g., training time), the method is an order of magnitude more efficient than the previous work. ", "summary_of_the_review": "The paper shows empirically good results, great efficiency, and it provides detailed ablation study. However, it was difficult for me to understand some technical parts (see the clarity section above). I would like to raise my rating after clearing up those concerns through a discussion phase. \n\n----\n\n__(Post discussion)__\n\nI would like to raise my rating to '8: accept, good paper'. The authors' response resolved my main concern about the clarity of the paper. I also read other reviews and responses on them, and they seem fairly well addressed. The paper mainly demonstrates empirically good accuracy over the direct competitor (AutoFlow) with better training efficiency. ", "strength_and_weaknesses": "__Strength__\n\n- Computational efficiency\n  Comparing with AutoFlow, the method is substantially more efficient (ref. Table 1). It enables the algorithm to run on a small-scale experiment setup. \n\n- Detail ablation study\n  The paper provides clear & detail ablation study for each design choice. It makes paper more transparent.\n\n---\n\n__Weakness__\n\n- I don't see any significant weakness of the paper so far. Rather there are some details that are better to be clarified. (continued in the following section).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster