CLIP-FLOW: CONTRASTIVE LEARNING WITH ITERATIVE PSEUDO LABELING FOR OPTICAL FLOW
Authors: Zhiqi Zhang
Nitin Bansal
Changjiang Cai
Pan Ji
Qingan Yan
Xiangyu Xu
Yi Xu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Optical Flow
Contrastvie Learning
Semi-supervised Learning

Abstract: Synthetic datasets are often used to pretrain end-to-end optical flow networks, due to the lack of a large amount of labeled, real scene data. But major drops in accuracy occur when moving from synthetic to real scenes. How do we better transfer the knowledge learned from synthetic to real domains? To this end, we propose CLIP-Flow, a semi-supervised iterative pseudo labeling framework to transfer the pretraining knowledge to the target real domain. We leverage large-scale, unlabeled real data to facilitate transfer learning with the supervision of iteratively updated pseudo ground truth labels, bridging the domain gap between the synthetic and the real. In addition, we propose a contrastive flow loss on reference features and the warped features by pseudo ground truth flows, to further boost the accurate matching and dampen the mismatching due to motion, occlusion, or noisy pseudo labels. We adopt RAFT as the backbone and obtain an F1-all error of 4.11%, i.e., a 19% error reduction from RAFT (5.10%) and ranking 2nd place at submission on KITTI 2015 benchmark. Our framework can also be extended to other models, e.g., CRAFT, reducing the F1-all error from 4.79% to 4.66% on KITTI 2015 benchmark.  

Under review as a conference paper at ICLR 2023
CLIP-FLOW: CONTRASTIVE LEARNING WITH ITERA- TIVE PSEUDO LABELING FOR OPTICAL FLOW
Anonymous authors Paper under double-blind review
ABSTRACT
Synthetic datasets are often used to pretrain end-to-end optical flow networks, due to the lack of a large amount of labeled real scene data. But major drops in accu- racy occur when moving from synthetic to real scenes. How do we better transfer the knowledge learned from synthetic to real domains? To this end, we propose CLIP-Flow, a semi-supert supi-supereised iterative pseudo labeling framework to transfer the pretraining knowledge to the target real domain. We leverage large-scale, unla- beled real data to facilitate transfer learning with the supervision of iteratively updated pscudo ground truth labels, bridging the domain gap between the syn- thetic and the real. In addition, we propose a contrastive flow loss on reference features and the warped features by pseudo ground truth flows, to further boost the accurate matching and dampen the mismatching due to motion, occlusion, or noisy pseudo labels. We adopt RAFT as backbone and obtain an FI-all error of 4.11%, %,, i.e. a 19% error reduction from RAFT (5.10%) and ranking 2nd place at submission on KITTI 2015 benchmark. Our framework can also be extended other models, e.g. CRAFT, reducing the Fl-all eπror from 4.79% to 4.66% on KITTI 2015 benchmark.
INTRODUCTION
Optical flow is critical in many high level vision problems, such as action recognition (Simonyan &
Zisserman 2014, (Sevilla--aar Lariilll--araaaaaaaaaaaaaaaaaaaa et al. 2018 Sun et al. 2018b), video segmentation (Yang et al. 2021
Yang& & Ramanan (221) and editing (Bonneel al. 2015). autonomous driving (Janai et al. 2020
and so on. Traditional methods (Horn & Schunck| 1981 Menze et al.) 2015 Ranftl et al. 2014
Zach et al.) (007)) mainly focus on formulating flow estimation as solving optimization problems us- ing hand-crafted features. The optimization is searched over the space of dense displacement fields between a pair of input images, which is often time-consuming. Recently, data driven deep learning methods Dosovitskiy et al. (2015); 1lg et al.) (2017); Teed & Deng (2020) have been proved suc- cessful estimating the optical flow thanks to the availability of kinds of high quality synthetic
datasets (Butler et al. 2012b Dosovitskiy et al. 2015 Mayer et al. 2016: Krispin et al. 2016).
Most of the recent works (Dosovitskiy et al. 2015 [lg ct al. 2017 Teed & Deng 2020 Jeong et al, 2022) mainly train the synthetic datasets given that there is sufficient real labeled optical flow datasets to be used to train a deep learning model. State-of-the-ar art (SOTA) models always get more accurate results on the synthetic dataset like Sintel (Butler et al. 2012a) than the real scene dataset like KITTI 2015 (Menze & Geiger 2015). This is mainly because that the model tends to overfit the small training data, which echos in Tab. [ i.e., there error and test Fl-all error when train and test on the KITTI dataset in all of the previous SOTA methods. Therefore, we argue that this gap in performance because of dearth of real training data, and a big distribution gap between the synthetic data and real scene data. Although the model can perfectly explain all kinds of synthetic data, however, when dealing with real data, it performs rather unsatisfactorily. Our proposed work focuses on bridging the glaring performance gap between the synthetic data and the real scene data. As in previous data driven approaches, smarter and longer training strategies prove to be beneficial and helps in obtaining better optical flow results. Through our work we also try to find answers of the following two questions: (i) How to take advantage of the current SOTA optical flow models to further consolidate gain on real datasets? and (ii) How can we use semi- -umi-supere learning along with contrastive feature representation learning strategies to effectively utilize the huge amount of unlabeled real data at our disposal?
big gap between the training Fl-all

---

Under review as a conference paper at ICLR 2023
Unsupervised visual representation learning (He et al. 2020 Chen al. 2020b has proved suc- cessful in boosting most of major vision related tasks like image classification, object detection and semantic segmentation to name a few. Work such (He et al. 2020; Chen et al. 2020b) also emphasizes the importance of the contrastive loss, when dealing with huge dense dataset. Given that optical flow tasks generally lack real ground truth labels, we ask if leveraging the unsupervised visual representation learning boosts optical flow performance? In order to answer this question, we examine the impact of contrastive learning and pseudo labeling during training under semi- supervised setting. We particularly conduct exhaustive experiments using KITTI-Raw Geiger et al. (2013) and KITTI 2015 (Menze & Geiger] 2015) datasets to evaluate its performance gain, and show encouraging results. We believe that gain seen in model's performance is reflective of the fact that employing representation learning techniques such as contrasting learning helps in achieving much more refined 4D cost correlation volume. To constrain the flow per pixel, we employ simple positional encoding of 2D cartesian coordinates on the input frames as suggested in (Liu et al. (018)) which further consolidates the gain achieved by contrastive learning. We follow this up with an iterative-floww crrraive-flow-refinnee refinemeen training using pseudo labeling which further consolidates on the previous gains to give us SOTA results. At this point we would also like highlight that follow a specific and well calibrated training strategy to fully exploit the gains of our method. Without loss of generality, we use RAFT (Teed & Deng, 2020) as the backbone network for our experiments. To fairly compare our method with existing SOTA methods, we tested our proposed method on the KITTI 2015 test dataset, and we achieve the best Fl-all error score among all the published methods by a significant margin.
To summarize our main contributions: 1) We provide a detailed training strategy, which uses SSL methods on top of the well known RAFT model to improve SOTA performance for optical flow estimation. 2) We present the ways to employ contrastive learning and pseudo labeling effectively and intelligently, such that both jointly help in improving upon existing benchmarks. 3) We discuss the positive impact of a simple 2D positional encoding, which benefits flow training both for Sintel and KITTI 2015 datasets.
2 RELATED WORK
Optical flow estimation. Maximizing visual similarity between neighboring frames by formulat- ing the problem as an energy minimization (Black& & Anandan [993] Bruhn al.) [0055] |unn al.]
2014) has been the primary approach for optical flow estimation. Previous works such as (Dosovit-
skiy et al. 2015 Mg et al., 2017 Ranjan & Black 2017 Sun et al. 2018a 2019 Hui et al. 2018 2020 Zou et al. 2018) have successfully established efficayy of deep neural networks in estimating optical flow both under supervised and self-supervised settings. Iterative improvement in model ar- chitecture and better regularization terms has been primarily responsible for achieving better results. But, most of these works fail to better handle occlusion, small fast-moving objects, capture global motion and rectify and recover from early mistakes.
To overcome these limitations, Teed & Deng (2020) proposed RAFT, which adopts a learning-to- optimize strategy using a recurrent GaU-based decoder to iteratively update a flow field f which is initialized at zero. Inspired by the success of RAFT, there has been a number of variants such as CRAFT (Sui et al.) 2022), GMA (Jiang et al. 2021), Sparse volume RAFT (Jiang et al. 2021) and FlowFormer (Huang et al. 2022). all of which benefits from all-pair correlation volume way of estimating optical flow. The current state-of-the-arr work RAFT-OCTC (Jenng et al. 2022) also uses RAFT based architecture, and it imposes consistency based on various proxy tasks to improve flow estimation. Considering RAFT's effectiveness, generalizability and relatively smaller model size, we adopt RAFT (Teed & Deng. 2020) as our base architecture and employ semi-supervised iterative pseudo labeling together with the contrastive flow loss, to achieve state-of-the-ae result on KITTI 2015 Menze & Geiger 2015) benchmark.
Semi-Supervised and Representation Learning. Semi-supervised learning (SSL) and represen- tation learning have shown success for a range of computer vision tasks, both during the pretext task training and during specific downstream tasks. Most of these methods leverage contrastive learning
(Chen al. 2020a bi He et al. 2020). clustering (Caron et al. 2020) and pseudo-labeling (Caron
ct al. 2021 Chen & He 2021 Grill et al. 2020
Hoyer et al.
2021 as an enforcing mechanism.
Recent works such (Caron et al. 2020
2021
Chen al. 2020a b 2021
Grill et al. 2020 He
2

---

Under review as a conference paper at ICLR 2023
GT Flow
Feature Warp
Contrastive
Wrspped Festure
Coordinate Frame
Frame
Backward Gradient
HvWxa
HkWWxs
Matrix Mulipl cation
Frame
Coordinate Frame
4D Correiation Volume
Forvard Flow
HoWhd
HoWod
0|0
Goordinate Frame
Iterative GRU
Flow Preciction
Fiow Loss
2D
Encoding
Encoder
Features
Map
Figure I: Overview of our Contrastive Flow ModeI(CLIP-Flowy Input frames firstly positionally encoded, according to individual 2D pixel locations. Subsequently contrastive loss are applied on encoded features to enforce pixel wise consistency using GT Flow.
et al. 2020 Xie al. 2021b] Yun al.. 2022) have empirically shown benefits of SSL for down- stream tasks such as image classification, object detection, instance and semantic segmentation. These works leverage contrastive loss in different shapes and forms facilitate better representa- tion learning. For example, (He et al. 20200 Chen et al. (0200) advocate for finding positive and negative keys with respect given encoded query to enforce contrastive loss. On similar lines, for dense prediction tasks, studies such as (O Pinheiro et al. 2020; Xiao et al. 202T Xie et al,, 2021a) enforce matching overlapping regions between two augmented images, where as Yunn al,, 2022 looks to form a positive/negative pair between adjacent patches. As part of our approach we use contrastive flow loss between features of the neighboring frames, where we draw a one-to-one positive pair relations between the reference features and the warped features using (pscudo) ground truth flow or flow estimates, as shown in Fig. 0
Pseudo labeling is another important approach in SSL training paradigm. Some studies leverage pseudo labeling to generate training labels as part of consistency training (Yun et al. 2019 Olsson et al. 2021 Hoyer et al.) (021), while other works propose pretext task training as in (Caron et al.) 2021] Chen & He 2021 Grill et al., 2020). Rather, we use pseudo labeling for an iterative refinement mechanism, through which correct flow estimate for the KITTI-Raw dataset (Geiger et al. 2013). To the best of our knowledge, ours is the first work which leverages both contrastive learning and pseudo labeling for estimating optical flow in an SSL fashion.
pscudo labeling improve training
effectively distill the
3 APPROACH
In this section, we describe our method CLIP-Flow, a semi-supervised framework for optical flow estimation by iterative pseudo labeling and contrastive flow loss. Based on two SOTA optical flow networks RAFT (Teed & Deng. 2020) and CRAFT (Sui et al. 2022) as backbone (c.f. Sec
3.1).
obtain non-trivial improvement by leveraging our iterative pseudo labeling (PL) (c.f. Sec. 3.2 and
the proposed contrastive flow loss (c.f. Sec. 3.3). It should be noted that CLIP-Flow can be
easily extended to other optical flow networks, e.g. FlowNet (Dosovitskiy et al. 2015 Ilg et al.)
2017), SpyNet (Ranjan & Black 2017) and PWC-Net (Sun al. 2018a), with little modification.
3.1 PRELIMINARIES
Given two consecutive RGB images I,, In Xx the optical flow f E HR xx is defined as a dense 2D motion field f = (fuz fo), which maps each pixel (u, v) in I to its counterpart (u', u') in I2, with u'
fú and v'
f..
RAFT. Among end-to-end optical flow methods Dosovitskiy et al.
2015 5 1gg et al.
Ranjan 2020). RAFT
2017
& Black 2017 Sun al. 2018a Sui al. 2022 Jeong al. 2022 Teed & Deng

---

Under review as a conference paper at ICLR 2023
features learning-to-oplle to-otimmize strategy using recurrent GRU-based decoder to iteratively update a flow field f which is initialized at zero. Specifically, it extracts features using convolutional encoder go from the imput images I and Iz, and outputs features at 1/8 resolution, ie., 96 (T1) R#'' xW' and go( R x='x where /8 and W for spatial dimension and C=256 for feature dimension. Also, a context network he is applied to the first input image 4t Then all-pair visual similarity is computed by constructing a 4D correlation volume V E R. x W' xX' xW" between features (o(11) and go({2). It can be computed via matrix multiplication (H (H"'"""xx w")x(I"WW") which is further as V -go(T) Se (T2), i.e., (R("F-W")xC (RCX(H"WW)) w')
Teed & Deng 2020)
reshaped to V E """''w"xx (W"""""'''" X xW", Then RAFT builds a 4-layer coorelation pyramid {V*)== by pooling the last two dimensions of V with kernel sizes 2 respectively. The GRU-based decoder estimates a sequence of flow estimates (f.....,ffy) (T=12 or 24) from a zero initialized fo = O. RAFT attains high accuracy, strong generalization as well as high efficiency. We take RAFT as the backbone and achieve boosted performance, i.e. Fl-all errors of 4.11 (ours) vs 5.10 (raft) on KITTI-2015 (Menze& & Geiger 2015) (c.f. Tab. I
CRAFT. To overcome the challenges of large displacements with motion blur and the limited field of view due locality of convolutional features in RAFT, CRAFT (Sui et al.) 2022) proposes to leverage transformer layers to learn global features by considering long-range dependence, and hence revitalize the 4D correlation volume V computation as in RAFT. We also use CRAFT as the backbone and attain improvement, i.e. F1-all errors of 4.66 (ours) vs 4.79 (craft) on KITTI-2015 (Menze & Geiger 2015) (c.f. Tab. D
3.2
ITERATIVE PSEUDO LABELING
Deep learning based optical flow methods are usually pretrained on synthetic data and fine-tuned on small real data. This begs an important question: How to effectively transfer the knowledge learned from synthetic domain to real world scenarios and bridge the big gap between them? Our semi- supervised framework is proposed to improve the performance on real datasets DR. by iteratively transferring the knowledge learned from synthetic data Ds and/or few of available real datasets D'r (with sparse or dense ground truth optical flow labels). Without loss of generality, we assume that the real data Dr consists of i) small amount of training data D' (e.g. KITTI 2015 (Menze & Geiger 2015) training set with 200 image pairs) due to the expensive and tedious labeling by human, ii) a number of testing data D (e.g. KITTI 2015 test set with 200 pairs), and iii) large amount of unlabeled data Di (e.g. KITTI raw dataset (Geiger et al. 2013) having 84,642 images pairs) which is quite similar to the test domain. Therefore, we propose use the unlabeled, real KITTI Raw data by generating pseudo ground truth labels using a master (or teacher) model to transfer the knowledge from pretraining on synthetic data or small real data to real data KITTI 2015 test set.
As shown in Fig. 2 our semi-supervised iterative pseudo labeling training strategy includes 3 steps: 1) Training on a large amount of unlabeled data (D)) supervised by a master (or teacher) model, which is chosen at the beginning as a model pretrained on large-scale synthetic and small real datasets, 2) Conducting k-fold cross validation on the labeled real dataset (D)r) to find best hyper-parameters, e.g. training steps for finetuing Syc, and 3) finetuning our model on the labeled dataset (Dfg using the best
parameters selected above, and updating the finetuned model as new version of the master (or teacher) model to repeat those steps for next iteration, until the pre- defined iteration steps N is reached or the gain of evaluation accuracy on test set (D$,) ) is marginal. The detailed algorithm is illustrated in Alg. T
Semi-supervised learning on unlabeled real dataset. Our proposed iterative pseudo labeling method aims at dealing with real imagery that usually lacks ground truth labels and is difficult be accurately modeled by simulators due to reflective surfaces, sensor noise and illumination conditions (Cai et al. 2020). As shown in Alg. I and Fig. A a pretrained baseline model cie. e.g. RAFT, and our pretrained model & e.g. RAFT-CF, a variant of the bascline RAFT by adding our contrastive flow loss (c.f. 3.3 are provided and they are pretrained using the same input datasets of synthetic images (Ds, including C+T+S for FlyingChairs. FlyingTThin Things3D, Sintel, respectively) and a small number of real ones (Dtr including K+H for KITTI 2015 and HDIK). Then based on the pretrained model do (c.f. line 6 in Alg. [}. we train our model your on the large unlabeled real dataset (D))
We assume the synthetic data large scale and have ground truth optical flow maps.
4

---

Under review as a conference paper at ICLR 2023
Fold Cross valdation
Film
Predictt
Framee
NOSEL
RIFT CP
Preccad
Roe
F2x
TRAINED
Preelood Row
ang Se
Row
Foome
Mester RAFT
Framee
Flow
NIK
OT PAWA
Frome
Bachoard Gradient
Benalive Refinementt
Forward Pass
Figure 2: Overview of our semi-superviied iterative pseudo labeling training strategy. which can be easily ap- plied to existing SOTA architectures. Taking RAFT (Teed & Deng 2020) as an example, given baseline RAFT and RAFT-CF-CE (with proposed Contrastive Flow loss and Coordinate encoding). both pretrained on C+T+S+K+H datasets, short FlyingChairs, FlyingThings3DD Sintel, KITTI 2015 and HDIK, respectively. More details about training provided 4.2
Algorithm Semi-supervised iterative pseudo labeling for optical flow estimation
Input:
synthetic datasets Ds. real datasets DR (including training data Dig test data D and a large amount of unlabeled data 2%)
Input: che
D baseline (e.g. RAFT and CRAFT) pretrained on D3 U D D. Ours (Øss. + pseudo labeling contrastive flow loss) index and total number of iterative pseudo labeling total training steps on Di P training on real data DR.
Input: <o
1: 4- 0 and N e.g.
2: 8 4 0 and S +- e.g. 1.2 x 10
b
3: while << <<N N do
4:
$ 5-0
5:
{= the (DR) if 0 else oour (D%)
pseudo ground truth initialization with our model Pour pretrained on Ds Dj training on unlabeled real data D
6:
Pour Dour
7:
while S do
8:
Oour + L Pour (DR),,
training with pseudo label by loss function C
9:
3 + end while
10:
s <- O Sr &- k-fold (oorr Oour, D
find best steps Sy for finetuing by k-fold cross validation
12:
while I Syc do
13:
Φour + L your (Df))
> finetuning real train data Dj by loss C.
14:
5 4- 1 end while
16:
i <- i+1
b increment index for next pseudo labeling iteration
17:
end while
with the supervision of the pseudo ground truth labels f (c.f. line 5 in Alg. I. The pseudo labels are generated by a master model (c.f. Fig. A), and the master model initialized the pretrained baseline dbs at the first step, and will be updated in the following iteration with our new model Φour-
k-Fold Cross validation on labeled small real dataset Gathering annotated data for real world applications may be too expensive or infeasible. In order to take most of the labeled dataset, we use k-fold cross validation to find the best training hyper-parameters, especially, the best training steps Stt for finetuing on the whole real dataset (Dir- e.g. KITTI 2015 training set). As shown in the Fig. BB we divide the validation dataset into & folds (e.g. k=5), and we evaluate the trained model Pour on the corresponding part of the validation set. As shown in Fig. 3. we pick the best hyper-paramette according to the lowest average validation error.
Finetuning the model. After getting the best hyper- papeer-prrmmterww via the k-fold cross validation, use the obtained hyper parameters to finetune our model dour on the whole real labeled data (D)r (c.f. line 13 in Alg. [T.. We use this finefuned model dour to update the master (or teacher) model to produce second round pseudo ground truth optical flow labels. These steps will be repeated until the predefined iteration N is reached or the gain of accuracy real dataset (D is marginal).
5

---

Under review as a conference paper at ICLR 2023
3.3
CONTRASTIVE FLOW
Self-supervised Contrastive Learning. Contrastive learning first introduced by (Hadsell et al, has become increasingly successful for self-supervised representation learning (Wu et al. Oord et al. 2018 Hjelm et al.) [019 Bachman et al.) 2019 Chen et al.) 2020a] He et al,
2006
2018
Chen al. 2021 computer vision. It learns visual representations such that two similar (positive) points have small distance and two dissimilar (negative) points have a large distance. This can be formulated as dictionary look-up problem given an encoded query 9 and a set of encoded samples (Ao, k1, .1........ as the keys of a dictionary (He et al. 2020; Chen et al. 2021). A general formula of a contrastive loss function, called InfoNCE (Oord et al. 2018), is defined as:
2020
exp (q k- /r)
Cg log exp (q. --) Ex exp (g /T)
where 1+ is the positive sample of the query q, and the set {k-} consists of negative samples of q. 7 is a temperature hyper-paramette and the operator "." is dot product. In general, the query representation = 9, (29)) where Bg an encoder network and 2 is an input query image, and similarly, k- g& (I**) for keys.
Semi-supervised Contrastive Flow. To improve the representations for optical flow, we propose a contrastive flow loss to explicitly supervise the network training for learning better features and hence an informative correlation volume in RAFT (Teed & Deng. 2020). Given the input images J and /2, the extracted features go (T)) and go (T2), and the optical flow f pixel index i in I with the feature gf go (41) is warped to a corresponding pixel index j in 12. with j = i + in 12 and the corresponding feature of E go (12) is sampled from feature map go (/2) via bilinear interpolation.
We consider the corresponding pair (i as a positive pair and other samples (i, k) for k + j as negative pairs. Therefore, we define the contrastive flow loss for index as
exp (g) 93/T)
4 exp (g) 9}/T) + EKELs exp (g) g
And the contrastive flow loss over all the valid pixels which have (pseudo) ground truth in image /7 defined as
N
LCt
log
(3)
N
1=0
where N₂ is the number of valid pixels and recall j = i f. The loss LCT in Eq. 3 can be efficiently computed matrix multiplication.
Coordinate Encoding. Optical flow the task of estimating per-pixel motion between video frames. Therefore, the coordinate of each pixel should be important cue when predicting the optical flow for each pixel pair in 41 and 1z. We proposed to concatenate 2D-coordinate map of each pixel as two additional channels to the original input frames. As shown in Fig. T the new input pair becomes 1 I2 E I H x x5, In the ablation study (Sec. 4.3). we demonstrate that incorporating the coordinate encoding consistently improves the optical flow estimates.
EXPERIMENTS
4.1
EXPERIMENTAL SETTING AND DATASET
In our experiments, we have mainly focused on two recent and well-known base models, which are RAFT(Teed & Deng 2020) and CRAFT et al. 2022). to verify the efficacy of our proposed training strategy. Our pretrained model based on RAFT and CRAFT are achieved by pretraining on dataset such as Sintel al. 2012a), HDIK (Krispin et al., 2016), FlyingChair(Dosoviisii
ct al. 2015). and Flying Mayy et al. 2016). We use similar training and hyperparamter
It could ground truth optical flow a pseudo label predicted by RAFT pretrained in synthetic datascts.

---

Under review as a conference paper at ICLR 2023
setting to achieve these models, as described in the original works. We refer readers to Teed & Deng 2020: Sui et al., 2022) for more details. Training during iterative refinement using pseudo labelling, is done on KITTI flow dataset[Menze & Geiger 2015 Menze et al. 2018), where use KITTI-Raw dataset, which consists of about 84642 images pairs but without any ground-truth labels for optical flow. We call the publicly available 200 KITTI flow dataset with manually labeled optical flow groundtruth as KITTI-Flow- Val, and the other 200 reserved KITTI flow testing data as KITTI-Flow-test.
For fair analysis, we conduct exhaustive ablation experiments and use the standard KITTI-Flow- test dataset, for evaluating our models. In the next few subsections we discuss in detail the re- sults achieved on KITTI dataset using our training strategy. We go on to show that our proposed model (RAFT-CF) along with iterative pseudo labelling helps us achieve state-of-art result for KITTI dataset and shows significant improvement for synthetic dataset as well.
4.2 RESULTS ON KITTI
RAFT Model Training on KITTI dataset could be broadly subdivided in to three stages as shown in the [ and is discussed in sec- tion 3.2 All 3 stages defined in A B, and C), forms one iteration of pseudo-label enabled training. In the first stage (Fig A), we train our proposed RAFT-CF-CE (RAFT with Con- trastive flow with Coordinate embedding) KITTI-Raw dataset using the pseudo flow la- bels, generated using original pretrained RAFT model as our Master RAFT model for 1.2m steps. In this stage our candidate RAFT-CF- CE model will be updated by the supervision of the pseudo flows. In the second stage (Fig BB) we take the updated RAFT-CF-CE model from part A and perform 5-fold cross validation on the KITTI-Flow- Val. by dividing the 200 im- age pairs into parts, where each part includes 40 images pair. We conduct a total of experi- ments on the updated RAFT-CF-CE model, and for each group of cross- voos-valii,tion, we evaluate the F1-all error score at every 200 steps. We then calculate the average F1-all error score for each step through all the 5 groups of training, and pick the one that gives the lowest Fl-all error score as the candidate best suited for fine-tuning we achieve an inflection point around 1.6k training steps, which gives us the best average accuracy on the validation dataset. Through this cross validation step, efficiently narrow down the opti- mum number of training steps required to further fine-tune the model on KITTI-Flow- Val dataset. It also brings the fact that, the model tends grossly over-fit the small training data even after 2k steps.
Fold
Then Fald
0.2 0.0
1.4
2.6
32
Staps On thoushands)
Figure 3: The Fl-all error score curve every 200 steps during the 5-fold cross validation. The (bold) brown curve is the average Fl-all error across all the 5-fold validation experiments.
updated RAFT-CF-CE model. As seen in 3 we find that,
Finally, In the third stage (Fig. 2C). we finetune the updated RAFT-CF-CE-PL model from part A with the whole KITTI-Flow- Val for the optimum steps, which we get from Part B. After finetune, we get our final updated RAFT-CF-CE model for the current iteration. We repeat this process for iterative refinement for few more iterations till the model converges.
The comparative results on KITTI-Flow- Val and methods is presented in Tab. [ Results seen in Tab. [ further validates robustness of our approach, as even-though all of our model have higher training error, performs much better on test dataset. Unlike other SOTA methods, which get lower training error but higher testing error, clearly reflecting towards an over-fitted model. This is also because of the fact that we follow a semi-supervised approach leveraging contrastive and pseudo labels during our training. Therefore, we claim that with the proposed training strategy, our model can be trained for more iterations and even more steps without any data-overiitting issues. In our case, conducted 3 iterations of iterative training and the last iteration has million steps, giving us a Fl-all error score of 4.11 on the KITTI-Flow Flow--tttttt with 2.78 training error on KITTI-Flow- Val. Qualitative improvement using our model is presented
among all the published SOTA
7

---

Under review as a conference paper at ICLR 2023
KTI5
Training dataset
Sintel (train)
KT15 (train)
Sintel (test)
Method
(test)
Clean
Fl-epe
Fl-all
Clean
Fl-all
LicFLowNer2 (Hui 2018
(1.30)
(1.62)
(1.47)
(4.80)
3.48
4.69
7.62
PWC-Nett (Sun el 2019
(1.71)
(2.34)
(1.50)
(5.30)
3.45
4.60
7.72
VCN Yang & Kamanan
2019
(1.66)
(2.24)
(1.16)
(4.10)
2.81
4.40
6.30
MaskFooW/eet Zhao
2020
2.52
4.17
6.10
RAFT Teed & Deng 2020
(0.76) (1.22)
(0.63)
(1.50)
1.61
2.86*
5.10
C-T+S+K+H
GMA Jiang et al. 2021
(0.62)
(1.06)
(0.57)
(1.20)
1.39*
2.47*
5.15
RAFT-OCTC (Jeong el al.
2022
(0.73)
(1.23)
(0.67)
(1.70)
1.82
3.09
4.72
RAFT-OCTC!
(0.74)
(1.24)
(0.71)
(2.00)
1.58
2.95
RAFT-OCTC!
(0.78)
(2.30)
1.41
2.57
4.33
CRAFT (Sui al. 2022
(0.60)
(1.06)
(0.58)
(1.34)
1,45
2.42
4.79
RAFT-A (Sun 2021
Ar
2,01
3.14
4.78
Ours (RAFT-CI-CE)
1.52
2.65
Ours (RAFT-CF-CE-PL)
(1.24)
(3.99)
4.38
Ours (RAFT-CF-CF-PI22)
(1.10)
(3.18)
C+T+S+K+H
Ours (RAFT-CF-CE-PL3)
(1.02)
(2.78)
4.11
Ours (CRAFT-CE-PLI)
(1.67)
(5.12)
4.66
Ours (CRAFT-CF-CE-PL.I)
(1.71)
(5.23)
4.68
Optical flow results on Sintel and KITTI 2015. RAFT-CF-CE is our modified version baseline RAFT, by adding contrastive flow (CF) loss and coordinate encoding (CE). Similarly, -PL. is for the adding of proposed semi-supervised iterative pseudo labeling (PL) training strategy. and 1PL1, and -PL3 repre- sent the 1st, 2nd and 3rd iteration respectively. The similar setup is for the backbone CRAFT "Results evaluated with the "warm-star"" strategy detailed in RAFT. (Result) denotes result on training sets, listed here for reference only. Bold for best results and the second best results are underlined.
Table I:
al. (2022).
in Fig. on KITTI -Flow-test 2015 (Menze & Geiger, 2015). The right column is our predicted optical flow results, the middle column the results from the baseline model RAFT. Our model, as seen from the results, is make finer flow predictions and also better handles occluded pixels.
CRAFT In order to demonstrate generalizability of the proposed contrastive loss and iterative pseudo labeling training strategy. we also trained the most recently released state-of-art model CRAFT(Sui et al., 2022) with our training strategy. We follow similar 3 stage approach as de- scribed in RAFT Model paragraph of 4.2 for the CRAFT model. During our experiments, we found that training CRAFT based model takes unexpectedly longer time compared to one based on RAFT. Due to this constrain, we ran the CRAFT based CRAFT-CE-PL and CRAFT-CF-CE-PP model for iteration with 620k and 420k steps respectively. We then tested our final model on KITTI-Flow- test and we get F1-all error score of 4.66 and 4.68, which outperforms original CRAFT model(4.79 F1-all error) by a good margin, as shown in table [
4.3
ABLATIONS
iteration
steps
Fi-ali(val)
Fi-all(test)
Models (@step 1.2m)
Fl-all (validation)
FI all (test)
1st
1.2m
3.99
4.38
2nd
1.2m
3.42
RAFT-PL
4.531
4.48
3rd
1.2m
3.18
4.13
CAFF-CE-PP
4.45
2.78
4.11
RAFT-CF-CE.P.
4.38
Table 2: The comparison of Fl-all enror on KITTI- Flow- Val and KITTI-Flow-tsst different iteration and different training steps.
The comparison of Fl-all error KITTI-Flow-Va and KITTI-Flow-test for differ- ent RAFT based model.
Table 3:
As outlined in the earlier sections, our method can be broadly divided in to three independent sub- parts, which are: 1) Employing iterative training with Pseudo Labeling (PL), 2) Adding Contrastive Flow loss to the baseline encoder features (CF), and 3) Applying Coordinate Encoding (CE) to the raw input frames. Here we will analyze how each of them contribute the improved performance. (b) shows the Fl-all error score on KITTI-Flow-V on trained on KITTI-Raw. The middle green curve is model RAFT-CE-PL, and the bottom blue is the RAFT-CF-CE-PL model. Through the experiments it shows that the proposed iterative pseudo labeling training strategy itself will highly increase the predicted optical flow accuracy. Adding coordinate embedding and contrastive flow loss will further increase the optical flow accuracy. The quantitative results shown in Table
The top red curve the RAFT-PL model
8

---

Under review as a conference paper at ICLR 2023
Fl-ell validation
Fl-all Validation Error
PAFT-CE-LL RIFT-F-CE-
heeation
0.3
Faalener
50-
420
600
90C
1000
1200
3400
200000
400000
G00000 800000
Sin0s Choushandds
(a)
(b)
Figure 4: (a) F1-all error curve on KITTI-Flow datasct when training (Our) RAPT-CF-CE-PL. model on the KITTI-Raw Raw dataset. 1st iteration (blue curve) supervised by pseudo flow generated using the original RAFT. In 2nd and 3rd iteration, it is supervised by the updated RAFT-CF-CE-PP and RAFT-CT-CE-PLI model respectively. (b) The curve shows, Fl-all error on KITTI-Flow dataset for different RAFT based model. sec our RAFT-CF-CE- model, performs the best.
Figure 5: Qualitative results of optical flows: left column the original image, middle column the optical flow of the original RAFT model, the right column(circled by green box) is the output of our model.
CONCLUSIONS
In this paper we have proposed a novel and effective semi-supervised learning strategy, with iterative pseudo label refinement and contrastive flow loss. Our framework aims transferring the knowledge pretrained on the synthetic data to the target real domain. Through the iterative pscudo label refine- ment, we can leverage the ubiquitous, unlabeled real data to facilitate dense optical flow training and bridge the domain gap between the synthetic and the real. The contrastive flow loss is applied pair of corresponding features (one warped to another via the pseudo ground truth flow), to boost accurate matching due to reliable pseudo labels and to dampen mismatching due to noisy pscudo labels, occlusion or global motion. Experiments results on KITTI 2015 and Sintel using two back- bones RAFT and CRAFT. demonstrate the effectiveness of proposed semi-supervised learning framework. We obtain the second best result (Fl-all error of 4.11%) on KITTI 2015 benchmark among all the non-stereo methods (c.f. Sec. A for detailed rankings).

---

Under review as a conference paper at ICLR 2023
REFERENCES
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. NeurlPS, 32, 2019.
Michael Black and Padmanabhan Anandan, A framework for the robust estimation of optical flow. In ICCV, pp. 231-236. IEEE, 1993.
Nicolas Bonneel, James Tompkin, Kalyan Sunkavalli, Deqing Sun, Sylvain Paris, and Hanspeter Pfister. Blind video temporal consistency. ACM Transactions Graphics (TOG), 34(6):1-9, 2015.
Andrés Bruhn, Joachim Weickert, and Christoph Schnörr. Lucas/kanade meets horn/schunck: Com- bining local and global optic flow methods. IJCV, 2133,211-21,,11
D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black. A naturalistic open source movie for optical flow evaluation. A. Fitzgibbon et al. (Eds.) (ed.), ECCV, Part IV, LNCS 7577, pp. 611-625. Springer- Verlag, October 20
Daniel Butler, Jonas Wulff, Garrett B Stanley, and Michael Black. A naturalistic open source movie for optical flow evaluation. In ECCV, pp. 61 611-625, Springer, 2012b.
Changjiang Cai, Matteo Poggi, Stefano Mattoccia, and Philippos Mordohai. Matching space stereo networks for cross-domain generalization. In 3/V, pp. 364-373 2020. doi: 10. 10.1 109/3///90 2020.00046.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. NeurIPS, 33:9912- 9924, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, pp. 9650- 9660, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, pp. 1597-1607. PMLR, 2020a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, pp- 15750-15758 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint S.Xiv:2003.04297, 2020b.
Xinlei Chen, Saining Xie, and Kaiming He. empirical study of training self-supervised vision transformers. In CVPR, pp. 9640-9644 2021.
Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In ICCV, pp. 2758-2766, 2015.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The KITTI dataset. The International Journal of Robotics Research, 32(11):1231-133) 2013.
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. NeurIPS, 33:21271- 21284, 2020.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In CVPR, 2006.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pp. 9739, 2020.
10

---

Under review as a conference paper at ICLR 2023
Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchilldnn, Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, April 2019.
B.K. Horn and B.G Schunck. Determining optical flow, in: Techniques and applications of image understanding. International Suciely for Optics and Photonics, 281, 1981.
Lukas Hoyer, Dengxin Dai, Yuhua Chen, Adrian Koring, Suman Saha, and Luc Van Gool. Three ways to improve semantic segmentation with self-supervised depth estimation. In CVPR, pp- 11130-11140, 2021.
Zhaoyang Huang, Xiaoyu Shi, Chao Zhang, Qiang Wang, Ka Chun Cheung, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Flowformer: transformer architecture for optical flow. arXiv preprint arXiv:2203.16194, 2022.
Tak-Wai Hui, Xiaoou Tang, and Chen Change Loy. Liteflownet: A lightweight convolutional neural network for optical flow estimation. In CVPR, pp. 8981-8989, 2018.
Tak-Wai Hui, Xiaoou Tang, and Chen Change Loy. A lightweight optical flow renn--iiiiiing data fidelity and regularization. PAMI, 43(8):2555 2020.
Eddy llg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. CVPR, pp. 2462-2470, 2017.
Joel Janai, Fatma Güney, Aseem Behl, and Andreas Geiger. Computer vision for autonomous vehi- cles: Problems, datasets and state-of-the-- Foundations and Trends in Computer Graphics and Vision, 12(1-3):1-308. 2020.
Jisoo Jeong, Jamie Menjay Lin, Fatih Porikli, and Nojun Kwak. Imposing consistency for optical flow estimation. In CVPR, pp. 3181-3191, 2022.
Shihao Jiang, Dylan Campbell, Yao Lu, Hongdong Li, and Richard Hartley. Learning to estimate hidden motions with global motion aggregation. In ICCV, pp. 9772-9781, 2021.
Jonas Krispin, Alexander Andrulis, Burkhard Brock, Mohsen Gussefeld, Sabine Rahimimoghad- dam, Claus Hofmann, and et al. Brenner. A large dataset train convolutional networks for disparity, optical flow, and scene flow estimation. CVPR workshop, 2016.
Rosanne Liu, Joel Lehman, Piero Molino, Felipe Petroski Such, Eric Frank, Alex Sergeev, and Jason Yosinski. An intriguing failing of convolutional neural networks and the coordconv solution. NeurlPS, 31, 2018.
Nikolaus Mayer, Eddy Ilg, Philip Häusser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. The hci benchmark suite: Stereo and flow ground truth with uncertainties for urban autonomous driving. In CVPR, 2016.
M. Menze and A Geiger. Object scene flow for autonomous vehicles. In CVPR, 2015.
M. Menze, C. Heipke, and A. Geiger. Diserete optimization for optical flow. In German Conference on Pattern Recognition, 2015.
Moritz Menze. Christian Heipke, and Andreas Geiger. Object scene flow. ISPRS Journal of Pho- togrammetry and Remote Sensing (JPRS), 2018.
Pedro O O Pinheiro, Amjad Almahairi, Ryan Benmalek, Florian Golemo, and Aaron C Courville. Unsupervised learning of dense visual representations. NeurlPS, 33:4489945000 2020.
Viktor Olsson, Wilhelm Tranheden, Juliano Pinto, and Lennart Svensson. Classmix: Segmentation- based data augmentation for cemi-supervised learning. WACV, pp. 1369-1378, 2021.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint / 1807003748 2018.

---

Under review as a conference paper at ICLR 2023
R. Ranftl, K. Bredies, and T. Pock. Non-local total generalized variation for optical flow estimation. In ECCV, 2014.
Anurag Ranjan and Michael Black. Optical flow estimation using a spatial pyramid network. In CVPR. pp. 4161-41777 2017.
Laura Sevilla-Lara, Yiyi Liao, Fatma Güney, Varun Jampani, Andreas Geiger, and Michael Black.
On the integration of optical flow and action recognition. In German conference on pattern recognition, pp. 281-297. Springer, 2018.
Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. NeurIPS, 27, 2014.
Xiuchao Sui, Shaohua Li, Xue Geng, Yan Wu, Xinxing Xu, Yong Liu, Rick Goh, and Hongyuan Zhu. Craft: aoossaatt flow transformer for robust optical flow. In CVPR, pp. 17602- 1, 2022.
Deqing Sun, Stefan Roth, and Michael J Black. A quantitative analysis of current practices in optical flow estimation and the principles behind them. IJCV, 106(2)115-137 2014.
Deqing Sun, Xiaodong Yang, Ming- Yu Liu, and Jan Kautz. Pwe-net: Cnns for optical flow using pyramid, warping, and cost volume. In CVPR, pp. 8934 8943, 2018a.
Deqing Sun, Xiaodong Yang. Ming-Yo Yu Liu, and Jan Kautz. Models matter, so does training: An empirical study of cnns for optical flow estimation. PAMI, 42(6): 140:--108222 2019.
Deqing Sun, Daniel Vlasic, Charles Herrmann, Varun Jampani, Michael Krainin, Huiwen Chang. Ramin Zabih, William T Freeman, and Ce Liu. AutoFlow: Learning better training set for optical flow. In CVPR, 2021.
Shuyang Sun, Zhanghui Kuang, Lu Sheng, Wanli Ouyang, and Wei Zhang. Optical flow guided feature: A fast and robust motion representation for video action recognition. In CVPR, pp- 1390-1399, 2018b.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In ECCV, pp. 402-419. Springer, 2020.
Zhirong Wu, Yuanjun Xiong, X Yu Stella, and Dahua Lin. Unsupervised feature learning via parametric instance discrimination. CVPR, 2018.
Tete Xiao, Colorado Reed, Xiaolong Wang, Kurt Keutzer, and Trevor Darrell. Region similarity representation learning. ICCV, pp. 10539-10548, 2021.
Enze Xie, Jian Ding, Wenhai Wang, Xiaohang Zhan, Hang Xu, Peize Sun, Zhenguo Li, and Ping Luo. Detco: Unsupervised contrastive learning for object detection. In ICCV, pp. 8392 8401, October 2021a.
Zhenda Xie, Yutong Lin, Zhuliang Yao, Zheng Zhang, Qi Dai, Yue Cao, and Han Hu. Self- supervised learning with swin transformers. arXiv preprint
2021b.
Charig Yang, Hala Lamdouar, Erika Lu, Andrew Zisserman, and Weidi Xie. Self-supervised video object segmentation by motion grouping. In ICCV, pp. 7177-7188, 2021.
Gengshan Yang and Deva Ramanan. Volumetric correspondence networks for optical flow. In NeurlPS, pp. 895- 2019.
Gengshan Yang and Deva Ramanan. Learning to segment rigid motions from two frames. In CVPR, pp. 1266-1275, 2021.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, pp. 6023-6032, 2019.
Sukmin Yun, Hankook Lee, Jachyung Kim, and Jinwoo Shin. Patch-level representation learning for self-supervised vision transformers. In CVPR, pp. 8354-8363 2022.
12

---

Under review as a conference paper at ICLR 2023
C. Zach, T. Pock, and H Bischof. A duality based approach for realtime tv-1 1 optical flow. In Joint patter recognition synposium, 2007.
Shengyu Zhao, Yilun Sheng, Yue Dong, Eric I-Chao Chang, and Yan Xu. Maskflownet: Asymmetric feature matching with learnable occlusion mask. In CVPR, 2020.
Yuliang Zou, Zelun Luo, and Jia-Bin Huang. Df-net: Unsupervised joint learning of depth and flow using cross-task consistency. In ECCV, pp. 36-53, 2018.
A
APPENDIX
We show additional results in this appendix.
A.1
SCREENSHOTS OF KITTI 2015 OPTICAL FLOW BENCHMARK
Fig. 6 shows the evaluation results the KITTI 2015 optical flow benchmark Based the
baseline RAFT (Teed & Deng 2020), our model RAFT-CF-PL3 (with CF for contrastive flow, and PL3 for iterative pseudo labeling the 3rd iteration) obtains Fl-all error of 4.11%, i.e. 19% error reduction with respect to RAFT (5.10%). Our model outperforms most of the evaluated non- stereo methods as the time of submission, except for RAFT-OCTC (Joong et al. 2022). The not-trivial improvement compared with RAFT, demonstrates the effectiveness of i) our proposed semi-supervised iterative pseudo labeling training strategy and, ii) the proposed contrastive flow loss, which facilitates the semi-supervised training by dampening the mismatching or displacement due to motion blur or occlusion when leveraging the pscudo labeling.
A.2
SCREENSHOTS OF SINTEL BENCHMARK
Fig. 7 and Fig. show the evaluation results on the MPT Sintel benchmark | on clean pass and final pass, respectively. Upon the baseline RAFT (Teed & Deng 2020), our model RAFT-CF (with CF for contrastive flow, and no iterative pscudo labeling due to Sintel having ground truth labels) obtains an epe of 1.519 (clean pass). i.e. a 6% eтror reduction from RAFT with 1.609; and an epe of 2.645 (final pass), 7% error reduction from RAFT with 2.855. The improved results show the effectiveness of proposed contrastive flow loss. It helps the supervised training on the synthetic Sintel dataset (Butler et al. 2012a) by mitigating the mismatching due to occlusion, small fast-moving objects, and global motion, even though having the ground truth optical flow.
www. www.evili........... hwtp://tttttttttttttt////aaaa scene flow www.:pppbbbccccaaaaa-oooooo
'Strroo methods uses left and right (stereo) images, but RAFT and ours use left images only.
is. www.mpg..e.. http://mm.....aaoooooo/o//ononne
13

---

Under review as a conference paper at ICLR 2023
Evaluation ground truth All pisels
Evaluation area
All picels
Method
Setting
Code
Fl-bg
F1-1g
Fiail
Density
Runtime
Environment
Compare
Camti Flowe
2.07 %
6.77 X
2.85 %
100.00 x
1.
GPU 8 2.5 Gtz (Pythan C/Cre)
Cars FFlw
sade
7,04
3.10%
100.00
1.25
GPU 2.5 Ghz (Pythan C/C+.)
Lio, Ta Na, Lka,
Chere Gankiiler Infrrctiionll Comerasi000 Datron for Jonn Octicnl Flow and Dine Entirationn CiRa
H-FUSE
2.66 X
7.47 X
3.46 X
100.00 X
1.35
GPU
RigidHask:ISE
code
2.63 x
7.85 x
3.50
100.00 X
3.3s
GPU e 2.5 Ghz (Python)
BAFT-OCTS
3.72
5
100.00 X
0.2
GPU 2.5 Ghz (Python)
Joong. Purkii Reak: insosios Consiionnes Setioal Snleration Soaterre Fassedss
3F258
code
3.17%
8.79 5
4.11%
100.00 8
2.75
GPU +155 Ghz (Python)
Sommec Schntoppll and Troe g2ace
Selection
DAGN German Conference Patter Recogriien 2892.
RAFT-CF-PL3
3.80 X
5.65
4.11
100.00 X
0.05
GPU @ 2.5 Ghz (Pythen)
DP
3.36% 5
5.96
4.21%
100.00 8
0.15
core e 2.5 Ghz (Python)
RAFT-30
28
3.39%
8.79%
4.29%
100.00 X
2.
CPU 8 2.5 Ghz (Pythan + C/Ca.)
prepries
3000.
B
4.11 X
5.34%
100.00
0.1
GPU 2.5 Ghz (Python)
GHFlow BVC
4.16 X
5.67%
4
100.00 X
0.25
GPU e 2.5 Ghz (Python)
12
CCH-Flow
4.20 X
5.30%
4
100.00 X
0.25
core @ 2.5 Ghz (Python)
4.27 X
5.60%
4.49%
100.00 X
0.2s
GPU (Python)
Sebarabielow
code
4.25 X
5.92%
4.53%
100.00 X
0.5
Zhang Moedfard, V. Risacaria and Tan: Snsurable Freer Leersie Mution Cesl Mates
Ostical Flom Ediestian Proceedings of the EEE/CW Intematinnll Corference en Computer Voien 2011
15
Netaflow
4.11%
6.77% 5|
4.55 8
100.00 8
0.2
core e 2.5 Ghz (Python)
16
5P6 XP6-flow
4.17 X
6.77 5
4.60
GPU @ 2.5 Ghz (Python) the (222 CWF Conference en Computer Vaion Pattern Racogrizon 2022
100.00 X
0.25
Luc, F Yang. X, and Uas
With
Patch Attecton
17
StefsTR-SAFT
code
4.32.%
6.05
4.61
100.00 X
0.6
8 cores @ 2.5 Ghz (Python)
18
BealFlow
4.20
6.76%
4.63%
100.00 X
0.2
8 cores @ 2.5 Ghz (Python)
DGé-Flnw
4.34.
6.11%
4.64%
100.00 8
0.25
core 2.5 Ghz (Python)
ECTR.m
4.45
5.63%
4.65
100.00
0.25
2.5 Ghz (Python)
FlowNAS RAFT-K
4.36
6.25%
4.67%
100.00
0.19
GPU 2.5 Ghz (Python)
FlowFormet
sode
4.37%
6.185
4.68%
100.00 8
0.35
GPU (Python)
Heang, Chang, Q. Weng, Oveing On.
Reeformen Arrende
Awchhlstthtttss.oooooooo Octicll Ros Comppee computer
CRAFT intramdees
sode
8
6.35%
4.68%
100.00 8
0.25
core (Python)
IPCV-RAFT
5.52
4.09%
100.00
0.25
Z.3ghz gpu
UbberTGG 0RISE
3.59%
10
4.73
100.00 X
0.75
CPU-GPU 2.5 Ghz (Python)
No. Wing, Ho, Rinn and R. Uitsue Deno Breid inttavee Sceee Ples CWN 2010.
26
édallatic
sode
4.46 5
6.23%
4.75
100.00%
0.25
core e 2.5 Ghz (Python)
27
Supar
4.43%
6.43%
4.76%
100.00 X
0.07
GPU @ 2.5 Ghz (Python)
RAFT-A
code
4.54
5.99%
4.78
100.00 X
0.7
GPU 8 2.5 Ghz (Pythan + C/Coe)
Sun, D. Vlasi C. Hermam, Jangani, M. Kralein H. Chang, R. 220/i W. Freemar and Lau Axtallen Leanine Better Taiiie Set for Sotioal Bane CVPA 2001.
CRAFT
code
4.58 K
5.85 8
4.79 %
100.00 8
0.2s
GPU e 2.5 Ghz (Python)
Ras 2022
GWPPinweeet
sode
4.39 5
6.84
4.79%
100.00
0.5
GPU e 2.5 Ghz (Python)
Dhan, 2. Thang, E. Metsams Global Overooeein Mitertion Spsiral Flow Extreatoss
CRAFT-uutttios
6.54%
4.84%
100.00
0.15
2.3 Ghz (Python)
SkFlow
4.64%
5.83%
4.84%
100.00
0.25
GPU 2.5 Ghz (Python)
RAFT-DFlow
4.52 K
6.48
4,84%
100.00 %
0.25
1 core 2.5 Ghz (C/C..)
34
BAFT-R5
4.60
6.09
4.85
100.00 8
0.15
GPU e 2.5 Ghz (Python)
Figure 6: Screenshot of the KITTI 2015 optical flow benchmark Menze & Geiger (2015)
evaluation
results. Our model RAFT-CF-PL3 (CF for contrastive flow, and PL3 for iterative pseudo labeling at 3rd iteration) modified based on the backhone RAFT, obtains an F1-all error of 4.11%, i.e. a 19% error reduction from RAFT (5. 10%). ranking the 2nd place among all evaluated non- noe-stereo methods as of September 28, 2022. Our model is slightly outperformed by RAFT-OCTC (Jeong al. 2022)
14

---

Under review as a conference paper at ICLR 2023
ESC
EPC matched
EPC unmatched
40-10
d10-00
203-140
60-10
x1C-40
a40+
Broundfiuuh
8.000
5.000
0.000
0.000
GMFow
0.335
5,800
0.88
2783
0.5
AploFFlww
0.511
R.081
3.43
0.777
0.853
NlowFommr"
8.428
7,637
1:37
0.000
Waudorr Read
SKPkw
8.557
7.39
1.500
2.31
0.100
0.999
7:188
Naudoo Reads
M48 PAPT
0.700
1.310
0.707
Maidoo Read
SwinTR-RAPT
1.579
8.029
8.344
1.222
0.400
2.27
0.325
0.817
7.728
Maudos Bexas
CMA
8.089
1.837
0.381
0.823
Nauatos Faxurs
GMFlowNet
1.892
8.620
6.466
1.275
0.386
3.298
0.314
0.991
7.098
Moudla Sexult
GNA-LCTT/Row
1.408
8.825
6.611
1.428
0.40c
3.251
0.279
0.878
8.290
Macalio Masutt
AGF-Flowd
1.40
8.025
6.16
1.433
0.408
2,245
0.278
0.875
5.022
RFPM
1
0.434
6.884
1.235
0.400
3.221
0.275
0.87
5:345
RAFT-OOTOO
1
0:041
5.874
1.485
0.442
2.242
0.201
0.040
5.112
Niu Wa-iBii
GMA-FS
1.430
8.832
6:771
1.579
0.470
3,865
0.355
0.077
73861
AdFlww
1.31
8.559
8.541
1.501
0.422
2.851
0.319
0.903
8.075
Warders Bondle
DIP
1,40e
8:579
8.899
1.702
0.407
23322
0.338
0.75
5.445
Tuaskor Readln
CRAFT (M)
1.001
0.011
8:254
1.574
0.500
3.84F
0.111
0.811
A1311
Wen Beadle
GNA base
1.450
8:531
E.410
1.532
0.470
2,082
0.321
0.811
8.51
Taaaoor Reado
SKFlow PAFT
1.461
8.817
6.366
1,895
0.514
3.007
0.389
1.021
8:173
Ancnymous
1.474
8.66
6.445
1.807
0.601
2,802
0.311
0.833
2,229
Secursblellan Zeievs 21
1
8.687
B.78
1.474
0.481
3,357
0.309
0.883
3,091
Ma Bacut
DEQ Flow-H
1.008
8:648
6:200
1.427
0.477
3,232
0.208
0.078
8.700
RAP T- PPI
1.619
8:631
8:21
1,809
0.424
3,228
0.201
0.827
9,441
FCTR.v
1.504
8:575
5:284
1.812
0.488
3,286
0.325
0.979
8.70
RAP Tworss AUCE
1.544
9:591
5.099
1,85
0.412
3,285
0.779
0.911
9.23
PM
1.545
8:535
5.255
1.535
0.477
3,206
0.348
1.023
5.738
RAFT-- 2T1
1.544
8.012
5.242
1.004
3,272
0.207
0.071
SCAR
1.573
8:635
84485
1,813
3,285
0.314
1.018
9.212
-08S 201
1.002
8.000
5.022
1.022
0.300
0.200
0.090
9.173
OER
8-477
1.557
2.335
0.32%
1.014
D1
8:366
2.364
0.316
0.981
2:315
8.971
2X
5:47
Figure 7: Screenshot of the MPI Sintel (clean pass) benchmark evaluation results. Our model RAFT- CF (CF for contrastive flow, and no iterative pseudo labeling due to Sintel having ground truth labels) modified based on the backbone RAFT, obtains an end aod-point-error (epe) of 1.519, i. 6% error reduction from RAFT (1.609), showing the effectiveness of our proposed contrastive flow loss.
15

---

Under review as a conference paper at ICLR 2023
GPC all
EPG matehed
IPS unmatched
40-10
410-40
600
140
60:10
615-40
640
GroundTruth
0.000
0000
0.000
0.000
Rptusow
0.500
1.22
11.03
RowFormer
0.519
1.40
11:438
Waunday Houitt
BKFlww
11.014
12,048
Wendde
GMRoww
2.097
0.455
14.377
Wausdao Houita
Asosymous
2.500
0.000
12,008
We Houult
CRAFT
0:547
0.530
13,055
Wauddo Hoviit
GMA-FS
0.584
0.287
1.088
13.578
Waudder Houita
ABRow
0.680
0.000
13.015
GMA
2.885
0.553
0.508
13.402
Wauddo Houilt
RAFT ocre
1.015
0:677
0.570
1.701
14.004
Masder Porz
SKFlow. RAFT
0.044
0.64
14.373
Yauddo Hooila
RAPT-CF
2.775
0.099
0.547
10.775
Mauddo Plouit
OMioowlle
1.271
2.816
1.090
0.716
0.000
1.784
14,417
Wasilo Phouitt
ABF Pow
2.008
0.838
0.612
15,483
Waudeo Hooila
MS RAFT
267
0.408
1.011
10.377
Mayslos Rom
Roperableeow Svicws
14.013
0.000
0.200
15.903
ALAe
0.646
1.521
15.49
Masiler Rooults
PCTR-m
2.007
1.201
14.310
2.887
1.032
0.788
0.278
1.701
15.380
Masddo Paoulta
RAPT-NCUU
0.108
1.000
0.000
0.005
1.044
14.043
Yauddo Prouita
AGF-Flawd
2.73
1.217
15.103
2.418
0.812
0.777
0.408
1.400
17::33
Masill Pac
GMAALCT1-FOO
2.419
0.814
0.788
0.465
1.411
17::31
Yaudeo Hooila
ve w
1.10
0.009
Masslb Rec
(lL- ext warm
2,700
1.319
14.007
3.086
1.145
0.37
0.006
1.879
15,002
Masddo Paoulta
LGT Flowo
3.781
2,790
0.866
0.620
1.089
16.455
Masilo Raoults
RAFT-PB-O
2.785
1.341
14.007
3.114
1.104
0:68
0.681
1.800
15,487
Masilo Recults
MPR
2.001
0.678
0.779
0.674
1.009
15.203
Wasddo Prouita
RAF AO
2.813
1.871
14,093
8.066
1.099
0.727
0.608
1.791
16:271
Maudlo Rect
Aduttutin
2.817
1.301
14.007
3.006
1.119
0.773
0.601
1.805
13,90
Wasstoo Hesult
NASPlow how
2.822
1.40
14.889
2.596
1.146
0.50
0.666
1.707
15:14
Yi Rec.
RAF TWs
2,885
1.859
$4.809
8.184
1.116
0.756
0.681
1.802
15.117
Mauilbo Ract
BAFTv2 OER- warm start
8 100
11583
0749
0.628
18.05a
Wasiloo Resulta
DEP
2.884
1.282
15.485
2.728
1.000
0.801
0.571
1.812
16,08
Miuilb Revt
DCG Po---io
2.861
1.089
14.064
2.66c
1.211
0.764
0.664
1,800
18,004
nnb-im
2.864
1.865
14,883
3.089
1.102
0.766
0.600
1.799
18,403
We Dive
BAFT
2,800
1.403
14,000
3.112
1.503
0.700
0.684
1.803
19,07
Figure 8: Screenshot of the MPI Sintel (final pass) benchmark evaluation results. Our model RAFT- CF (CF for contrastive flow, and no iterative pseudo labeling due to Sintel having ground truth labels) modified based on the backbone RAFT, obtains an end-point-ert (epe) of 2.645, 7% eтror reduction from RAFT (2.855), showing the effectiveness of our proposed contrastive flow loss.
16


Reviews:
[{"review_id": "s69_ufsDAfU", "reviewer": "Reviewer_9251", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a semi-supervised optical flow approach based on an iterative pseudo-labeling scheme.\nGiven a pre-trained model (e.g., trained on synthetic datasets), the method computes a pseudo ground truth on an unlabeled target dataset and trains the model on it. The method then fine-tunes the model using a small amount of available ground truth data. \nAfter that, the model generates the pseudo ground truth on the unlabeled target dataset again and loops the process.\nThe paper also proposes a contrastive loss that further improves the accuracy. \nThe proposed ideas consistently improve the accuracy of baseline models.", "summary_of_the_review": "The proposed idea is interesting, and the experiments validate its effectiveness. However, the experiment is limited to only one dataset with one model. It would be really great to see more ablation studies using other models (e.g., CRAFT) on other datasets (e.g, Sintel) in order to see that the method can be generalized to other methods/datasets.\n\nFurthermore, it would be great if some concerns about technical designs and experiment setups, as stated in the weakness section.\n\n\n----\n\n__After the authors' response__\n\nI appreciate the authors' response! The responses resolved most of my main concerns. However, due to the following reasons, I would like to keep my original rating, marginally below the threshold.\n- Is the contrastive loss really helpful? Numbers in the tables show that it doesn't give consistent improvement.\n  - In the last two rows in Table 3 (main paper), adding the contrastive loss (CF) marginally degrades the test accuracy on KITTI 2015 test.\n  - In the ablation study on Sintel (Table 4 in the authors' response), the accuracy dropped quite a bit on EPE-Final (test).\n- Novelty concern: As other reviewers mentioned, I am not so sure the novelty is significant enough. It was a bit difficult to find exciting insights. That said, I don't strongly object to accepting the paper.\n- If the target scenario is a real-world domain, an experiment with other real-world datasets would be also interesting (I am sorry that I didn't think about suggesting it before). It could be HD1k, Slow Flow (their ground truth might not be super accurate), Middlebury (not so many training images though), DAVIS (probably reporting keypoint tracking accuracy?), or any suitable dataset.\n", "strength_and_weaknesses": "__Strength__\n\n- Interesting idea: iterative training + pseudo ground truth generation\n\n  The proposed semi-supervised learning scheme is interesting and is successfully validated through experiments. It was interesting to see that the accuracy on the validation set keeps improving along with the iteration steps, like a positive feedback loop.\n\n- Contrastive learning loss\n\n  The paper demonstrates the effectiveness of the contrastive learning loss for the optical flow task; it helps to learn more discriminative features for the matching task. Its contribution would affect and help other following-up works. \n\n---\n\n__Weakness__\n\n- Limited generalization: Analysis on only one dataset with one model.\n  \n  The paper provides an ablation study only on the KITTI dataset. I wonder if the method can also show the same consistent accuracy improvement on the other dataset, such as Sintel. It would be great if the paper can demonstrate the same ablation study (e.g., Table 2 and 3) on the Sintel dataset. In my humble opinion, KITTI is a quite restricted setup with street scenes, perspective motion, and objects with rigid motion. Thus overfitting on the KITTI domain certainly improves the accuracy. On the other hand, Sintel shows much more diverse scenes with various visual effects. \n\n   Furthermore, the paper shows the analysis with RAFT only. Can this scheme generalize to other models, such as CRAFT? Considering the accuracy improvement on CRAFT, it seems a bit limited. I wonder if the paper can also provide the same ablation study for CRAFT. \n\n- Accuracy after the training with pseudo ground truth (i.e., before finetuning)\n\n  I am wondering how the accuracy is, just right after the training with pseudo ground truth for each iteration step. (I guess that Table 2 shows the accuracy after the finetuning step.) I wonder if the accuracy after the pseudo ground truth training improves along with the number of iteration steps. it would be great to include the numbers in Table 2 as a separate column. \n\n- Overlap between the unlabeled data and the validation set\n\n  The KITTI Optical Flow 2015 dataset (which has ground truth labels) is actually included in the KITTI raw dataset. I am wondering if those images are excluded when during the training with pseudo ground truth.\n\n- Contrastive loss on occluded region\n\n   I wonder if the contrastive loss is also applied on an occluded region (or what are the \"valid pixels\" in the text on the page 6?). The contrastive loss shows its benefits but might hurt the accuracy when the loss is applied on occluded pixels because their corresponding pixels don't exist, so the loss will be applied to mismatching pixels. \n\n- Final loss\n  \n  What is the final loss applied to the model? Is it a weighted sum of $L_q$ and $L_{CT}$? (eq.1 and eq. 3?)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "i_jQtLP8Cx", "reviewer": "Reviewer_bSsw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes CLIP-FLOW, which is a method to improve the performance of optical flow models under the semi-supervised learning setting. The proposed method consists of three orthogonal parts: 1) Semi-supervised Contrastive Flow: an additional contrastive loss for training optical flow models; 2) Coordinate Encoding: an additional 2D-coordinate map of each pixel as two additional channels to the original input frames; 3) Semi-supervised iterative training on unlabeled real data by pseudo-labeling. They show on KITTI 2015 dataset the proposed process can improve RAFT and CRAFT models, and achieve a second place on the benchmark.", "summary_of_the_review": "Overall I think the proposed method makes sense and the performance on KITTI 2015 is good. But I am mostly concerned about the limited ablation of the proposed model.\n\n----\nAfter rebuttal comments:\nThanks the authors for carrying out the rebuttal. But my concerns on limited ablation are not fully addressed:\n1) I think RAFT-CE-CF and CRAFT-CE-CF evaluation is still necessary for validating the effectiveness of CE-CF alone.\n2) The necessity of fine-tuning step on $D_R^{\\text{tr}}$ should be validated.\nand for \"shows significant improvement for the synthetic dataset as well\", I was wondering if the final model, rather than only RAFT-CF-CE, also has significant improvement for the synthetic dataset?", "strength_and_weaknesses": "Strength:\n- The overall writing is clear and easy to understand.\n- The related work section covers quite a complete set of recent works on optical flow, semi-supervised and representation learning.\n- The proposed method is simple and achieves good results in a semi-supervised setting.\n\nWeaknesses:\n- The iterative algorithm is not quite clear:\n  - For input $\\phi_\\text{our}^0$, should it be \"Ours ( $\\phi_\\text{bs}$+ pseudo labeling + contrastive flow loss), ...\"?\n  - Why the 0-th round pseudo-labels are generated by $\\phi_\\text{bs}$ rather than $\\phi_\\text{our}^0$? Since the assumption is that $\\phi_\\text{our}^0$ is better than $\\phi_\\text{bs}$?\n- \"shows significant improvement for synthetic dataset as well\" - it seems I could not find relevant experiments in the paper to back this up.\n- The ablation is not quite complete: there are three orthogonal parts in the proposed method, and CE and CF can be applied to RAFT and CRAFT without the pseudo-label re-training part. Wondering how the performances are for RAFT-CE-CF and CRAFT-CE-CF on KT15 (without PL)? \n  - And for the ablation experiments in table 3 and figure 4, it would be nice to show the performance after conducting the same 3-round iterative training process.\n  - Is the fine-tuning step on $D_R^{\\text{tr}}$ necessary?\n  - How to choose the $S$ for the total training steps on $D_R^{\\text{u}}$?\n- Limitations are not discussed in the paper.\n- Will the authors release code upon acceptance?\n\nMinor:\n- missing bracket on the line right above 3.3: ($\\mathcal{D}_R$ is marginal \n- in section 4.2, \"we finetune the updated RAFT-CF-CE-PL model from part A\": should this be RAFT-CF-CE?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UtDz48XlhC", "reviewer": "Reviewer_8YE1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Due to the difficulty of labeling GT optical flow data, this paper aims to explore how to better use of synthetic data and transfer the knowledge from the synthetic domain to the real domain. Thus, the two main contributions of this paper are 1) the iterative pseudo labeling and 2) supervised contrastive flow loss. In addition, this paper also discusses the impact of 2D positional encoding in the optical flow task.", "summary_of_the_review": "This work is a good attempt to explore the data usage given the difficulty of the optical flow, but its technique novelty is not very strong.", "strength_and_weaknesses": "Strength:\n\nThis work is a good attempt to explore the (synthetic and real) data usage given the difficulty of the optical flow labeling, and the experiments show good insight on the performance improvements we can obtain by pseudo labeling and contrastive loss. And I think the 2D positional encoding idea is interesting. \n\nWeaknesses:\n\nThe technique novelty of this paper is not very strong. Pseudo labeling and contrastive loss have been widely used recently, and the design of the semi-supervised iterative labeling training strategy (also algorithm 1) seems to over-complicate the approach. See more details in the below section.\n\nThe writing can be improved as I see some obvious typos in the paper. See some examples in the below section.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject