Towards Online Real-Time Memory-based Video Inpainting Transformers
Authors: Guillaume Thiry
Hao Tang
Radu Timofte
Luc Van Gool
Venue: Submitted to ICLR 2023 (2023)

Abstract: Video inpainting tasks have seen significant improvements in the past years with the rise of deep neural networks and, in particular, vision transformers. Although these models show promising reconstruction quality and temporal consistency, they are still unsuitable for live videos, one of the last steps to make them completely convincing and usable. The main limitations are that these state-of-the-art models inpaint using the whole video (offline processing) and show an insufficient frame rate. In our approach, we propose a framework to adapt existing inpainting transformers to these constraints by memorizing and refining redundant computations while maintaining a decent inpainting quality. Using this framework with some of the most recent inpainting models, we show great online results with a consistent throughput above 20 frames per second. Code and pretrained models will be made available upon acceptance.

Under review as a conference paper at ICLR 2023
TOWARDS ONLINE REAL-TIME MEMORY-BASED VIDEO INPAINTING TRANSFORMERS
Anonymous authors Paper under double-blind review
ABSTRACT
Video inpainting tasks have seen significant improvements in the past years with the rise of deep neural networks and, in particular, vision transformers. Although these models show promising reconstruction quality and temporal consistency, they are still unsuitable for live videos, one of the last steps to make them com- pletely convincing and usable. The main limitations are that these state-of-the-ar models inpaint using the whole video (offline processing) and show an insufficient frame rate. In approach, we propose framework to adapt existing inpainting transformers to these constraints by memorizing and refining redundant compu- tations while maintaining decent inpainting quality. Using this framework with some of the most recent inpainting models, we show great online results with a consistent throughput above 20 frames per second. Code and pretrained models will be made available upon acceptance.
INTRODUCTION
Video inpainting is the task of filling missing regions in a video with plausible and coherent content. can be seen as an extension of the more known image inpainting but with the extra temporal dimension, bringing new challenges. A good video inpainting can be used for various applications, such as object removal (Ebdelli et al., 2015), video restoration (Lee et (209)) or video completion (Chang et al. 2019b).
To be convineing, video inpainting must spatially coherent, that is, the content filled in each frame fits with the rest of the image. It must also be temporally coherent, meaning that the video is smooth and without artifacts when being played. Models leveraging a deep learning approach (Kim)
al. 2019 Oh et al. 2019 Chang et al. 2019a) have made significant progress recently, especially on the temporal consistency that was lacking from more traditional methods (Wexler et al. 2007 Granados et al. 2012 Newson al. 2014). Among them, the transformers (Zeng et al. 2020
et al. 2021a) showed the best performance both in terms of quality and speed.
With more and more live content today (e.g. cultural and sport events, social media streaming), online and real-time video inpainting is necessary to deal with these new types of broadcast. Such techniques could also prove to be useful the augmented perception field. These models should be able to inpaint an ongoing video, with sufficient frame rate to be 'live", While a few previous works investigated this (Herling & Broll, 2014, Kari et al. 202T), none of the current state-of-the-ar approaches meet the criteria to be called either online or real-time, limiting the potential real-life use cases of this technology.
In this work, we propose a framework to adapt the most recent transformer- techniques of video inpainting to both online and real-time standards, with as little loss of quality as possible.
I. Online We explore the natural modifications to make any inpainting model work online. By doing that, we derive an online baseline trading off inpainting quality. The main drawback this approach is the frame rate, which is still too low for real time.
II. Memory
We then add memory and keep the successive results of these transformers, to reduce the number of calculations to do for the next frames. With that, we increase the number of frames per second by factor of 3, passing the real-time threshold at the cost of yet another quality drop.

---

Under review as a conference paper at ICLR 2023
III. Refined Finally, we refine the memory based framework to temper the loss of inpainting qual- ity while maintaining a real-time throughput. To do that, two models run side by side and commu- nicate together the live video goes The first one inpaints the frames in real-time they come, using as much previous knowledge as it can have. Simultaneously, the second model reinpaints al- ready gone frames with more time and more care. It then communicates its results to the first model, giving it valuable information to use.
We demonstrate the proposed techniques (Online, Memory, Refined) on three of the most recent transformer-based models and achieve online real-time operating points when testing on the usual video inpainting tasks and datasets.
The remainder of the paper is structured as follows. In Section 2 and current research on video inpainting. Then, we detail our online video inpainting models in Section 3 Finally, we report all our results and discuss them in Section
we give an overview of the former
RELATED WORK
2.1 IMAGE INPAINTING
The first works image inpainting were proposed decades ago and were relying on the of known textures to fill the missing content (Bertalmio et al.] 2000 Efros & Leung 1999 Efros & Freeman 2001). These textures were sampled from other areas of the image thanks to patches that were matched with the corrupted area with a similarity score. Variations around this approach have then been proposed (Hays & Efros, 2007) including PatchMatch (Barnes et al.) 2009), using an approximation of the patch matching to obtain tool fast enough for commercial uses.
With the development of more complex models such as convolutional neural networks (CNN) Krizhevsky et al. (012)), recurrent neural networks (RNN) (He et al. (016) or generative adver- sarial networks (GAN) (Goodfellow et al. 2014), recent image inpainting works have focused on the use of deep neural networks (c.g. encoders) trained with adversarial losses, with great results (lizuka et al. 2017: Pathak et al. 2016 Yu et al. 2018 Nazeri et al. 2019).
2.2 TRADITIONAL VIDEO INPAINTING
As for image inpainting, the first models proposed were handcrafted, relying on more traditional image manipulation techniques. Most models adopted a similar patch-based approach which the video was cut into spatial-tempora temporal patches. A score was also computed between the patches to fill the missing content with information from similar patches (Wexler et al.) 2004 2007, Newson et al. 2014| Patwardhan et al. 2005). To later deal with videos having more complex movements, refinements have been proposed, for example, introducing the use of flows to help the patch-based inpainting (Huang et al. (016). Other techniques not using patches were also proposed, using, for instance, image transformations to align the frames together (Granados et al. 2012).
2.3
DEEP VIDEO INPAINTING
Deep learning brought significant improvement in the quality of video inpainting in the same way it did with image inpainting. Deep neural networks have been leveraged in various ways to inpaint video (Ouyang et al. 2022 Lee et al.) 2019 Ke et al., 2021), with most of them belonging to three main categories, as described by Zou et al. (2021).
One way to video inpaint is by employing encoder-decoder model with mix of 2D and 3D convolutions (Wang et al. 2019). VINet (Kim et al., 2019) was one of the first deep neural models capable of competing with state-of-the-art models using traditional techniques (Huang et al. 2016). Improvements were later made by adding gated convolutions (Chang et al., (219a) and designing a video-specific GAN loss called T-PatchGAN (Chang et al. 2019b) used in many works since then ZZng et al., 2020 Liu et al. 2021b). This strategy is, however, impeded by the heavy calculations that 3D convolutions bring.
Other models perform video inpainting by utilizing the optical flows of the videos (Xu et al. 2019
Gao et al. 2020). Forward and backward flows are first computed for the known part of the video
2

---

Under review as a conference paper at ICLR 2023
with traditional techniques such as FlowNet (Dosovitskiy al. 2013 1lg et al. 2017) before being completed by the models. Missing content then filled by propagating known pixels through the obtained flows. This approach shows great results, especially at the border of the missing area, which is completed really smoothly. The main drawback of these models is the speed, with a throughput far from real-time because of the flow estimation and completion.
A final category encompasses the so-called attention-based models. While the earliest ones still leverage attention mechanisms in a traditional manner (Oh et al. recent and powerful ones all rely on transformers (Vaswani et al. 2017 Dosovitskiy et al. 2021
2019 Li al.) 2020), the most
Liu et al. 2021c) with different ways of splitting the frames into patches. While STTN (Zeng al. 2020) uses patches of different sizes, one for each head of the attention mechanism, DSTT (Liu et al., 202Tb) divides the transformers into two categories (spatial transformers and temporal transformers) to mix the patches differently. With FuseFormer (Liu et al., 2021a), the idea of having the patches overlapping to add more consistency gives even better results.
Finally, some works try to be more transverse and merge together different approaches. This is the case of TSAM (Zou et al.) 202T) which combines 3D convolutions with flows. The current state-of-the-art work, E2FGVI et al. 2022), is also following this approach by leveraging both FuseFormer-Iie like filling with flows propagation.
2.4 ONLINE VIDEO INPAINTING
To our best knowledge, only two works seriously tried to tackle the challenge of online video in- painting. They both focus on object removal as part of an entire end-to-end framework also including object segmentation and tracking. The first one, PixMix (Herling & Broll 2014), is more traditional and relies on pixel-wise homography mapping. which can only be used for specific movements of the camera and the object. The second one, TransforMR (Kari et al. 2021), is more recent and it adapts two of the early deep inpainting models VINet (Kim et 2019) and LGTSM (Chang et al.) 2019a) without being able to get both quality and throughput simultaneously: one is fast but gives a quite poor result, the other is slow but inpaints better. Unfortunately, none of these two works actually provide quantitative results compare with.
3
THE PROPOSED MODELS
3.1 PROBLEM FORMULATION AND EXISTING APPROACH
A video inpainting task is given by a set of consecutive corrupted frames X := [X N and corresponding masks M := {M..... My with N the length of the video. Each pixel of a mask M, has either the value "0" if the true pixel is known or "1" the value missing. The goal of video inpainting is to use X and M to reconstruct the ground truth Y == {Yi,..., Yn J. For the training, as and Y are rarely known together (e.g., in object removal), X usually obtained in practice by corrupting perfectly normal video Y using an arbitrary mask M: X = @ (1 M) with C being an semment-wise multiplication.
Recent attention- based techniques aim at completing given frame by using the information in both short-term context (neighboring frames) and long-term context (reference frames). This enables a good temporal consistency while being able find information further away in the video, which is for example useful when the mask is moving too slowly for the neighboring frames to be helpful. More specifically, all frames in a temporal window of radius k centered around frame f are com- pleted simultaneously using all the frames of the window, on top of reference frames sampled in the whole video at a sampling rate r:
-
where j and X/+* the reconstructed and original frames in window from frame k to frame f + k, X, is the set of frames from the whole video sampled at rate r, and M is the inpainting model. An illustrated example is given in Figure (a).
These frames are then encoded and cut into patches in different ways. depending on the model chosen (Liu al. 2021a b. Zeng et al. 2020). They are then completed using self-attention
3

---

Under review as a conference paper at ICLR 2023
Noighboring frames
X
Reference frames
X
1
Ailo
15
18
19
21
22
23
25
30
40
50
14
15
16
22
25
Inpeinted window %
9/2
(a) Offline model
(b) Online model
Figure 1: Original inpainting model and its natural online adaptation. (a) The model inpaints
window centered around f 20 with radius k Reference frames sampled rate 10 are added as input of the model. (b) In online inpainting, we only see the past frames. To inpaint the frame 18, we use a window and sampled frames from the past. The whole window still predicted but only the last frame is effectively used.
framework composed of consecutive transformer blocks. The transformers follow the same general pipeline as the original vision transformer (Dosovitskiy et al. (021)) with slight modifications for each model. For instance, the final weedwffrward layer replaced by a custom Fusion- cession-feed-fiee feed-forward layer in FuseFormer (Liu et al., 2021a). One model is also using additional flow-based module (Li
2022
help the transformers completion.
3.2 CHALLENGES OF ONLINE INPAINTING
Inpainting a video in real-time raises three main issues regarding the current state-of-the-ee the-art models:
1. A frame must be inpainted using only information from the frames before because cannot "see" in the future.
2. A new frame must be inpainted as soon as possible when received by the model, therefore we cannot wait for several frames inpaint them together.
3. A sufficient throughput is required to be able to call that "real time". We fix the value of 20 frames per second (FPS) as our goal here.
3.3 ONLINE VIDEO INPAINTING TRANSFORMERS
Given these new constraints, a drop in the quality of video inpainting is expected and partly in- evitable. For that reason, important to define proper baseline for online inpainting that can fairly assess our approach. Comparing an online model with a regular all-seeing one would be unfair as they do not process the same information and are meant for the same usage.
Taking into account the first two constraints, straightforward baseline can be derived from any of the existing inpainting models. Frames are inpainted one by one, not anymore by blocks, and given frame is inpainted only with information from the past. Using the same notations, this model described in Equation equation 2 and Figure (b).
- -
While drop in inpainting quality is expected, this model also shows very poor frame rate (see Tables D and 2]. This explained by the fact that the model still inpaints a whole window of frames in classical video inpainting, but only the prediction for the last frame is used here given constraints and 2 (Figure [T. Starting from this baseline, we propose two models significantly increasing the frame rate while staying close as possible this one in terms of inpainting quality.

---

Under review as a conference paper at ICLR 2023
Impaininnge memory
10
13
14
Trandiiee
14
14
Transhore
18 +
Transker
18
18
18
completty
(a) Without memory
[b] With memory
Figure 2: Transformers in the baseline and memory-based models. (a) Without memory, the baseline model processes all the frames in each transformer, making it quadratically complex. (b) When the memory of the previous inpaintings is kept, only the new frame (18) needs to be computed, while the transformers can still use the other frames (0 to 17) as context. After each transformer, the new result is saved for later. Each frame is saved in the memory as much times as there are transformers, each value being different. Following Equation equation 3 we have here f = 18, 8 and 10.
3.4 MEMORY-BASER ONLINE VIDEO INPAINTING TRANSFORMERS
This model addresses the frame rate problem by narrowing the prediction pipeline to only one frame, the last one available at a given time. The query vector in the attention mechanism is changed to only contain patches from this last frame, avoiding any unnecessary prediction of the other frames. Meanwhile, the key and value vectors must still represent all the frames selected for the prediction (ncighboring and reference frames) to keep good impainting. This raises a problem the original inpainting model composed of several chained transformers (e.g., 8 transformers in Fuseformer STTN), in which the results from the first serve input for the second one construct the query, key and value vectors, and so on with the next ones (Figure 2 (a)).
To deal with that, we propose save in memory the successive outputs of these transformers for each predicted frame and reuse them later for the next prediction. This approach effectively transforms a quadratic attention computation frames predict n frames) into a linear one (n frames predict frame) as shown in Figure 2 (b). Moreover, the rest of the prediction pipeline now forwards one frame only as the other frames are only used by the transformers. This enables gain of time on other operations such as convolutional encoding or patch splitting, which is not negligible in a model like Fuseformer using overlapping patches. The resulting equation is:
8, = MM (/:
with M' the new model, M the transformers memory. s and r are the memory span and sampling rate: frames from index f f- to f - 1, well previous frames sampled rate r will be used to inpaint frame f. A visual example can be found in Figures (b) 3nd
As detailed in Tables T and this approach greatly improves the frame rate, leading to values admissible as "real time". i.e. above 20 FPS. The main drawback of this model is the significant reduction in video quality, measured all 4 metrics.
3.5 REFINED MEMORY- BASED ONLINE VIDEO INPAINTING TRANSFORMERS
Investigations made on the previous model show that if frame is poorly inpainted, the results will still be saved in memory, never recomputed, and reused later for the following frames, maintaining poor inpainting through the whole video. This is even worsened by the fact that the inpainting results created independently, only frame is predicted at time. Such problem doesn't happen in the baseline model because a) frames are inpainted together in windows and not independently and b) poor inpainting result for given frame will not be used for the following inpaintings.
5

---

Under review as a conference paper at ICLR 2023
Online memory MC
Xax
22
memory
MB Mila
Xxa
10
12
18
15
16
18
19
21
02
hame
Online Impairter
Impainter
Refined memory
S822 Rigo
Reetining Impaiiteo
memory
With refined memory
Figure 3: Memory-based and refined models. (a) Thanks to inpainting memory the last seen frames, the new frame is the only to be computed but the inpainting still benefits from this previous context. Following Equation equation [3] we have here f = 18, = 5, and r = 10. (b) In this model, the online inpainter still uses memory of the last frames it inpainted, but it also receives information from the inpaintings of the second model. This refining inpainter performs a slower but better inpainting the video at the same speed, so that the refined memory is always relevant to the online inpainter. Following Equation equation we have here 18. 14 (for this example), s' 3. and r' = 10 (tunable parameters).
it can work directly windows. If tuned correctly, both models process
This improved model addresses the issue by combining both worlds of high FPS and good quality with two inpainting modules working side by side (Figure (b)). The first one, called the Online Inpainter, is very similar to the first model as it inpaints previous inpaintings. As computes attention linearly and not quadratically in the transformers, this inpainter displays real-time performances. The second module is called the Refining Inpainter and it works on already gone frames. Like the original approach, it inpaints frames with small windows to get the best inpainting quality allowed by the model. However, these inpainted frames are not meant to be displayed as result have been already shown, and the live video is now further away. Instead, these impaintings are only created be used by the Online Inpainter, whose quality will increase as result. The underlying idea is that if fine-tuned properly, both models will progress approximately the same speed, making the refined inpaintings relevant for the Online Inpainter. It will therefore keep its real-time inpainting speed, but will be provided with much better previous inpaintings, increasing its own inpainting quality. For maximum speed, we implement this model using two GPUs, each of them carrying one of the inpainting modules. As detailed in Equation equation 4 found the best input to be composed of neighboring frames from the Online Inpainter's memory as well as neighboring and reference frames from the Refined Inpainter's one.
frame
time reusing results from
y, = M" (X,, M/=1 M; M,
(4)
The new model M uses the same memory M with span and now also the refined memory M, with span s' and sampling rate r'. I the index of the last inpainted frame available this memory.
EXPERIMENTS AND RESULTS
4.1 BACKBONES
The models that we have implemented are online adaptations of state-of-the-art video inpainting transformers. There are three in total, and we will refer them as "backbones". The first one the Decoupled Spatial-Tempal decoupled and spatially same frame while the others try find relevant content in other frames but at the same spot in the frame. The second model adapted in our work is the FuseFormer (Liu et al., 2021a), which uses traditional vision transformers blended together to enable a better reconstruction of the missing area. The last backbone the End- En-End Framework for Flow-Guided Video Inpainting (R2FGVI) (Li al. 2022). and it combines
Transformer (DSTT) (Liu et al. (021b) which interweaves temporally- transformers. The first ones focus on finding information in the
overlapping patches of the vidco. These patches
then

---

Under review as a conference paper at ICLR 2023
Table 1:
Results the video reconstruction task for the DAVIS dataset.
Backbone
Model
PSNR
SSIM f
VFID 4
Ewerp (x 10 3)
FPS f
Offline
31.25
0.959
0.137
0.152
19.2
DSTT
+ Online
30.63
0.954
0.144
0.153
12.4
Memory
30,20
0.951
0.142
0.154
39.5
+ Refined
30.43
0.953
0.142
0.154
27.1
Offline
31.74
0.962
0.126
0.152
10.8
FuseFormer
+ Online
31.11
0.958
0.135
0.153
7.1
+ Memory
30.49
0.953
0.142
0.153
31.0
+ Refined
30.75
0.956
0.139
0.153
21.3
Offline
32.08
0.964
0.117
0.148
8.8
E2FGVI
Online
31.20
0.959
0.128
0.150
5.6
+ Memory
30.54
0.954
0.129
0.151
11,4
+ Refined
30.92
0.957
0.129
0.153
9.4
Table 2: Results on the video reconstruction task for the YouTube- VOS dataset.
Backbone
Model
PSNR 1
SSIM f
VFID +
Ewerp (x10 2)
FPS t
Offline
31.98
0.960
0.054
0.099
17.9
DSTT
Online
31.63
0.958
0.059
0.099
11.7
+ Memory
31.28
0.955
0.058
0.100
34.9
+ Refined
31.54
0.957
0.058
0.099
21.2
Offline
32.34
0.962
0.053
0.098
10.3
FuseFormer
+ Online
32.01
0.960
0.057
0.099
6.6
Memory
31.56
0.957
0.057
0.099
26.6
+ Refined
31.87
0.959
0.057
0.099
16.9
Offline
32,65
0.963
0,049
0.096
8.5
E2FGVI
+ Online
32.15
0.961
0,053
0,097
5.1
+ Memory
31.61
0.958
0.052
0.097
10.2
+ Refined
31.98
0.960
0.052
0.099
8.3
focal vision transformers with flow-based approach. Bidirectional flows are computed to propagate content to the border of the missing region while the remaining part is completed by the transformers.
4.2 DATASETS
We compare the different approaches on two widely used datasets for video inpainting. They are composed of small videos with rather simple motions, more complex videos still being out of the scope current state-of-the-ar models. YouTube- VOS (Xu et al. 2018) is composed of 4519 videos of about 150 frames, while DAVIS (Perazzi et al. 2016 Pont- liit--Tusee Tuset al. 2017) contains 150 videos of about 120 frames. As we aim to adapt already existing models with our framework, we do not conduct supplementary training and reuse the weights of the pre-trained models. Therefore, we only use the test sets of these datasets. Following previous works (Liu et al., 2021a b), we first quantitatively evaluate these videos on reconstruction task using the same stationary masks as FuseFormer (Liu et al., 2021a). We then visualize some object removal inpaintings, on the DAVIS dataset. The lack of ground truth for this last task makes quantitative analysis impossible.
4.3 QUANTITATIVE RESULTS
To assess quantitatively our models, we use metrics frequently used in previous works (Zeng al. 2020]: PSNR, SSIM (Wang et al.) 2004), VFID (Wang et al., 2018) and Ewerp (Lai et al.) 2018). More specifically, PSNR (Peak Signal to Noise Ratio) and SSIM (Structural Similarity) are widely used techniques for assessing the quality of reconstruction of an image compared to its original ver- sion. VFID (Video- based Fréchet Inception Distance) assesses the visual quality of the whole video by calculating its closeness with natural videos using pre-trained I3D model (Carreira & Zisser- man (017). Ewarp (Flow warping error) is a more recently used metric temporal consistency and vary from publication to another it depends the flow model used. The frame rate is

---

Under review as a conference paper at ICLR 2023
measured on RTX 2080 Ti GPUs. Except for the refined memory-based model, which runs on two GPUs (one for each inpainter), all models run on one GPU only.
As the ground truth video is required for all these metrics, only the reconstruction task is possible here: stationary masks are applied to normal videos, and the model tries to reconstruct the original video close as possible. We compare our three online models, as well the original offline model, using inputs of similar sizes (same number of neighboring and reference frames). The results for this task are reported in Table for DAVIS and Table 2 for YouTube-VOS.
for
The first thing we can observe in both tables is the lead of the offiin model, able to use information further away in the video to inpaint given frame. This shows the need to establish proper baseline as online video inpainting proves to be substantially harder than regular video inpainting. Then, we observe recurrent trends among the online models, regardless of the backbone. First, the basic online model is the best in terms of quality but also the slower. This is because every frame is inpainted by processing a whole window of previous frames. This gives great inpainting, however only the inpainting of the last frame can really be used, making it too slow.
Then, using our memory-based technique, we observe significant gain the frame rate, with a factor of 2 for E2FGVI and even a factor of 3 for DSTT and FuseFormer. This is because after each transformer, the inpainting results are saved in memory so that it not necessary to recompute them later for a future frame. This causes, however, a loss in quality the frames are not really inpainted together anymore: one frame is inpainted at a time, and even it uses information from past frames, it is not the same inpainting them all together at once.
The last model, using the memory approach with refinement, tries to combine the best of both worlds. It still uses an inpainting memory to skip extra calculations, so the frame rate is still greatly improved compared to the baseline model (with a factor of 2 on DSTT and FuseFormer). And because it also reinpaints frames together
also reduce most of the loss observed with memory only. We should point out that it is not unfair to use two GPUs here, because the Online Inpainter alone is fast enough to inpaint the frames in real time, something a slower model cannot do, even with more GPUs (only one can inpaint a given frame).
the side reuse these results,
While the quality gains and losses are similar compared to other backbones, our models coupled with E2FGVI show less pronounced results on the frame rate. The reason is that our memory approach saves a lot of time only on the attention calculations inside the transformers, as shown in Figure 2 For DSTT and FuseFormer, these operations represent an overwhelming part of the time spent, while this less true for E2FGVI: the calculation of flows takes time, as well as the calculation of the windows for the focal attention. These parts may surely be optimized well, but this lies outside the scope of our framework at the moment.
4.4 MODELS COMPARISON
can be difficult to fairly compare two models if we include the frame rate as a criterion as there is manifest trade-off between speed and quality when changing the size of the input window: the more frames provided as context, the better the inpainting but also slower. To take that into account, evaluated them using different input sizes and reported the results in a quality/speed plot, shown in Figure The PSNR is chosen as a measure of the inpainting quality here.
With that plot, we can observe that the different models belong to different domains regardless of the input size: the baseline online model will never be as fast as the cemoryybbased one, even with really small inputs, and vice-versa for the quality. One model will then be preferable to another, depending on the need of the user. As expected, the baseline model is the best for quality, the memory- based gives the best frame rate, and the refined memory- fits in the middle. We can nevertheless note that for similar PSNR values, the refined model always shows higher frame rates compared to the baseline model, and is consistently above 20 FPS for both DSTT and FuseFormer backbones. As already mentioned, the results for E2FGVI are less outstanding and fail to meet the real-time speed.
4.5 ABLATION STUDY
In Table 3 we propose an ablation study of the previously chosen model to see the importance of each component in the input. As recalled by Equation equation 4 the refined memory-based
8

---

Under review as a conference paper at ICLR 2023
Online
Online Meroery
Online Memory + Retined
(4) DSTT
(6) FuseFarmer
E2FaVI
02
41
30.75
31.50
30.35
00.00
75
31.00
Figure PSNR/FPS operating points on each backbone, using different input sizes.
Table 3: Ablation study on the input of our proposed model
Neighboring Refined Neighboring Online Reference Refined PSNR
SSIM
FPS
29.94
0.948
29.5
30.40
0.952
23.3
30.58
0.954
23.0
30.87
0.956
19.7
model takes as imput a) neighboring frames from the online memory, b) neighboring frames from the refined memory, and c) reference frames from the refined memory. As confirmed by the ablation, all components are helpful to inpaint the video. Reference frames seem especially important and cannot be simply replaced by more neighboring frames. Moreover, neighboring frames from the online memory are also really useful, certainly because they are really close to the current frame. More details about the parameter tuning of the model can be found in Annex of this paper.
4.6 QUALITATIVE VISUALIZATION
As one of the most promising in our study, we pick the refined memory- based model coupled with FuseFormer, having both good quality and frame rate. We perform an object removal task on the DAVIS dataset and report visual results for some of the videos in Figure
5 DISCUSSION AND CONCLUSION
In this work, we presented a reliable method to adapt existing inpainting transformers to online and real-time standards while tempering quality loss. Some issues still remain, such as cases when little to no information is available to predict content, for example for the very first frames of the video as discussed in Appendix ??. Moreover, our framework does not work yet with non-transformer models, limiting the adaptation of future techniques.
Nonetheless, we hope that by proving that both quality and speed could be met at the same time in video inpainting, paved the way for more research in that direction. Eventually, stronger models will make online video inpainting accessible to the public, changing the way we produce live content.
uf der Siegerselte. ewz.ch ewz.ch/jobss
der Slegersehe. ewz.ch/ Jobs
Figure 5: Visualization of some object removals performed 20 FPS.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. Patchmatch: A random- ized correspondence algorithm for structural image editing. ACM Trans. Graph., 2009.
Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester. Image inpainting. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, 2000.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Ya-Liang Chang, Zhe Yu Liu, Kuan- Ying Lee, and Winston Hsu. Learnable gated temporal shift module for deep video inpainting. arXiv preprint arXiv:1907.01131 2019a.
Ya-Liang Chang, Zhe Yu Liu, Kuan- Kuan-Ying Lee, and Winston Hsu. Free-form video inpainting with 3d gated convolution and temporal patchgan. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019b.
Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg. Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In Proceedings of the IEEE international conference on computer vision, 2015.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
Mounira Ebdelli, Olivier Le Meur, and Christine Guillemot. Video inpainting with short-term win- dows: application object removal and error concealment. IEEE Transactions on Image Pro- cessing, 2015.
Alexei A Efros and William T Freeman. Image quilting for texture synthesis and transfer. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, 2001.
Alexei A Efros and Thomas K Leung. Texture synthesis by con-paraeetric sampling. Proceed- ings of the seventh IEEE international conference on computer vision, 1999.
Chen Gao, Ayush Saraf, Jia-Bin Huang, and Johannes Kopf. Flow-edge guided video completion. In European Conference on Computer Vision. Springer, 2020.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 2014.
Miguel Granados, Kwang In Kim, James Tompkin, Jan Kautz, and Christian Theobalt. Background inpainting for videos with dynamic objects and free-moving camera. In European Conference on Computer Vision. Springer, 2012.
James Hays and Alexei A Efros. Scene completion using millions of photographs. ACM Transac- tions on Graphics (ToG), 2007.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
Jan Herling and Wolfgang Broll. High-quality real-time video inpainting with pixmix. IEEE Trans- actions on Visualization and Computer Graphics, 2014.
Jia-Bin Huang, Sing Bing Kang. Narendra Ahuja, and Johannes Kopf. Temporally coherent com- pletion of dynamic video. ACM Transactions Graphics (TOG), 2016.
10

---

Under review as a conference paper at ICLR 2023
Satoshi lizuka, Edgar completion. ACM Transactions on Graphics (ToG), 2017.
and Hiroshi Ishikawa. Globally and locally consistent image
Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.
Mohamed Kari, Tobias Grosse- Puppee-lup Luis Falconeri Coelho, Andreas Rene Fender, David Bethge, Reinhard Schütte, and Christian Holz. Transformr: Pose-aware object substitution for composing alternate mixed realities. In 2021 IEEE International Symposium on Mixed and Aug- mented Reality (ISMAR), 2021.
Lei Ke, Yu-We Lu-Wing Tai, and Chi-Keung Tang. Ocelusion-aware video object inpainting. Proceed- ings of the IEEE/CVF International Conference on Computer Vision, 2021.
Dahun Kim, Sanghyun Woo, Joon- Young Lee, and In So Kweon. Deep video inpainting. In Proceed- ings of the IEEE//VF Conference on Computer Vision and Pattern Recognition, pp. 5792-5801, 2019.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 2012.
Wei-Sheng Lai, Jia-Bin Huang. Oliver Wang, Eli Shechtman, Ersin Yumer, and Ming- Hsuan Yang. Learning blind video temporal consistency. In Proceedings of the European conference com- puter vision (ECCV), 2018.
Sungho Lee, Seoung Wug Oh, Yaun Won, and Seon Joo Kim. Copy-and-paase networks for deep video inpainting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019.
Ang Li, Shanshan Zhao, Xingjun Ma, Mingming Gong, Jianzhong Qi, Rui Zhang, Dacheng Tao, and Ramamohanarao Kotagiri. Short-term and long-term context aggregation network for video inpainting. In European Conference on Computer Vision. Springer, 2020.
Zhen Li, Cheng-Ze Lu, Jianhua Qin, Chun-Le Guo, and Ming-Ming Cheng. Towards end-to- end framework for flow-guided video inpainting. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Rui Liu, Hanming Deng, Yangyi Huang, Xiaoyu Shi, Lewei La, Wenxiu Sun, Xiaogang Wang, Jifeng Dai, and Hongsheng Li. Fuseformer: Fusing fine- anne-grined information in transformers for video inpainting. In International Conference on Computer Vision (ICCV), 202
Rui Liu, Hamming Deng, Yangyi Huang, Xiaoyu Shi, Lewei Lu, Wenxiu Sun, Xiaogang Wang, and Li Hongsheng. Decoupled spatial-temporal transformer for video inpainting. arXiv preprint 2021b.
arXiv:2104.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021c
Kamyar Nazeri, Eric Ng. Tony Joseph, Faisal Z Qureshi, and Mehran Ebrahimi. Edgeconnect:
Generative image inpainting with adversarial edge learning. arXiv preprint 2019.
Alasdair Newson, Andrés Almansa, Matthieu Fradet, Yann Gousseau, and Patrick Pérez. Video inpainting of complex scenes. Siam journal imaging sciences, 2014.
Scoung Wug Oh, Sungho Lee, Joon- Joon-Young Lee, and Seon Joo Kim. Onion-peel networks for deep video completion. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, 2019.
Hao Ouyang, Tengfei Wang, and Qifeng Chen. Internal video inpainting by implicit langerange propagation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.

---

Under review as a conference paper at ICLR 2023
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
Kedar A Patwardhan, Guillermo Sapiro, and Marcelo Bertalmio. Video inpainting of occluding and occluded objects. In IEEE International Conference on Image Processing 2005, 2005.
F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine Sorkine-Horunn.. A benchmark dataset and evaluation methodology for video object segmentation. In Computer Vi- sion and Pattern Recognition, 2016.
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alexander Sorkine-Hornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv:1704.00675. 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 2017.
Chuan Wang, Haibin Huang, Xiaoguang Han, and Jue Wang- Video inpainting by jointly learn- ing temporal structure and spatial details. In Proceedings of the AAAI Conference on Artificial Intelligence, 2019.
Ting-Chun Wang, Ming- Yu Liu, Jun- Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-to-video synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 2004.
Yonatan Wexler, Eli Shechtman, and Michal Irani. Space-time video completion. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Puttern Recognition (CVPR), 2004.
Yonatan Wexler, Eli Shechtman, and Michal Irani. Space-time completion of video. IEEE Transac- tions on pattern analysis and machine intelligence, 2007.
Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, and Thomas Huang. Youtube-vor vos: A large-scale video object segmentation benchmark. arXiv preprint (rXiv:/809.0332. 2018.
Rui Xu, Xiaoxiao Li, Bolei Zhou, and Chen Change Loy. Deep flow-guided video inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018.
Yanhong Zeng, Jianlong Fu, and Hongyang Chao. Learning joint -entiall transforma- tions for video inpainting. In The Proceedings of the European Conference on Computer Vision (ECCV), 2020.
Xueyan Zou, Linjie Yang. Ding Liu, and Yong Jae Lee. Progressive temporal feature alignment network for video inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
12


Reviews:
[{"review_id": "pemecyhm82", "reviewer": "Reviewer_WiHT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work presents a video inpainting method based on transformers by memorizing and refining redundant computations while obtaining a decent inpainting quality. Experimental results show that the developed method can achieve online results with 20 frames per second.", "summary_of_the_review": "This work has its merits. The motivations of this work are clearly presented, and the authors also present methods to address the target online real-time and memory-based video inpainting transformers. But this work also has several issues to be addressed; please refer to the weaknesses. ", "strength_and_weaknesses": "Strengths:\n1.\t This method presents an online, memory, and refined video inpainting method.\n2.\t Experimental results show the effectiveness of the developed method.\nWeaknesses:\n1.\tThe motivations are litter confused. As we all know, the memory mechanism and the transformer mechanism are time-consuming, but this work aims to achieve a real-time video inpainting performance with such two mechanisms. \n2.\tIn the Introduction section, the authors are suggested to summarize the technical novelties of this method.\n3.\tIt seems that the transformer mechanism has been widely used for video restoration. The authors should compare the developed method with them.\n4.\tIt is possible to extend the developed online, real-time memory-based transformers in this work for other video restoration tasks?\n5.\tIn Table 1 and Table 2, the authors are suggested to compare the model size of different methods.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "ojV1OVpeoY", "reviewer": "Reviewer_g9oW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new method for real time video inpainting. More spefically, the authors base their method on recent inpainting models and use memory and re-organize calculation order to make them more efficient.", "summary_of_the_review": "This paper proposes a new method for real time video inpainting. It addresses practical issues which preventing a real time usage and propose to use memory and refinement steps to accelerate recent inpainting models.\n\nThe proposed method is technically sound but seems lack of enough novelty.\nAnd the effectiveness has not been thoroughly explored: visual results and other quantitative metrics would be better.", "strength_and_weaknesses": "Strengths\n1. The proposed problem is of great practical value, which have long been ignored in previous methods.\n2. The proposed memory and updating mechanics are very reasonable and practical.\n\nWeaknesses\n1. The main concern is the novelty part. Althought it is very important and very effective, the proposed memory, calculation order and memory refinement steps seem to be technical details, rather than novel contributions.\nMaybe a new network design plus above mentioned memory mechanics would be a better and complete version, which can better highlight the value of above proposed modules.\n\n2. In Table 1 and 2, since the proposed method is definitely worse than offline version, it is hard to evaluate quantitative results. How to more convincably prove its effectiveness ?\n\n3. As a video processing task, and quantitative results cannot easily prove the method's effectiveness, I think more visual results are needed. However, only one is given in Fig. 5.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "oSIgZ9eltpG", "reviewer": "Reviewer_ApZX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to design a real-time video inpainting framework. Specifically, the authors propose to use two inpainters, i.e. an online inpainter and a refining inpainter, to achieve a better trade-off between speed and video quality. This paper shows experimental results on DAVIS and Youtube-VOS based on three baselines, i.e., DSTT, FuseFormer and E2FGVI. ", "summary_of_the_review": "This paper targets at real-time video inpaitning, which is a significant and interesting topic. However, the novelty of the proposed framework is limited (seems like a combination of existing video transformer networks), and the experiments are not convincing (see more details in Weaknesses). Therefore, I tend to reject this paper. If all the concerns in Weaknesses can be well-addressed, I will raise my rating. ", "strength_and_weaknesses": "### Strength\nThe task of real-time video inpainting is significant. \nThis paper is straightforward and easy to follow. \n\n### Weaknesses \n\n**Clarity**\n\nThe clarity of this paper needs grand improvements. Some important details are missing, for example, \n1. the authors claimed ‘real-time’ in the Abstract but without providing the device information and video resolution. \n2. In the caption of Figure 2, the authors should give a brief explanation of the ‘s’ when it first appears. \n3. The authors should provide the memory cost for a comprehensive comparison of the experiments.\n4. The authors didn’t provide video resolution in experiments. \n\n**Novelty**\nThe core contribution of this paper is to use two inpainters to improve inpainting results. This solution is quite more like an engineering solution and the novelty is limited. \n\n**Correctness** \n1. The authors only conduct experiments on three transformer networks. However, some CNN-based video inpainting approaches have achieved a better trade-off between speed and quality, e.g., VINet (Kim et al., 2019) and LGTSM (Chang et al., 2019a). The authors should report and compare the proposed approach with them.\n2. As mentioned in Section 4.2, the authors aim to adapt already existing models with the proposed framework. We found such a framework does not require a transformer-based approach. We suggest the authors adapt some CNN-based approaches with this two-inpainters framework to see the benefits. \n3. How to ensure the inpainting memory in Figure 3 will get better? If a video frame with a large missing region results in a bad video inpainting frame in the memory. Is it better to discard this frame to avoid error accumulation rather than keeping refine it by a refining inpainter? \n4. It’s not fair to report as ‘real-time’ on a 2080 Ti GPU, since this device is more powerful than most customer devices (e.g., phones, laptops).\n5. Since the target is video inpainting, we suggest the authors provide a video demo for evaluation. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "iosRKyP2HLz", "reviewer": "Reviewer_6qxK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed an online video inpainting framework by using a special caching method and a parallel model to balance the quality and speed. Extensive experiments are conducted to evaluate the efficiency and quality, and the proposed method outperform the baselines when being applied to online cases.", "summary_of_the_review": "The main concerns are,\n- The task is interesting but not practical. Without online mask generation, online video inpainting is meaningless.\n- The discussions of different types of video inpainting is not sufficient. The proposed methods can only be used to attention-based models. Optimization on the optical-flow / convolution / warping-based methods may be easier for improving the efficiency and quality at the same time. The authors may not be required to optimize them in this paper, but at least should compare the results in a similar way to E2FGVI.\n-  The evaluation may not be clear enough, and computation complexity should be reported. While comparing FPS, the input frame size should be reported.  ", "strength_and_weaknesses": "Strength:\n- The paper is well-written, and the models are well evaluated both quantitatively and qualitatively. \n- The idea of reusing precomputed frames in the attention layer to reduce the running time is innovative.\n- The proposed two-modules pipeline can further balance the speed and quality.\n\nWeaknesses:\n- Online video inpainting is not a well-defined task. Video inpainting requires the mask inputs, while in this work, all the masks are not predicted but form the ground truth. Then it is not proper to call it online video inpainting, since the masks are not generated along with the new frames. Ground truth masks are practical for offline video inpainting since it is a post-processing task. What is the real user cases of online video inpainting? Though interesting, the reviewer is still confused about the values of the task. \n- It seems the proposed method is designed for transformer-based / attention-based video inpainting work. However, for efficiency, flow-based or warping-based video inpainting may be more valuable. E2FGVI is time-consuming in computing the optical flow, making the proposed framework useless. The reviewer is curious about how those warping-based methods work in speed and quality, and whether we can turn to optimize the speed of convolutional-based model / optical-flow / homography-based model if we really want an online video inpainting model. The discussions related to it are not sufficient in this paper. \n- In Table 1 and 2, why the online model can be slower than offline model? Are they using the same window size?\n- For the refined model, is the window size smaller than the offline or baseline model? Why not display the results of the second refinement module directly? It's not very clear why we still need the first pipeline to save time given we already have an equal-fast module producing better results. Could we compare the results of refinement module only with the combined version?\n- Figure 3 is not clear to parse. Different colors indicate the frame categories but very confusing. \n- The reviewer cannot easily agree with the authors that using 2GPUs is a fair comparison. One can always use parallelization tricks to make the computation of attention more efficient. The metrics the authors need to add is the computation complexity, but not only the FPS. Given different input sizes and different GPU types, frame rate is less meaningful. For practical usage, it's not that common to deploy an online video inpainting model to a machine with 2 high-performing GPUs. \n- The overall paper looks more like a technical report for engineering tricks, but not a paper discussing learning methods. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject