Unsupervised Image-to-Video Adaptation via Category-aware Flow Memory Bank and Realistic Video Generation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-05 15:15:34

Abstract: Image-to-Video adaptation is proposed to train a model using labeled images and unlabeled videos to facilitate the classification of unlabeled videos.
The latest work synthesizes videos using still images to mitigate the modality gap between images and videos. However, the synthesized videos are not realistic due to the camera movements are only simulated in 2D space. Therefore, we generate realistic videos by simulating arbitrary camera movements in 3D scenes, and then the model can be trained using the generated source videos.
Unfortunately, the optical flows from the generated videos have unexpected negative impacts, resulting in suboptimal performance. To address this issue, we propose the Category-aware Flow Memory Bank, which replaces optical flows in source videos with real target flows, and the new composed videos are beneficial for training.
In addition, we leverage the video pace prediction task to enhance the speed awareness of the model in order to solve the problem that the model performs poorly in handling some categories with similar appearances but significant speed differences. Our method achieves state-of-the-art performance and comparable performance on three Image-to-Video benchmarks.

Main paper: 
 related work: 
 Image-to-video adaptation methods focus on transferring knowledge from the image domain to the video domain. Existing unsupervised image-to-video adaptation tasks assume that only the labels of images are accessible, while the labels of the target videos are inaccessible. Mitigating the modality gap and reducing the distribution discrepancy are the primary objectives of image-to-video adaptation approaches [16, 20, 47] . For example, generative adversarial network (GAN) [12] is used to learn the mapping between image features and video features in HiGAN [48] and SymGAN [47] . The spatio-temporal causal graph [4] pursues similar goals. In order to mitigate the inherent modality gap between images and videos during domain adaptation, these methods leverage the strong generative modeling capabilities of GANs to transfer knowledge across modalities. CycDA [21] employs a four-stage method for adaptation. Class-agnostic alignment is performed in the first stage to derive pseudo-labels for training an independent spatio-temporal model in the second stage. The next two stages conduct iterative spatial alignment and spatio-temporal learning, with bidirectional knowledge transfer between the two components. Zhuo et al. [51] propose a new framework ST-I2V which synthesizes videos from source static images, thereby converting the image-to-video adaptation task into video-to-video adaptation task. ST-I2V only simulates the transformation of the camera position in the 2D space, resulting in suboptimal performance. In contrast, we generate more realistic video frames by adjusting camera viewpoints at different positions in the 3D space. Additionally, we adopt both RGB and flow branches to construct our spatio-temporal model. Video-to-video adaptation. Different from image-to-video adaptation, video-to-video adaptation methods are proposed to adapt labeled source videos to unlabeled target videos [6, 18, 25, 29] , with their primary focus on addressing the challenges of domain alignment. Discrepancy-based methods are introduced to explicitly minimize domain discrepancies. For example, PTC [10] minimizes the Maximum Mean Discrepancy (MMD) [13] loss across both RGB and optical flow modalities to reduce the domain shift, resulting in improved performance. DVM [43] employs MixUp [50] to mitigate the domain-wise gap. This is achieved by progressively fusing the target videos with the source videos at the pixel-level, allowing for better alignment and adaptation between the domains. In our method, we propose a category-aware flow memory bank to replace the flow data in the generated videos in source domain, thereby reducing the domain gap.Video self-supervised learning offers a promising annotationfree approach for representation learning in video domain. However, learning video representations is challenging due to temporal dynamics, motion, and other environmental factors. One key motivation behind defining pretext tasks is the idea that if a model can perform well on a complex task that requires a high-level understanding of video content, then it will learn more generalizable representations. For example, Jing et al. [15] and Wang et al. [40] design a task that applies appearance augmentations to video clips, and then the model is asked to classify the specific augmentation method that is applied. Fernando et al. [8] and Xu et al. [44] both design their approaches in a way that involves shuffling the order of video frames and having the model predict whether the video segment has been frame-shuffled. In order to enhance the model's awareness of speed and avoid overfitting problem in source domain, we introduce another simple video self-supervised learning method called video pace prediction [41] . 
methodology: 
 The goal of our method is to train a model that can achieves effective classification performance on target videos where the ground-truth annotations come from the labeled source image domain only. We train and evaluate our model in closed-set setting which means only the data from common categories in source and target domains will be used. It is supposed that there are a labeled source image domain 𝐼 𝑠 = {(𝑥 𝑠 𝑖 , 𝑦 𝑠 𝑖 )} 𝑛 𝑠 𝑖=1 and an unlabeled target video domain𝑉 𝑡 = {𝑣 𝑡 𝑗 } 𝑛 𝑡 𝑗=1 .Both domains contain the same 𝐶 classes. Our overall framework is shown in Fig. 3 . Since a large modality gap exists between images and videos, we first convert the imageto-video task into a video-to-video task. To simply construct a spatio-temporal model, we employ the I3D [3] network pretrained on the Kinetics dataset [17] as the backbone for both the RGB and flow branches. Following the instructions of I3D, we extract optical flows from generated source videos and real target videos. The extracted flows are denoted by 𝑓 𝑠 in source domain and 𝑓 𝑡 in target domain. We address the issue of suboptimal performance caused by utilizing the original flows from the generated videos, with our proposed CFB. Furthermore, we enhance the model's ability to perceive speed by applying video pace prediction task [41] .Although Zhuo et al. [51] have provided an effective method for source video generation, the generated videos may not be sufficiently realistic. This is because the approach only simulates the movement of the camera's viewpoint in 2D space, neglecting the fact that the body actions within the video should exist in 3D space.To address this issue, we first generate source videos from source images via a virtual camera motion engine module, inspired by [1] . For a given source image I, we employ MiDaS [34] to estimate the depth map D. The estimated D is then utilized to project pixels in I into 3D space based on the inverse intrinsic matrix M -1 (the intrinsic matrix is used to transform 3D world coordinates into 2D image coordinates captured by a camera). Assuming that I is captured by the camera at 3D location P 0 , we apply an arbitrary virtual motion to the camera, moving it to a new position P 1 . Specifically, we generate a rotation matrix R and a translation vector ì 𝑡 by sampling a random triplet of Euler angles and a random 3D vector, respectively. The transformation matrix is then defined as T 0→1 = (R| ì 𝑡) which is corresponding to the virtual motion path P 0 → P 1 . For each pixel with coordinate 𝑝 in I, the coordinate 𝑝 ′ of its corresponding pixel in I ′ acquired from the new viewpoint P 1 can be obtained by:EQUATION)where D (𝑝) is the depth value with coordinate 𝑝. Finally, the new image I ′ is obtained through forward warping.Following these steps, we alternately generate the subsequent video frame by utilizing the previously generated frame, thereby generating a source video 𝑣 𝑠 𝑖 with continuous action. Subsequently, we utilize the TV-L1 algorithm [49] to compute optical flow 𝑓 𝑠 𝑖 from the i-th source video 𝑣 𝑠 𝑖 and optical flow 𝑓 𝑡 𝑖 from the i-th target video 𝑣 𝑡 𝑖 , respectively. Therefore, the source domain is denoted by𝑉 𝑠 = {(𝑣 𝑠 𝑖 , 𝑓 𝑠 𝑖 , 𝑦 𝑠 𝑖 )} 𝑛 𝑠 𝑖=1, and the target domain is denoted by 𝑉 𝑡 = {(𝑣 𝑡 𝑖 , 𝑓 𝑡 𝑖 )} 𝑛 𝑡 𝑖=1 . Given the source video (𝑣 𝑠 𝑖 , 𝑓 𝑠 𝑖 , 𝑦 𝑠 𝑖 ), we use the cross-entropy loss as the classification loss at the frame-level and video-level referring to I3D [3] and ST-I2V [51] . The frame-level classification entropy loss named local loss L 𝑙𝑜𝑐 is defined as below,EQUATIONHere, CE(•, •) represents the cross-entropy loss, and p (𝑣 𝑠 𝑖 ) ∈ R 𝐾 ×𝐶 corresponds to the network's logits over 𝐾 RGB frames from a generated video 𝑉 𝑠 𝑖 . The variable 𝐵 denotes the batch size. Following the instructions in I3D [3] , we train the RGB branch and the flow branch individually. So the video-level classification entropy loss L 𝑐 is defined as below, where p (𝑣 𝑠 𝑖 ) ∈ R 𝐾 ×𝐶 is an average over logits of 𝐾 RGB frames from video 𝑉 𝑠 𝑖 and p (𝑓 𝑠 𝑖 ) ∈ R 𝐾 ×𝐶 is an average over logits of 𝐾 flows from video 𝑉 𝑠 𝑖 . During the inference phase, the class logits p (𝑣 𝑠 𝑖 ) and p (𝑓 𝑠 𝑖 ) obtained from both RGB and flow branches are normalized via softmax activation function. The normalized results are then summed together, yielding probabilities for each action category. Finally, the action category with the highest probability is selected as the predicted category for the current video sample.EQUATIONAs shown in Fig. 2 , the optical flow frames extracted from generated source videos contain more interference in comparison to the cleaner optical flow frames present in target videos. This phenomenon indicates the significant distribution gap between the source and target domains.To address the negative impacts of the original flows from source videos on the performance of model, we propose a Category-aware Flow memory Bank (CFB). As shown in Fig. 3 , we train the model using the original source videos 𝑣 𝑠 and 𝑓 𝑠 during the warm-up epochs. After the warm-up phase, we utilize the trained model to assign pseudo label for each target video sample. Subsequently, we sort the samples based on the confidences of these pseudo-labels and retain only the top 𝑁 samples for each pseudo category. In this manner, we construct a memory bank of size 𝐶 × 𝑁 , where 𝐶 represents the number of categories. We denote the 𝑛-th flow of category 𝑐 as F 𝑛 𝑐 .Given EQUATIONwhere p (F 𝑛 𝑐 ) ∈ R 𝐾 ×𝐶 is an average over logits of K flows of selected flow F 𝑛 𝑐 .Distinguishing categories with similar visual appearances but significant differences in speeds, such as 'walk' and 'run', poses a challenge for the model. To address this issue, we leverage video pace prediction task [41] to empower the model with the capability to perceive speed by altering video playback speeds. When provided with a video in its natural pace containing 𝐾 frames, we sample video segments 𝑣 𝑡 by various video pace rates 𝑟 . These pace rates correspond to labels from a predefined pace label space R 𝐶 𝑣𝑝 . For example, we produce three pace rate candidates {normal, fast, super fast}, where the corresponding pace labels 𝑟 are {1, 2, 3}, respectively. We randomly choose the starting frame over 𝐾 frames for each target video and then loop over the video at a regular interval 𝑟 until we obtain the desired number of frames for training.With the video segment 𝑣 𝑡 which is sampled by pace rate 𝑟 , the objective of pace prediction task is to understand the content of the video segment and predict the correct pace rate. Subsequently, we utilize the video pace labels to train our model using the crossentropy loss L 𝑣𝑐 , which is defined as follows:EQUATIONwhere 𝑟 𝑖 denotes the video pace label of 𝑖-th video, r ∈ R 𝐶 𝑣𝑝 is the predicted pace logits and 𝐶 𝑣𝑝 denotes the size of pace label space. Overall, all loss functions mentioned above form the complete objective:EQUATIONwhere 𝜆 1 and 𝜆 2 are trade-off parameters. 
experiments 4.1 datasets and setup: 
 We evaluate our method through experiments on three standard image-to-video adaptation benchmarks: E→H, B→U and S→U.In the case of the E→H, we utilize the EADs [5] dataset, which comprises Stanford40 [45] and the HII dataset [37] , as the source image domain, and HMDB51 [19] as the target video domain. There are 13 common classes between EADs and HMDB51 for imageto-video adaptation. The labeled source images and the unlabeled target videos are used to train a model. Regarding B→U, we employ the BU101 [26] dataset as the labeled source image domain and UCF101 [36] as the unlabeled target video domain. We use a total of 101 classes for the image-to-video adaptation task, as the classes in BU101 completely correspond to those in UCF101. For the S→U benchmark, we substitute the source image domain from the B→U benchmark with the Stanford40 [45] dataset. To perform the image-to-video adaptation task, the 12 common classes between Stanford40 and UCF101 are selected for training and evaluation.To generate videos for source domain, we utilize MiDaS [34] whose backbone is pretrained BEIT-Large-512 [2] to extract depth maps from still images in source domain. Subsequently, we utilize Depthstillation [1] to generate 16 video frames using the extracted depth maps and still images. The coefficient of translation vector ì 𝑡 is set to 0.01. Some generated frames can be found in Supplementary material. We use all 16 frames of the generated source videos during training following ST-I2V [51] for fair comparison.For constructing a category-aware memory bank with highquality pseudo-labels, we train the model for 10 epochs as warm-up phase. We then employ the model to assign pseudo-labels for target videos in each subsequent training epoch. We select the top 𝑁 = 60 samples with the highest confidence for each category based on the confidences of the pseudo-labels and store their flow data in the memory bank. The influence of number of selected samples 𝑁 can be found in Parameter sensitivity analysis of subsection 4.4.For building a spatio-temporal model, we use I3D model [3] with both RGB and flow branches pretrained on the Kinetics dataset [17] . We replace the last classifier layer with a fully connected layer that includes 𝐶 neurons. We freeze the first three Unit3D blocks following ST-I2V [51] to accelerate the training process.We train the model with mini-batch stochastic gradient descent optimizer where the momentum and weight decay are set to 0.9 and 0.0001, respectively. The initial learning rates, batch sizes and total epochs are set to (0.05, 0.1, 0.015), (16, 32, 32) and (60, 30, 20) for E→H, B→U and S→U benchmarks, respectively. We also adopt multistep decaying learning rate with a 0.1 decay rate where the milestones are half of the total epochs and the 2/3 of the total epochs. After the warm-up training phase, CFB is applied when the pseudo labels of target videos are more accurate. Following ST-I2V [51] , the values of hyper-parameter 𝜆 1 are set to (1, 20, 1) for E→H, B→U and S→U benchmarks, respectively.We set the size of video pace label space to 5 for all benchmarks, which includes five video pace labels {1, 2, 3, 4, 5}. We randomly select a beginning frame for each target video and loop over the video at the generated video pace rate until the training video clip contains 16 frames. The generated video pace rate is treated as the pace label. For example, if there are 30 frames in the video, and loop over it starting from the 20th frame at 2× speed, the indices of the selected frames are {20, 22, 24, 26, 28, 30, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20}. And the 2× pace is regarded as pace label. During inference, we follow the approach of ST-I2V [51] and extract 32 frames uniformly from each target video for fair comparison. The values of hyper-parameter 𝜆 2 are set to (0.2, 0.001, 0.01) for E→H, B→U and S→U benchmarks, respectively.In our experiments, we conduct comparative evaluations against several prevailing approaches: The DANN [9] pioneers domain adversarial training for classical image-level adaptation. JAN [24] reduces the image-level domain shift by aligning the joint distributions of multiple domain-specific layers. DAL [27] is another image-level domain adaptation method that introduces a novel domain adaptation layer to align source and target distributions with a reference distribution. MEDA [39] minimizes structural risk to train a domain-agnostic classifier on the Grassmann manifold and dynamically aligns the distributions of multiple domains while evaluating the significance of marginal and conditional distributions. HiGAN [48] and SymGAN [47] attempt at bridging the modal gap by mapping image embeddings to video space using GAN [12] . DANN+I3D baseline leverages DANN adapted image features to train an I3D architecture with pseudo-labels which is implemented by Lin et al. [21] . CycDA [21] is a four-stage method that reduces domain discrepancies by using both class-agnostic and class-aware domain alignment techniques. It also utilizes pseudo labels to train a I3D model, effectively bridging the modality gap. ST-I2V [51] is the recent state-of-the-art approach, which employs Grad-CAM [35] and an additional classifier to generate source videos. It transforms the image-to-video domain adaptation task into a video-to-video domain adaptation task. Additionally, for reference, we include the lower bound (SO (Img), where SO stands for Source-only.) and the upper bound (ground truth supervised target) from works [21, 51] .In Tab. 1, we present comparison results. Our approach achieves new state-of-the-art performances on the E→H and B→U benchmarks and demonstrates comparable result on the S→U benchmark. Specifically, our method outperforms ST-I2V [51] note that the E→H benchmark is considerably more challenging than B→U and S→U, given the difficulties in distinguishing categories within the HMDB51 dataset. Nevertheless, the performance of our model on the S→U benchmark still lags behind the current state-of-the-art method. This gap may arise from the fact that our approach primarily focuses on enhancing temporal information, while S→U benchmark relies less on temporal information which can be verified from the superior performance of SO (RGB) in Tab. 2.On the other hand, we can adopt some existing domain adaptation techniques like BNM [7] and MCC [14] to further improve our model on S→U benchmark, achieving new state-of-the-art result, as shown in Tab. 4. The experimental results indicate the effectiveness of our source video generation method, the proposed category-aware flow memory bank and the speed awareness enhancement approach, in the context of image-to-video domain adaptation learning.To study the contribution of each component in our approach towards the overall performance, we conduct the ablation study of our proposed approach. We evaluate the following variants of model: (1) SO (RGB), which denotes the model that contains RGB branch only and is trained with the labeled generated source videos.(2) SO (RGB+flow), the model is trained simultaneously using both the RGB branch and the flow branch with source data. ( 3) SO (RGB+flow) + CFB, is the model trained by replacing the original flow data in source generated video samples with the retrieved flow data from CFB after the warm-up phase. (4) Full Model, that is trained with incorporating VPT (video pace prediction task) into the baseline model SO (RGB+flow) + CFB. Additionally, we include the performance of SO ( [51] ) for reference, which is trained with synthesized videos generated by Zhuo et al. [51] .The ablation study results are shown in Tab. 2. Comparing with SO ( [51] ), we can observe that our model SO (RGB), when using only the RGB branch (the same as SO ( [51] )), has brought improvements of 0.8%, 16.2% and 0.3% on E→H, B→U and S→U, respectively. This means that our video generation approach is more effective to learn a discriminative spatio-temporal model against ST-I2V [51] .When training the flow branch using the original flow data from source videos, it still provides some improvements on the E→H benchmark but has negative effects on S→U and B→U benchmarks. This phenomenon demonstrates the improper flow from source video is one of the key factors contributing to the poor performance of spatio-temporal models.Our proposed Category-aware Flow Memory Bank (CFB) aims to address this issue. The results of SO (RGB+flow) + CFB show that our approach brings performance improvements across three benchmarks, is specially with gains of 13.4% and 5.6% in E→H and B→U benchmarks, respectively. After enhancing the model's perception of speed by introducing VPT, the performance of Full Model is further improved, validating the effectiveness of our approach. Parameter sensitivity analysis. To evaluate the parameter sensitivity, we conduct a series of experiments on the E→H benchmark. Fig. 4 reports the results of parameter sensitivity analysis, and more results can be found in Supplementary material.The weight 𝜆 1 is one of the key factors that influences the performance of the model. We evaluate the impact of different values of 𝜆 1 using the baseline SO (RGB) and present the results in Fig. 4 (a). As observed, a small 𝜆 1 leads to the model ignoring appearance characteristics of video frames, while a large 𝜆 1 results in the model excessively focusing on appearance characteristics at the expense of temporal features. This further hinders the model's ability to comprehend the content of the video.An appropriate dimension of CFB is a critical factor in determining the performance of model. So we investigate the impact on performance by setting different numbers of flow samples 𝑁 for each category in baseline SO (RGB+flow) + CFB and the results are shown in Fig. 4 (b) . It is evident that the value of 𝑁 can significantly influences the performance of model. An excessively large 𝑁 may introduces noisy flow samples with low-quality pseudo labels, which leads to inferior performance. Conversely, a small 𝑁 may leads to insufficient model generalization.To investigate the proper settings for VPT, we explore different video pace prediction loss weights 𝜆 2 and sizes of video pace label space 𝐶 𝑣𝑝 based on baseline SO (RGB+flow) and the results are shown in Fig. 4 (c) and Fig. 4 (d) respectively. From the above experimental results, it is observed that employing video pace prediction task on SO (RGB+flow) can achieves competitive results under a appropriate range of 𝜆 2 values. Additionally, the small size of the video pace label space restricts the model's ability to perceive speed. On the other hand, a over-sized video pace label space increases the difficulty of the speed prediction task, making it challenging for the model to learn meaningful semantic representations. The results indicate that regardless of using mean pooling or weighted summation to aggregate retrieved flows, the constructed flows lead to suboptimal results or even have significant negative impacts on the performance of model. This could be due to that the aggregated flows lose too much information and dissimilar to real flow data. With less (top5) flows for aggregation, the performance is improved. With only top1 flow, the performance is further improved. So we only use 1 flow without aggregation. We use Random 1/60 as we think that randomly choosing 1 flow may boosts the robustness of the model and the experimental result verifies our conjecture. So we use Random 1/60 in our manuscript.Integrating domain adaptation techniques. We employ several typical domain adaptation techniques into the constructed baseline SO (RGB+flow) + CFB, including DAN [23] , MCC [14] and BNM [7] . Specifically, we use L ′ = L ′ 𝑐 + 𝜆 1 L 𝑙𝑜𝑐 + 𝜆 3 L 𝑡 𝑓 to train the model. L 𝑡 𝑓 is the transfer loss likes MMD [23] loss, and BNM [7] loss.The values of hyper-parameter 𝜆 3 are set to (0.05, 0.3) for E→H and S→U, respectively. The results on E→H and S→U are shown in Tab. 4 and we also report the results of SO (RGB+flow) + CFB for better demonstration.It is observed that the performance of our model can still be greatly improved by applying typical domain adaptation methods on our constructed video-to-video domain adaptation baseline which involves generating source videos and utilizing CFB. With recent state-of-the-art domain adaptation techniques MCC [14] and BNM [7] , our method outperforms CycDA [21] on S→U benchmark, achieving new state-of-the-art performance. In addition, the results also demonstrate that the combination of our method with DA techniques shows good adaptability on E→H. Compatibility with CLIP. We replace the network of RGB branch in sec. 4.2 with visual encoder of CLIP [31] whose backbone architecture is VIT-B/32. Different from the experimental hyper parameter settings described in sec. 4.2, we set total epochs to (40, 20) and the number of warm-up training phase to (10, 5) for E→H and S→U benchmarks respectively. And setting the batch size to 8, the learning rate of CLIP to 5e-5 and the prompt for text encoder to 'a video of a person {}.' for both benchmarks. {} in prompt indicates category names like 'climb', 'run' and so on.For reference, we include the results of our approach employing the I3D network in RGB backbone. The results shown in Tab. 5 indicates that the performance improvements can be attributed to the robust representation learning capability and the powerful knowledge base of CLIP itself. By integrating CLIP into our method, we can leverage its knowledge and capabilities to enhance the performance. Additionally, the compatibility between our method and CLIP is crucial for achieving further performance gains. The ability of our method to effectively incorporate CLIP's features and merge them with the existing framework allows for a synergistic effect, especially resulting in improved performance on E→H benchmark. Our approach's compatibility with prevailing large multimodal models like CLIP showcases its strength and demonstrates its ability to achieve better results. T-SNE visualization. We visualize the representations of the baseline models, SO (RGB+flow) and SO (RGB+flow) + CFB, in Fig. 5 using t-SNE [38] . We project the source and target videos of E→H benchmark into 2-dimensional representations. Intuitively, there are more confusions among the representations from SO (RGB+flow) (Fig. 5 (a)), which can be improved by incorporating CFB, as shown in Fig. 5 (b). Specifically, for categories like 'pour', 'kick', and 'push', the incorporation of CFB enables the model to learn more discriminative representations, enabling better differentiation from others. Moreover, for categories of 'talk', 'smoke', 'climb' and 'wave', the representations from both the source and target domains become closer after replacing the flows retrieved from CFB, as depicted in Fig. 5 (b) . It is suggested that our proposed CFB further reduces the distribution discrepancies between source and target domains, which is beneficial for training a model with great generalization. 
conclusion: 
 We overcome the challenges of image-to-video domain adaptation task, aiming to enhance the spatio-temporal model's discriminative ability for unlabeled video classification in the target domain. To mitigate the modality gap between labeled source images and unlabeled target videos, we generate realistic source videos by simulating diverse camera movements in 3D scenes and the new perspectives are served as frames. To further mitigate the negative influences of the flows extracted from the generated source videos, we propose the category-aware flow memory bank (CFB). By replacing the optical flow in a generated source video with real target flow which is retrieved from CFB, we create a new video sample that closely resembles the target video. Additionally, we leverage the video pace prediction task to enhance the model's perception of speed. Our proposed method demonstrates promising results compared with the current state-of-the-art approaches. In our current method, the domain discrepancy is not fully concerned, which could be further improved with video-to-video domain adaptation methods in the future work. 
: 
 Figure 1 : Generating a video from a single source image is achieved through the Depthstillation [1] pipeline. Initially, we project the pixels in the input image I into 3D space, guided by the corresponding estimated depth map D. Subsequently, we move the camera along a virtual motion path from P 0 to P 1 . Finally, this process yields a new view I ′ . By combining these synthesized views, we can construct a more realistic source video.video retrieval [11, 42] , intelligent video surveillance [33, 46] , and video captioning [28, 32] . However, training a high-performance video classifier requires collecting and annotating a large amount of video data, which is costly and time-consuming. As images are easier to annotate than videos, and there are numerous labeled image datasets accessible, image-to-video domain adaptation methods [4, 21, 22, 30, 51] that leverage the labeled images and unlabeled videos for training high performance video classifier appear as a challenging task and attract much attention.The first challenge of image-to-video adaptation is the modality gap between images and videos. This gap refers to the fact that the temporal information in videos does not exist in source images. Bridging this modality gap is necessary for transferring knowledge from source domain to target domain effectively. Another challenge arises from domain discrepancy caused by variations in scenes, image styles and so on between source images and target video frames. Domain discrepancy is a key factor causing models trained in the source domain to perform poorly in the target domain. Overcoming these two key challenges of modality gap and domain discrepancy is crucial for achieving effective image-to-video adaptation.Existing approaches [47, 48] predominantly employ a two-stage paradigm to address the challenges of domain discrepancy and modality gap. The first stage involves frame-level adaptation to reduce domain discrepancies between source images and target video frames. The second stage entails learning a spatio-temporal model to bridge the modality gap and incorporate temporal information. For example, Wei et al. [21] first employs DANN [9] for frame-level alignment and then leverages pseudo-labels from the first stage for self-supervised learning on the target videos. Recently, Zhuo et al. [51] proposes a single stage method ST-I2V by synthesizing source videos from static images with the help of Figure 2 : We employ TV-L1 [49] to extract optical flows from both source and target videos for category 'jump', as shown in (a) and (b) respectively. It is evident that there is a significant discrepancy between the flows of the source and target videos. Specifically, the flows of the source video exhibit more interference and noise, while those of the target video appear cleaner.Grad-CAM [35] , and solve the image-to-video adaptation problems with video-to-video domain adaptation methods.Though being simple and effective, ST-I2V [51] randomly selects regions within image as intermediate frames to simulate camera displacement in 2D space which results in unrealistic synthesized video with improper temporal information. To address this issue, we rethink the imaging process in 3D space. As shown in Fig. 1 , by recovering the position of camera, we can simulate the arbitrary movement of camera and generate realistic frame. Compared with ST-I2V [51] , our method is simpler as it does not require training an additional classifier to locate major objects for an action. Besides, the complete original images are retained, avoiding any loss of crucial appearance information. The generated source video preserves static appearance and dynamic motion information, which is beneficial for training a discriminative spatio-temporal model.Utilizing the generated source videos, we can train a simple but effective spatio-temporal baseline model through cross entropy loss with labels of source domain. However, we observe an unfavorable phenomenon that the optical flows extracted from generated videos are not helpful for training a discriminative model and even lead to suboptimal performance. As shown in Fig. 2 , the optical flows from generated source videos exhibit more noise and interference compared with those of the target videos. Consequently, there is a noticeable distribution gap between the source and target videos, resulting in suboptimal performance.To address the significant discrepancies between the flows in source and target videos, we construct a Category-aware Flow Memory Bank (CFB). The memory bank stores real flow data for each category within the real videos from target domain, where the category is determined by the pseudo label of target video. For a source video with ground-truth label 𝑙, we randomly select a flow of corresponding category 𝑙 from the CFB. Then the selected flow is used to replace the original flow in the source video. As the new source video is more similar to the target video, the performance of the model is greatly improved. Nevertheless, it is still difficult for the trained spatio-temporal model to distinguish categories with similar visual appearances but significant differences in speeds, such as 'walk' and 'run'. So we leverage the video pace prediction task [41] to enhance the model's perception of speed by altering video playback speeds. Specifically, we sample video clips at varying pace rates and treat the pace rates as labels. Subsequently, the spatio-temporal model is also trained with cross-entropy loss with video playback speed labels. Additionally, integrating video playback speed prediction task prevents the model overfitting on the source domain and enables the model to better generalize to target domain.We validate our method on three widely used image-to-video adaptation benchmarks. The experimental results show that our method performs favorably against the current state-of-the-art approaches. We achieve the best-published results on the challenging E→H and B→U benchmarks, and competitive results on the S→U benchmark. Ablation studies are presented to verify the contribution of each key component in our approach.In a nutshell, our contributions are as follows:• To generate realistic source videos, we simulate camera movements in a 3D scene and capture new camera views that serve as source video frames, inspired by [1] . The generated source videos are very promising in training a discriminative spatio-temporal model. • We propose a Category-aware Flow Memory Bank (CFB) to compensate the improper temporal information of the generated videos in source domain. By replacing the original flows of source videos with the retrieved flows from CFB for training, a remarkable improvement in the performance of model is achieved. • We integrate video pace prediction task [41] to enhance the model's perception of speed, which enables the model to distinguish categories with similar visual appearances but differences in speeds. • Extensive experimental results show that our method achieves the best performance on the challenging E→H and B→U benchmarks and attains comparable results on the S→U benchmark. 
 
 Appendix:  


Decision: This paper introduces a novel approach for unsupervised image-to-video adaptation using category-aware flow memory banks and realistic video generation techniques. Despite concerns raised about the rationale behind optical flow replacement and the novelty of some methodologies, the authors provided satisfactory responses that addressed these issues. As a result, all reviewers reached a consensus to accept this paper.

Reviews: review: paper_summary: paper_summary: This paper aims to tackle the task of Image-to-Video Adaptation in an unsupervised manner, which is based on the previous work and thus is incremental.
strength_weakness: strength_weakness: The paper writing is well organized and the quantitative result looks promising.
limitations: limitations: Author utilizes optical flow values retrieved from the Category-aware Flow Memory Bank to replace optical flow in the generated source video. I question the reasonableness of this design since optical flow should be data-dependent. It is unclear to me how replacing the optical flow values from one video with those from another can be rational. Optical flow represents the actual movement of specific positions in a video, and substituting these values does not seem to make sense. For this design to be effective, the source and target videos would need to exhibit similar motions in both direction and magnitude, which is not commonly observed.
paper_summary: paper_summary: This manuscript aims to generate realistic videos by simulating camera movements in 3D scenes. To solve the interference and noise problem exhibited in the flows of the source videos, the authors introduce the Category-aware Flow Memory Bank, which replaces optical flows in source videos with real target flows. To solve the problem of identifying different categories of videos with similar appearance, the authors leverage the video pace prediction task to enhance the speed awareness.
strength_weakness: strength_weakness: 1. For the task of image-to-video adaptation, the authors propose to simulate arbitarary camera movements in 3D scenes, which can provides more spatial-temporal information for the realistic video generation. 
2. The authors propose a Category-aware Flow Memory Bank (CFB) to compensate the improper temporal information of the generated videos in source domain, reducing the improper impacts caused by the optial flow.
3. The writing of this paper is clear, and readers can understand the key points and effectiveness of the proposed method easily.
limitations: limitations: 1.	The idea of simulating the camera movement in 3D scenes to generate video lacks novelty. The manuscript leverages a method called dephtstilled mentioned in "Learning optical flow from still images. CVPR'21". This method learns to obtain new perspective images and optical flow by giving a single image and the corresponding depth map, which contributes more to the cleanliness of the optical flows of the generated videos.
2.	Figure 2 shows the optical flow comparison between the source domain and the target domain, but it does not seem to indicate the generation results of the source domain video from which method, and it is doubtful whether the "interference and noise" problem is common.
3.	It is doubtful whether the proposed CFB strategy is reasonable. The authors claim that the Memory Bank stores the optical flow information extracted from the real target video and serves as pseudo labels for the video in the corresponding category of the source domain. Due to the similarity of optical flow in the same category (Line 172), the authors replace the optical flow of the source domain video in training. I think the judgment that the same category is equivalent to optical flow similarity is not rigorous enough. On the one hand, there may be large differences in optical flow within the same category, and the optical flow content between any two videos is misaligned. There may also be large similarities in optical flow between categories. On the other hand, there is a lack of sufficient experimental analysis to support the author's hypothesis.
4.	To deal with difficult similar video action categories, the authors sample video clips at varying pace rates and treat the rates as the video playback speed (line 539). I think it lacks rationality. For example, the difference between the two actions of "walk" and "run" is the size of the body movements, not the video playback speed. Moreover, the method proposed by the author appeared in "Self-supervised Video Representation Learning by Pace Prediction, ECCV2020".
5.	The effectiveness of CFB shown in Figure 5 is not intuitive, and the representations of a considerable number of categories are still difficult to distinguish after using the CFB strategy.
paper_summary: paper_summary: This paper proposes a new image to video adaptation method. First, it generates more realistic videos to mitigate the modality gap between source images and target videos. Then, to mitigate the gap between the flows extracted from the generated videos and target videos, it then proposes the category-aware flow memory bank, by replacing the optical flow in a generated source video with the optical flow selected from target videos, and then generate new composed videos for training. They also leverage the video pace prediction task to enhance the model's perception of speed. The proposed method achieves state-of-the-art performance in the field of image to video adaptation.
strength_weakness: strength_weakness: 1. The proposed several approaches, such as the diverse camera movements simulation and the category-aware flow memory bank, are effective for unsupervised image to video adaption with good illustrations.
2. The proposed method achieves superior performance in the field of unsupervised image to video adaption. 
3. The overall structure of this paper is well organized.
limitations: limitations: 1. Is it possible for the authors to report the performance of "SO [51] + flow" and "SO (RGB) + CFB" to better demonstrate the effectiveness of the proposed category-aware flow memory bank? 
2. It seems that the generated videos with 3D scenes are only used during the warm-up epochs if the category-aware flow memory bank will be used. If we could approach other pretrained image to video adaption models, are there any ways to directly use a category-aware flow memory bank to generate videos for training?
paper_summary: paper_summary: Image-to-Video adaptation refers to training models using labeled images and unlabeled videos to classify unlabeled videos. Existing research uses still images to synthesize videos, however, simulating camera motion in 2D space leads to synthesized videos that are not realistic. Therefore, this manuscript generates realistic videos by simulating arbitrary camera motions in 3D scenes, and then trains the model using the generated source videos.
strength_weakness: strength_weakness: 1. This manuscript provides effective solutions to address the challenges of image-to-video adaptation, including the large modality gap between labeled source images and unlabeled target videos, the negative influences of extracted flow from generated source videos on model performance, and the difficulty in distinguishing categories with similar visual appearance but differences in speed. The experimental results show that the proposed method achieves state-of-the-art and comparable performance on three Image-to-Video benchmarks.
2. The manuscript has strong logical coherence and the effectiveness of the proposed method has been verified and analyzed through sufficient ablation experiments.
limitations: limitations: I am not very familiar with this field, but I believe that there are still the following issues that need to be clarified in this manuscript.
1. In Figure 4(b), the proposed model achieves optimal performance when N is set to 70. Why was it ultimately set to 60?
2. Why is the performance on benchmark B->U not reported in Tables 4 and 5?