$ANFluid: Animate Natural Fluid Photos base on Physics-Aware Simulation and Dual-Flow Texture Learning$

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-03 13:28:54

Abstract: Generating photorealistic animations from a single still photo represents a significant advancement in multimedia editing and artistic creation. While existing AIGC methods have reached milestone successes, they often struggle with maintaining consistency with real-world physical laws, particularly in fluid dynamics. To address this issue, this paper introduces ANFluid, a physics solver and data-driven coupled framework that combines physics-aware simulation (PAS) and dual-flow texture learning (DFTL) to animate natural fluid photos effectively. The PAS component of ANFluid ensures that motion guides adhere to physical laws, and can be automatically tailored with specific numerical solver to meet the diversities of different fluid scenes. Concurrently, DFTL focuses on enhancing texture prediction. It employs bidirectional self-supervised optical flow estimation and multi-scale wrapping to strengthen dynamic relationships and elevate the overall animation quality. Notably, despite being built on a transformer architecture, the innovative encoder-decoder design in DFTL does not increase the parameter count but rather enhances inference efficiency. Extensive quantitative experiments have shown that our ANFluid surpasses most current methods on the Holynski and CLAW datasets. User studies further confirm that animations produced by ANFluid maintain better physical and content consistency with the real world and the original input, respectively. Moreover, ANFluid supports interactive editing during the simulation process, enriching the animation content and broadening its application potential.

Main paper: 
 introduction: 
 Drawing inspiration from the enchanting photographs and newspapers of the Wizarding World in Harry Potter, the transformation of static images into animations represents a captivating and burgeoning field, pivotal to the advancement of multimedia editing. This process seeks to amplify the visual imagination, transcending the mere texture of a solitary image. Diffusion models [13] and advanced generative models [1, 2, 40] have been instrumental in achieving realistic and high-fidelity animations through comprehensive end-to-end learning processes. However, accurately simulating the intricacies of nature phenomena, especially fluid dynamics in the real world, without the direct application of physical laws continues to be a formidable challenge.For dynamic fluid photo creation, an effective strategy to address these challenges involves the extraction and estimation of motion fields from static images. Holynski and Mahapatra et al. [14, 26] introduced a specialized phase for estimation of the motion field, which manipulates the deep features extracted to introduce dynamic qualities into static images. Furthermore, an innovative approach within animation generation networks [11] incorporated a physics solver to enhance animation realism by simulating motion. However, these methods typically employ a unified physical solver across various scenarios without fully considering the solver's specific conditions of applicability. This oversight can result in a generalized lack of perception regarding the fluid characteristics relevant to different scenes, thereby compromising the accuracy of the solution. Meanwhile, with respect to texture feature acquisition, these methods enhance animation effects by expanding the parameterization of neural networks, which is offset by the substantial increase in training costs. Moreover, due to inherent design constraints, they do not fully address meticulous feature extraction and effective texture mapping, resulting in consistency imperfections, such as hollow textures and lack of sharpness.As shown in Figure 1 , this study aims to animate natural fluid photos (AnFluid) by introducing Physics-Aware Simulation (PAS) and Dual-Flow Texture Learning (DFTL) methodologies to breathe life into static natural fluid photos. PAS is adept at deriving a physical model (as shown in Figure 1(b) ) from the initial inputs, subsequently deducing a plausible motion trajectory (see Figure 1(c) ) to serve as the cornerstone for the animation. PAS judiciously selects an appropriate physics solver tailored to each scene, thereby preserving the rich tapestry of physical laws observed in nature. To address common animation pitfalls such as holes and blurriness, DFTL forecasts dynamic textures based on the estimated motion, significantly elevating the animation's quality. Furthermore, our innovative ANFluid framework facilitates interactive editing during simulation, thus enriching the animation content and expanding the scope of the application. This suite of techniques ensures higher fidelity to the physical realities of the real world and greater alignment with the original image's content.• We propose a fluid short animation creation method from static photos that leverages physics-aware simulation (PAS) to produce more physically realistic fluid dynimics. The physics-guidance approach to motion estimation aligns more closely with the objective laws of physics. • We design a dual-flow texture learning (DFTL) that effectively mitigates affects such as holes and blurriness textures during the animation generation process. Bidirectional selfsupervised optical flow estimation and multi-scale wrapping can strengthen feature detail extraction and dynamic texture association ability. • We integrate the physics-based and date-driven-based methods within the AnFfluid framework, thus achieve more physically realistic fluid animation generation effects and obtain highly competitive results on the public Holynski and CLAW datasets. 
related work: 
 This article discusses the enhancement of still images with animated motions. Initially, Chuang [6] et al. introduced a stochastic motion texture for simple harmonic motion in dynamic picture areas, requiring users to specify motion parameters. Okabe [29] et al. used a fluid video library to synthesize fluid animations from a single image, prompting users to specify motion regions. Machine learning advancements have led to various approaches for fluid animation generation, categorized into direct generation and staged approaches involving feature extraction and evolution for realistic animations.In the early stages of research, Yitong Li et al. proposed a framework utilizing a hybrid VAE-GAN [19] end-to-end model to generate videos from text [22] . This framework was designed by integrating three network modules: a conditional keypoint generator, a video generator, and a video discriminator. This design allowed the synthesis from text to video to be initially effective. Chao [4] and Walker [37] focused on improving CNN video generators by focusing on specific functionalities such as human pose generation. However, generating fluid dynamics poses a more complex challenge because of the variability of fluid shapes and motion. Generating videos from text lacks the ability to specify initial scenes, leading to random scene generation based on descriptions. Denton [8] and others proposed deterministic trajectories for dynamic effects and random collisions during motion, combining Fixed Prior (SVG-FP) and Learned Prior (SVG-LP) models. They introduced a recurrent inference network for estimating potential distributions at each time step, showing promising results on the MNIST dataset. However, the effectiveness of this approach in the generation of fluid animations remains unexplored. With advances in computing power and dataset sizes, AIGC has shown significant progress in video generation. Runway and Open AI have introduced General World Models and Video generation models as world simulators, relying on vast priors learned by large models for predictions. Despite offering control over video effects through input prompts, inherent randomness poses challenges in precise control [10] . This randomness is particularly noticeable in video generation, leading to variations in animated scenes and phenomena such as "unextinguishable candles" and "ghost chairs". These issues stem from the limitations of probability statistics in expressing physical causality, Sora's inability to assess global rationality, and overlooking critical thresholds in physical processes.In essence, the underlying issue lies in the fact that these methods have not established authentic physical models as the basis for animation generation. The approach proposed in this paper aims to address this fundamental problem by incorporating a more intelligent physics-solving module. This, in turn, enables the efficient generation of physically realistic fluid animations from a single static image.Thi-Ngoc-Hanh [20] divides the entire animation generation process into four steps: extraction of the animation region, flow generation, preservation of curve deformation, and cyclic deformation. However, they did not employ machine learning techniques but used a unified algorithm, similar to the approach used by Yung-Yu Chuang and others [6] in the early stages, to extract flow, generate curve deformations, and cyclic deformations. Holynski [15] introduced the Eulerian motion, a physical model, as a characteristic evolution throughout the generation process. This idea provided inspiration for subsequent work by Siming Fan [11] , who further developed this concept by replacing the simple Eulerian motion module with Surface-only Fluid Simulation (SFS). Additionally, Fan proposed a Surface-Based Layered Representation (SLR) that complements SFS, enhancing the physical realism of the generation process and the granularity of editing.Our work integrates the strengths of the aforementioned studies and introduces a novel motion prediction module, an intelligent perception-based physics solving module to improve the performance of physics, and a video generation module based on the transformer architecture, which addresses the challenges of voids in high-velocity scenarios, resulting in further improvements and advancements in the overall fluid animation generation process. 
method: 
 Given a static image containing a liquid region along with prompts, our task is to generate an animated video that dynamically represents the fluid's characteristics through motion effects while ensuring that the flow adheres to physical principles. This paper proposes ANFluid (as shown in Figure 2 ), a physics solver and data-driven coupled framework that combines physics-aware simulation (PAS) and dual-flow texture learning (DFTL). For the input image and prompts, data preprocessing (DP) is performed to extract the depth and initial motion information. PAS algorithm infers a physically plausible motion field ( Sec. This paper proposes a self-supervised optical flow estimation network (BSF) based on the advanced network architecture proposed by [34] , which incorporates bidirectional constraints. This aims to address the aforementioned limitations and meet the requirements of fluid optical flow estimation more effectively to bi-fluid constraint motion information for MWT. The detailed structure of the network is shown in Figure 3 . This structure mainly draws on the network architecture proposed in recent advanced work [34] , and in order to further enhance the network's accurate estimation of the transparent fluid region, bi-directional sequences are used for constraint, strengthening the network's grasp of subtle details.During the training phase of the model, given the lack of high quality ground truth optical flow data in our dataset, we have incorporated the principles of unsupervised optical flow estimation training methodologies [24, 27, 35] and incorporated commonly employed loss functions. To further enhance the performance of the model, we have integrated several unsupervised components that have exhibited efficacy in previous research, including the loss of smoothness Ls [38] , the loss of census Lc [27] , and the loss of distortion of the boundary expansion [25] . Moreover, to better capture the effects of bidirectional sequence constraints, we have incorporated a constraint loss, formulated as described in Eq. 1:EQUATIONwhere 𝑁 represents the total number of pixels, 𝑥 𝑓 , 𝑦 𝑓 represent the 𝑥 and 𝑦 directional optical flow values obtained from the forward sequence, and 𝑥 𝑏 , 𝑦 𝑏 represent the 𝑥 and 𝑦 directional optical flow values obtained from the backward sequence. This loss function further imposes bidirectional consistency constraints at the pixel level, enhancing the model's ability to estimate complex motions in bidirectional optical flow estimation, thereby improving estimation accuracy.To efficiently extract image features and maintain their continuity, we utilized the Transformer structure [36] , which has been proven effective in a wide range of generative tasks [9, 18, 43, 44] . It performs well in tasks that require warping operations on features. Our multi-scale wrapping image texture feature learning network (MWT) of dual-flow texture learning (DFTL) incorporates the essence of the Transformer structure from Swin-Unet [3] . We employ the U-Net [31] architecture commonly used in generative models [5, 30, 33, 39] , adopting a multiscale skip connection approach to enhance the detailed texture features. For the extracted features, moderate distortion of the image features is applied based on the estimated motion field (𝑀) to obtain the features of the moved image. A detailed explanation of the estimation of the motion field (𝑀) will be provided in Sec. 3.3, briefly mentioned here. This process can be formalized as shown in Eq. 2:EQUATIONwhere 𝐼 𝑓 (𝑇 0 ) represents the first frame of input image, 𝑀 𝑖 (𝑥, 𝑦) represents the motion field for a pixel, where i denotes the ith frame. 𝐼 𝑓 (𝑇 0 → 𝑇 𝑖 ) represents the animation generation process from the 0th frame to the ith frame. 𝐸 ( * ) and 𝐷 ( * ) represent the encoder process and decoder process, respectively. Ultimately, the wrapped image features are processed through the decoder to generate a frame in the animation. Repeating this process yields the complete fluid animation. To generate higher quality animations, we adopt a method similar to [11] , considering 𝑇 𝑖 frames as a linear combination of 𝑇 0 and 𝑇 𝑛 frames for training and using 𝑇 0 frames instead of 𝑇 𝑛 frames during testing to generate loop animations. In practical implementation, to maintain the smoothness and continuity of the generated animation texture, we utilize softmax splatting [28] and learnable composition factors 𝑍 (𝑇 0 ) and 𝑍 (𝑇 𝑛 ) to determine the contribution of overlapping pixels to the transformation of 𝐼 𝑓 (𝑇 0 ) and 𝐼 𝑓 (𝑇 𝑛 ), as proven in [11, 26] . Unlike other methods, we emphasize the importance of 𝑍 ( * ) in image synthesis, which affects the synthesis results. Therefore, based on the feature encoder mentioned above, we construct an independent Z channel for learning 𝑍 ( * ), with 𝑍 ( * ) learning being influenced by the feature encoder but not causing a reciprocal effect.We train the animation generation network using the loss function as shown in Eq. 3:EQUATIONwhere 𝐼 (𝑇 𝑖 ) is the generated frame image at time frame 𝑖, 𝐼 gt (𝑇 𝑖 ) is the ground truth frame image at time frame 𝑖. 𝜆 0 , 𝜆 1 , and 𝜆 2 are weighting parameters. During training, our aim is to optimize the quality of generated images while considering human perception. We use constraint techniques from previous research, including pixel-level constraints, perceptual loss, and discrimination based on image authenticity. The neural network of the transformer architecture shows a smoother continuity of feature extraction. We introduce a transformer-based MAE network for the perceptual loss function, which has been shown to be effective [23] . We input the generated image (𝐼 recon ), and the reference image (𝐼 ref ), into a pre-trained MAE model to extract the representations. Then, we define local perceptual loss as the Euclidean distance between the feature representations Eq. 4.EQUATIONIn Eq. 4, 𝐹 𝑙 denotes the MAE backbone, which produces representations in set {𝑇 𝑙 , 𝑄 𝑙 , 𝐾 𝑙 , 𝑉 𝑙 }. Unlike CNN, the shadow layers of the transformer tend to capture local semantic information, while the deeper layers favor to present the global semantic information. We refer to the specific implementation of [23] .By implementing the comparison of the feature distributions between the generated images and the reference images based on the optimal transport theory Eq. 5 proposed in [23] :EQUATIONwhere 𝐹 𝑙 𝑖 (𝐼 recon ) is one of the extracted features from 𝐹 𝑙 (𝐼 recon ) and similarly for 𝐹 𝑙 (𝐼 ref ). Computing the Wasserstein distance for all the labeled images, we can obtain the distribution-aware loss as the sum of the Wasserstein distances 𝑊 𝑝 𝑝 (𝑢, 𝑣) on 𝐹 𝑙 𝑖 (𝐼 recon ) and 𝐹 𝑙 𝑖 (𝐼 ref ).In Sec. 3.2, we present a comprehensive elucidation of the operational mechanics of animation generation models. However, we omitted a comprehensive elucidation of the motion-field estimation.In this section, we will expound upon our motion field estimation methodology, which imbues a physics-aware simulation. This approach encompasses the generation of initial motion fields through interactions, the incorporation of scene-aware physics-based solving techniques, and motion field smoothing methodologies while taking into account the scene complexity.To allow users control over the final generated effects, we refer to the methods employed by [11, 26] . This enables users to specify dynamic components while providing discrete motion directions that do not exceed 10. We utilized nearest-neighbor averaging to determine the initial velocity for all pixels within the liquid region. Specifically, the velocity at pixel (𝑖, 𝑗) is the exponential average of adjacent pixels.EQUATIONwhere 𝑣 𝑖,𝑗 represents the velocity of the sequentially numbered pixel points in the fluid region. 𝑉 𝑘 denotes the k-th labeled velocity, 𝑑 (𝑘, 𝑖 𝑗) represents the Euclidean distance between the pixel (𝑖, 𝑗) in the image and the position of the k-th label. 𝜎 is a parameter related to the size of the image.Introducing scene-aware fluid simulation. As stated in Sec.3.2 previously, the primary task for static image input is scene categorization to facilitate the selection of an appropriate physical solver. In this regard, we propose a joint integration of the scene perception network with the encoder component of the animation generation network, as depicted in Figure 2 . To achieve this, we enhance the existing encoder architecture by introducing a classification head. Moreover, we enforce constraints on the classification head using the cross-entropy loss function, which takes the form as described in Eq. 7:EQUATIONwhere y represents the true labels and ŷ represents the predicted labels. During the training process, we incorporate this classification task into the constraint loss function of the animation generation network, resulting in the final formulation of the loss function as presented in Eq. 8:EQUATIONwhere 𝜆 3 is weighting parameter. For different scenes, we have adopted two physical solvers, namely the particle-in-cell (PIC) method and the Shallow Water Equation (SWE) method. These solvers are chosen based on different characteristics of the scene, such as waterfalls, rivers, and springs.PIC [12, 45] is a physics simulation method that combines both Eulerian and Lagrangian perspectives. We initialize it using a particle perspective while selecting a standard cubic grid as the framework for the grid perspective. The specific process is as follows.We use a monocular depth estimation network to obtain the depth and position information of the particles. This process can be described as follows (Eq. 9):EQUATIONwhere 𝑥, 𝑦, 𝑧 represent the extracted particle position, 𝑑 is the depth, 𝑢, 𝑣 are the pixel coordinates on the image, and the camera parameters are set to 𝑓 𝑥 , 𝑓 𝑦 , 𝑐 𝑥 , 𝑐 𝑦 , corresponding to a perspective camera with a field of view (FOV) angle of 90 degrees in height.The 3D velocity and position information of particles required for PIC simulation, as well as boundary information, are obtained through predefined camera poses, user input prompts, and 2D velocity transformations. The specific method can be referred to [11] (Eq. 10):EQUATIONwhere 𝑓 𝑥 , 𝑓 𝑦 are the camera's intrinsic parameters, 𝑢, 𝑣 are velocities on the projected image, and 𝑥, 𝑦, 𝑧 are 3D positions in the scene. Once all initialization data are obtained, we use a Taichi [16, 41, 42] implementation of a mixed Eulerian-Lagrangian method [7, 21, 45] to simulate the subsequent motion of each particle. The motion equations are as follows (Eq. 11):EQUATIONwhere 𝑢 represents fluid velocity, 𝜌 represents density, 𝑝 represents internal pressure of the fluid, and 𝑔 represents external forces, considering only gravity. Finally, we perform an inverse operation using the above equations to obtain the 2D motion field required by the network.SWE [32] find extensive applications in domains such as oceanic currents and atmospheric circulation. They represent a specialized expression of the Navier-Stokes equations, assuming a scenario where the depth of the fluid is significantly smaller than the lateral dimensions, which naturally is a 3D simulation based on 2D information. Specifically, SWE requires information such as water surface height, water surface velocity, and boundary information, they all correspond well to the parameters in the equation. Furthermore, this reduction in dimensionality leads to a concomitant decrease in computational complexity. The conservative form of these equations is expressed as follows:EQUATION𝜕(𝜌𝜂𝑢) 𝜕𝑡EQUATION𝜕(𝜌𝜂𝑣) 𝜕𝑡EQUATION)where 𝜂 is the total fluid column height (instantaneous fluid depth as a function of 𝑥, 𝑦 and 𝑡), and the 2D vector (𝑢, 𝑣) is the fluid's horizontal flow velocity, averaged across the vertical column. Further g is acceleration due to gravity and 𝜌 is the fluid density.Similarly to the PIC method, we obtain the water surface height through the estimation of the monocular depth and use the method described in Sec. 3.3.1 to obtain the water surface velocity. Obstacle information is obtained through user input prompts, and boundary conditions are set as reflective boundaries to ensure the free flow of water into and out of the boundaries. As the water surface height information obtained from the monocular depth estimation is represented by values in the range [0,255], the values obtained for the water surface flow velocity under the user's guidance are aligned with this magnitude. In the simulator, we standardize the units to meters. We have employed the open-source solver Anuga to solve the shallow-water equations, configuring it with the aforementioned information to simulate the velocity field after a certain period.Through the aforementioned physical evolution process, we obtained a sequence of velocity fields that evolve over time. However, the simulation-generated results only account for the velocity on the fluid surface, neglecting variations in fluid thickness [11] . As a result, the outcomes are fragile when dealing with complex initial states. Therefore, we need additional approaches to compensate for the detailed motion of fluid textures. Following the baseline approach, we employed a convolutional network to transform the simulated motion field into a more detailed and realistic motion. This network was trained for a conventionalstyle transfer task, with L2 loss between the output velocity and the ground truth. For specific configurations, refer to the pipeline described in [17] . 
evaluation: 
 In this section, we compare our proposed DFTL ( Sec. 4.1) against state-of-the-art learning-based fluid animation methods on the Holynski and CLAW datasets to virify its effectiveness. We then discuss the effects of different motion field estimates on the quality of fluid animation, including our proposed scene-aware physicsbased solver ( Sec. 4.2).Dataset and Evaluation Metrics. For evaluation, we used Holynski data set validation and the CLAW dataset to assess the performance of the animation generation network. It is important to note that in this section the motion of all the methods compared is guided by the optical flow of the entire video to differentiate the performance of synthesis techniques. For quantitative results, we first utilized the 60th frame of each sequence and compared the metrics for all intermediate frames. We used PSNR to show the overall average error, SSIM to show errors in regions with a significant amount of texture, and LPIPS (Alexnet version) to display perceptual loss. This setup is consistent with that of previous work [11] .Quantitative Comparison. Table . 1 presents quantitative results that compare our method with previous approaches. The Reproduced Holynski approach is a typical method based on singlelayer learning, which globally animates scenes. The SLR approach is a method that animates scenes using a surface-based layered representation. Quantitative evaluation is carried out on Holynski's validation set [14] and the CLAW test set [11] , focusing on the first 60 frames. The "fluid region" refers to the static background region replaced by input images during metric computation to improve quality. The baselines are compared under the same ground truth motion, following the settings of previous work [11] . The metrics of other works are taken from previous research [11] . We can observe three main findings from Table 1 : (1) Our method outperforms previous work in most terms of metrics on the CLAW dataset, indicating its superior performance in generating outdoor fluid animations, particularly in complex scene animation tasks. (2) Specifically considering the LPIPS metric, our method surpasses previous work on both the CLAW Testset and the Holynski Common Validation Set, highlighting its ability to generate fluid animations that align better with human perception and provide a more impressive visual experience. (3) Our method, compared to the baseline SLR model, achieves outstanding performance with fewer model parameters and surpasses prior research in multiple metrics.Quantitative Ablation. Table . 2 presents the results of our ablation experiments, where SLR serves as the baseline method. Ours (Non-Zlayer) indicates the use of a Transformer architecture but without employing a separate Z-channel. "Ours" (Non-Multi-scale warp) incorporates a transformer architecture, a separate Z-channel without multi-scale warping. Our model, denoted as "Ours", incorporates a transformer architecture, a separate Z-channel, multiscale warping, and an MAE based perceptual loss. Compared to the SLR method, our approach "Ours" has shown significant improvement in the LPIPS metric and achieved excellent results in other metrics. This suggests that the addition of a separate Z-channel plays a crucial role in the linear combination of features between the initial frame (𝑇 0 ) and the target frame (𝑇 𝑛 ) during the synthesis process. This enhancement improves the accuracy of combining features from preceding and subsequent frames, thereby further enhancing the generation quality. Using multi-scale warping enhances the learning of fine-grained features, leading to comprehensive improvements in all metrics. Building upon this, we incorporated MAEloss into our method, further enhancing its performance on the LPIPS metric, surpassing all previous research efforts.Qualitative Comparison. Figure 4 illustrates the visual detail comparison among Runway1 , baseline, and our method. In Figure 4 (a), the indicated portion shows that our method has improved the problem of holes present at the beginning of the study. In Figure 4 (b), the indicated portion demonstrates that our method does not produce erroneous textures like that of Runway. In Figure 4 (c), the indicated portion shows that the clarity of the images generated by our method is higher than the baseline.User Study. In order to clarify the strengths and weaknesses of our method in real-world applications, we conducted a serious user study, inviting participants to subjectively evaluate fluid videos generated by our model. Three methods were evaluated in user studies: SLR-SFS used in the baseline, Runway, and our proposed method. To visually present the performance of each method in various aspects, we designed five evaluation metrics: video quality, picture integrity, fluid motion authenticity, editing comprehension ability, and texture realism. The detailed explanation of each item can be found in our supplemental appendix.In the study, we presented participants with obfuscated optical flow images as input and contrasted these with fluid animation outcomes that were synthesized using the three specified methods. The participants were then asked to evaluate and rank the quality of the results along a specific dimension across varying scenarios. We aggregated data from a total of 170 completed questionnaires. Figure 6 shows the average scores obtained by converting the rankings into scores based on weighted ranking. Our approach has demonstrated multifaceted enhancements. Notably, there has been a significant boost in the perceived credibility of fluid motion and overall image coherence over the baseline, underscoring the efficacy of the PAS and DFTL techniques. Additionally, unexpected improvements were also observed in video quality and editing comprehension. On the contrary, the animations produced by Runway were found to lag in all evaluated dimensions, with the authenticity of textures being particularly inferior. Such shortcomings stem from the marked deviation of the generative models from authentic textures, yielding a sub-par visual experience. These findings accentuate the merits of our staged animation generation approach.Figure 5 presents the results obtained from different motion generation pipelines. All the results from the following motion estimates share the same input, derived from the algorithmically assigned velocity for each pixel point based on user input. We present the results of our study, which includes the comparison of motion maps generated by SWE and PIC methods in different environments.Additionally, we explore the use of neural networks for motion estimation without physics simulation to highlight differences.From Figure 5 , we derive two important conclusions: Firstly, both PIC and SWE outcomes, due to their reliance on motion-field interpolation, exhibit greater continuity in their results, as exemplified by the image on the left. This stands in contrast to end-to-end methodologies that generate discontinuous velocities, diverging from the natural continuity of water surface movements. Secondly, while PIC typically provides a correctly perceived overall motion, it is prone to loss in accuracy. SWE, on the other hand, consistently produces more coherent and finer motion fields, particularly favoring contexts conducive to shallow-water equations. In the cases of significant elevation differences, the PIC approach outperforms SWE, which has difficulty accurately capturing water flow. For example, utilizing SWE in high-drop scenarios, as shown in the right scene, would neglect the critical vertical descent of waterfalls, recommending the PIC's use in such instances for more verisimilar animations. The contrast in performance between these two simulators endorses our proposed intelligent perception physical-solving module. 
conclusion: 
 This paper introduces ANFluid, an innovative framework that synergizes a physics solver with a data-driven approach, integrating physics-aware simulation with empirical learning to positively animate natural fluid imagery. Unlike previous research, our method capitalizes on PAS to deduce motion fields, which aligns the resultant animations more rigorously with physical laws. DFTL harnesses the power of bidirectional self-supervised optical flow estimation coupled with multi-scale warping to bolster dynamic correspondences, thereby improving the quality of the final animations. There is hope for our work to advance the creation of dynamic fluid photo animation, transitioning from the still photos to the more complex task of dynamic short video generation. With a commitment to fostering collaborative research in this field, we will open-source our code to empower the broader research community to contribute and propel the progress in this area.Notwithstanding the use of a competitive transformer architecture in the generation process, a degree of blurriness is occasionally observed in the output. In anticipation of this, future work will explore the adoption of more refined network architectures with the aim of further advancing the standard of animation generation.  
 
 Appendix: We utilized the same functionality as in Runway (i. e., providing a single image, prompts to generate videos), and provided consistent inputs to serve as end-to-end comparison examples. 


Decision: This paper received a mixed set of reviews. The reviewer who gave the most negative review was mainly concerned with the writing of this paper. Another two reviewers were concerned with the lack of novelty of this paper as the proposed method adopts many of the components from existing papers. After the rebuttal, one of these reviewers was convinced by the rebuttal and switched to a positive recommendation. Another positive reviewer found that the proposed method of combining physics-based and data-driven methods is interesting. Most of the reviewers find that the results are good. Given these, we'd like to recommend the acceptance of this paper and encourage the authors to carefully revise the paper to incorporate the reviews.

Reviews: review: paper_summary: paper_summary: This paper presents an innovative approach combining physics-aware simulation and data-driven learning to animate natural fluid images. While the work demonstrates some degree of novelty, the presentation and clarity require significant improvement.
strength_weakness: strength_weakness: A novel approach combining physics-based and data-driven methods.
limitations: limitations: The writing is often unclear, and the proposed methods are not described comprehensively, leading to a poor reading experience. Here are specific comments and suggestions:

1. Related Work (Section 2.2): The statement, "However, they did not employ machine learning techniques but used a unified algorithm," suggests a misunderstanding by the authors. Not every research needs to employ machine learning techniques. The authors should acknowledge that alternative approaches can be equally valid and effective.

2. Figure 3: The pipeline of the BSF (Bidirectional Self-supervised Optical Flow Estimation) is not clearly depicted. A more detailed and step-by-step explanation is needed to understand the workflow.

3. Section 3.2: The term "(M)" is mentioned, but it is not explained or illustrated in Figure 2. The authors should clarify what "(M)" refers to and ensure it is properly integrated into the diagram.

4. Iterative Animation Generation (Figure 2): The method described involves iteratively generating multiple frames, but Figure 2 does not clearly illustrate the iterative process. The flowchart is confusing and needs to be reorganized to better reflect the iterative steps.

5. Design of MWT: The rationale behind the design of the MWT (Multi-Scale Wrapping Image Texture Feature Learning Network) is not well-explained. Figure 2 provides limited information, and further elaboration on the design choices and their justifications is required.

6. Baseline Model (SLR): The paper uses SLR as the baseline model. The authors need to justify why SLR was chosen as the baseline and how it compares to other potential baseline models.

7. "Modified Holynski": The term "Modified Holynski" is used but not explained. The authors should provide a clear definition and description of what modifications were made to the original Holynski method.

8. User Study: The user study appears to have a high degree of subjectivity. The necessity and validity of such a subjective evaluation should be thoroughly discussed. Additionally, more objective metrics and evaluations could strengthen the study.

Overall, while the paper introduces some novel concepts, the presentation needs significant revision for clarity and comprehensiveness. The authors should address the above points to improve the readability and robustness of their work.
paper_summary: paper_summary: The authors address the task of image animation for natural fluid photos with physics awareness. They propose AnFluid, an integrated system combining both physics-based and data-driven methods. Specifically, they leverage the physics-aware simulation (PAS) to obtain motion guidance, which is better aligned with the laws of physics. They also design a dual-flow texture learning method to mitigate the artifacts existing in previous methods, such as holes and blurriness. The proposed system is evaluated and compared with existing state-of-the-art competitors and outperforms them in both quantitative and qualitative measurements.
strength_weakness: strength_weakness: 1. The paper presented demonstrates good clarity in writing, and the approach is convincing and sound.
2. The application of PAS allows for the selection of an appropriate physics solver tailored to each scene during inference, which can enhance the accuracy of the animation.
3. The integration of data-driven-based training and physics-based inference (in terms of motion field generation) is interesting and valuable for the fluid image animation field.
limitations: limitations: 1. During training, AnFluid utilizes self-supervised optical flow estimation (BSF) for motion field generation and warps the intermediate features. However, during inference, motion fields are obtained based on physics simulation models. How does the model address this mismatch in performance, and does it impact the model's effectiveness?
2. From my perspective, although the results generated by AnFluid appear more "physically" accurate, they seem less realistic compared to flow-based or surface-based methods. How is the realism of these results evaluated?
3. The proposed approach relies on the performance of the motion estimation module. It may not be feasible for users to manually select different physics models for motion estimation. I am curious if, in general, the physics-based method outperforms flow-based methods in terms of the final animation results.
4. The authors should discuss some of the limitations of the proposed method.
Typos:
L162: date -> data
L627: virify -> verify
paper_summary: paper_summary: This paper introduces ANFluid, a physics solver and data-driven coupled framework that combines physics-aware simulation (PAS) and dual-flow texture learning (DFTL) to animate natural fluid photos effectively. The PAS ensures that motion guides adhere to physical laws, and DFTL focuses on enhancing texture prediction. ANFluid surpasses most current methods on the Holynski and CLAW datasets. 

However, the novelty of this paper is limited.
strength_weakness: strength_weakness: - High-Quality Results: 
- -  This paper generates high-quality visual results.
- Writing
- - The writing of this paper is easy to follow.
limitations: limitations: - Dependency on initial inputs:
- - The animation's accuracy depends on the quality of the initial depth and motion field estimations, which might be challenging to obtain in some cases.

- Novelty：
- - The novelty of this paper is limited; the self-supervised optical flow estimation network (BSF) is based on [34]. This will affect the novelty of this paper.
- - The optical flow estimation is also based on the previous approach [24, 27, 35]. The losses include the loss of smoothness Ls [38], the loss of census Lc [27], and the loss of distortion of the boundary expansion [25].
- - The transformer structure is borrowed from [9, 18, 43, 44].
- - The multi-scale wrapping image texture feature learning network is from Swin-Unet [3].
- - The Initial motion of interactive sparse labels is generated by [11, 26].
- - The Motion Field Smoothing is from [17].

- Related works:
- - Some related methods are worth deep discussion: 

- - - Zeng, B., Liu, X., Gao, S., Liu, B., Li, H., Liu, J., & Zhang, B. (2023). Face animation with an attribute-guided diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 628-637).

- - - Shen, S., Zhao, W., Meng, Z., Li, W., Zhu, Z., Zhou, J., & Lu, J. (2023). Difftalk: Crafting diffusion models for generalized audio-driven portraits animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1982-1991).


- Reproducibility:
 - - It could be difficult to reproduce this paper since many configurations are not well mentioned or referenced to another paper.
- Training data
- - The training relies on high-quality trianing data.
paper_summary: paper_summary: The paper introduces ANFluid as a framework to animate natural fluid photos by combining physics-aware simulation (PAS) and dual-flow texture learning (DFTL). The PAS component aims to make motion guides adhere to physical laws and can automatically choose suitable numerical solver for different fluid scenes. Meanwhile, the DFTL component enhances texture prediction by employing bidirectional self-supervised optical flow estimation and multi-scale wrapping. The DFTL design uses less parameters and improves inference efficiency. Experiments are conducted on the Holynski and CLAW datasets and show that ANFluid outperforms prior work.
strength_weakness: strength_weakness: For fluid region, the proposed method generally has better performance than SLR.
limitations: limitations: 1. The paper is not well-organized and challenging to follow.
2. Many components of the proposed method are adapted from "Simulating Fluids in Real-World Still Images", and the key contribution of this paper is unclear.
3. For the Holynski dataset in Table 1, the proposed method has worse PSNR and SSIM than SLR when considering all regions.