LoopGaussian: Creating 3D Cinemagraph with Multi-view Images via Eulerian Motion Field

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-07 11:38:17

Abstract: Cinemagraph is a unique form of visual media that combines elements of still photography and subtle motion to create a captivating experience. However, the majority of videos generated by recent works lack depth information and are confined to the constraints of 2D image space. In this paper, inspired by significant progress in the field of novel view synthesis (NVS) achieved by 3D Gaussian Splatting (3D-GS), we propose \textbf{\textit{LoopGaussian}} to elevate cinemagraph from 2D image space to 3D space using 3D Gaussian modeling. To achieve this, we first employ the 3D-GS method to reconstruct 3D Gaussian point clouds from multi-view images of static scenes, incorporating shape regularization terms to prevent blurring or artifacts caused by object deformation. We then adopt an autoencoder tailored for 3D Gaussian to project it into feature space. To maintain the local continuity of the scene, we devise SuperGaussian for clustering based on the acquired features. By calculating the similarity between clusters and employing a two-stage estimation method, we derive an Eulerian motion field to describe velocities across the entire scene. The 3D Gaussian points then move within the estimated Eulerian motion field. Through bidirectional animation techniques, we ultimately generate a 3D Cinemagraph that exhibits natural and seamlessly loopable dynamics. Experiment results validate the effectiveness of our approach, demonstrating high-quality and visually appealing scene generation.

Main paper: 
 introduction: 
 Cinemagraphs are static images in which a minor and repeated movement appears, forming a seamlessly looping video clip [10] . It offers a unique blend of static imagery and subtle motion by combining elements of both photography and videography, captivating audiences with its mesmerizing allure. In light of recent advancements in concepts such as augmented reality, mixed reality, and metaverse, there has been a growing demand for creating natural and realistic cinemagraphs [2, 19] . However, generating dynamic 3D scenes within cinemagraphs typically entails extensive manual labor from professional artists, leading to considerable costs.Existing works [12, 14, 30] mainly focus on automatically creating cinemagraphs from static images in 2D image space. Mahapatra and Kulkarni [30] propose to interactively animate fluid elements in still images based on flow directions and regions of interest provided by users. However, these manipulations are all done in 2D image space where the view direction is inevitably fixed, which lacks a sense of visual fidelity.A few methods [19, 29] that explore the generation of 3D cinemagraphs have recently emerged. For example, Li et al. Li et al. [19] take the first step to create realistic animations of scenes with compelling parallax effects by jointly learning image animation and novel view synthesis. Ma et al. Ma et al. [29] introduce a pipeline to create cinemagraphs from asynchronous multi-view videos, thereby facilitating the exploration of diverse viewpoints. Nevertheless, we argue that these approaches cannot be considered as authentic 3D cinemagraphs, as they merely employ representations of multi-layer two-dimensional images with depth information such as MPI [55] or LDI [44] , failing to reconstruct the underlying three-dimensional geometric structure of scenes. Consequently, these methods struggle to produce the effect of camera movement or need to restrict the camera movement within a confined viewing angle and range. In addition, the lack of geometric information may lead to artifacts or produce geometric inconsistency.In consideration of these constraints in existing methodologies, we present LoopGaussian, an innovative framework for curating authentic 3D cinemagraphs from multi-view images of static scenes. LoopGaussian is grounded in the reconstruction of the 3D structure of the scene from multi-view images, taking advantage of the state-of-the-art 3D Gaussian Splatting [17] . Consequently, we can efficiently perform operations that are previously unattainable in 2D space, such as novel view synthesis and dynamic scene rendering. Moreover, our method can effectively leverage the inherent self-similarity within the scene representations, eliminating the necessity for pre-training on extensive datasets.We streamline our framework into three steps, as illustrated in Fig. 1: (a) . Firstly, we train a 3D Gaussian scene using multi-view static images, followed by an autoencoder that maps 3D Gaussian points into an appropriate feature space. (b) Next, we devise a clustering method on these 3D Gaussian points and exploit selfsimilarity among clusters to construct a velocity field. The velocity field is further refined with a multi-layer perceptron to obtain an Eulerian motion field. (c) Finally, we generate a seamlessly loopable video based on the derived Eulerian motion field with bidirectional animation techniques. The generated video clip is more plausible and visually captivating owing to its generation process in the three-dimensional space. In our experiments, we consider simulating motions of soft and non-rigid objects, such as tree branches, flags, and hanging clothes. Our experimental results demonstrate the effectiveness of our proposed method. In summary, our main contributions are as follows:• We propose a novel framework capable of generating authentic 3D Cinemagraphs from multi-view images of static scenes, which achieves seamless loopable dynamics of the scene and can be rendered from a novel viewpoint. • We innovatively describe the dynamics of the scene in terms of Eulerian motion fields in 3D space. Leveraging the scene's self-similarity, we employ a two-stage optimization strategy to estimate the Eulerian motion field.• Our framework is heuristic, obviating the necessity for pre-training on large datasets, and it offers flexibility by enabling users to control the magnitude of the scene dynamics. 
related work 2.1 cinemagraph: 
 A cinemagraph [10, 43] is a combination of a static image and a dynamic video, where most of the scene is still while a fraction of it changes in a continuous loop. This concept has gained popularity in diverse domains, ranging from artistic expression and digital storytelling to advertising and brand marketing. While advanced digital tools have empowered artists and photographers to craft photo-realistic cinemagraphs, the manual creation process is still labor-intensive and time-consuming.There exists a rich body of works [1, 21, 22, 34, 45, 52] that explore an automatic creation of cinemagraphs. Earlier methods commonly take a video as input to generate a seamlessly looping video clip. For instance, by identifying segments with cyclic motion properties [21, 45, 52] . Agarwala et al. [1] create panoramic video textures from the output of a single panning video camera. Oh et al. [34] introduce an end-to-end approach that extracts high-level semantics from the input video to facilitate cinemagraph generation. In addition, there are several works [6, 7, 23 ] that aim at creating a cinemagraph from a single image. Chuang et al. [7] propose a semi-automated method that lets users manually segment the scene into multiple layers. Subsequently, stochastic motion textures are automatically synthesized for each layer, which are then integrated to create the final video. Lin et al. [23] demonstrate the capability of generating waterfall animations from static waterfall images. Choi et al. [6] automatically generate cinemagraphs from a still landscape image using a pre-trained StyleGAN [16] . Text2Cinemagraph [31] presents a fully automatic method that synthesizes cinemagraphs from text descriptions. However, these works are all constricted to 2D image space, which may fail to deliver an immersive experience for audiences.Apart from 2D cinemagraph generation, Li et al. [19] pioneered a framework for generating 3D cinemagraphs from a single still image. This approach utilizes a dense depth map to separate the scene into several layers and expand 2D motion into 3D scene flow. In contrast, we directly handle 3D points derived from the scene in 3D space without endeavor of predicting any depth maps, yet still achieving visually appealing looping effect. Moreover, our method is capable of exploring representation similarity in 3D space to construct cinemagraphs, thus alleviating the need for pre-training on large-scale datasets.Neural scene representation aims at modeling the scene via neural networks, in which way the entire rendering pipeline is differentiable, and the whole scene is learnable. Neural Radiance Field (NeRF) [32] is one of the most popular methods for implicit neural scene representation. It models the view-dependent color and opacity at each spatial position through Multi-Layer Perceptrons (MLPs), and enables novel view synthesis through volume rendering. The success of NeRF has inspired a significant body of subsequent research. Some studies are devoted to improving the training and rendering speed [13, 27, 33, 41, 53] , while others aim at elevating the rendering quality [3, 18, 42, 47, 48] . Additionally, several efforts are made towards adapting NeRF for dynamic scenes [4, 11, 37, 39] .3D Gaussian Splatting (3D-GS) [17] is an emerging neural scene representation approach that explicitly models 3D information of a scene. It employs a collection of semi-transparent anisotropic Gaussian ellipsoids to represent the input scene, and designs a differentiable rasterization rendering pipeline to enable real-time high-fidelity rendering. While 3D-GS is initially designed for static scenes, some subsequent works have extended it to handle dynamic scenes. One line of work is centered around the idea of predicting the temporal evolution of individual 3D Gaussians using neural networks [8, 15, 24, 49, 51] , while another entails representing a dynamic scene within a 4D space where each moment is represented as a slice of this space [9, 20, 28] . Although 3D-GS achieves notable success in modeling dynamic scenes, existing approaches are generally based on the Lagrangian perspective. In this paper, we make an early attempt to describe the progressive dynamics of a scene from an Eulerian perspective. 
preliminaries 3.1 3d gaussian splatting: 
 3D Gaussian Splatting (3D-GS) [17] has recently emerged as an explicit scene representation and rendering approach, which allows high-quality and real-time rendering for scenes captured with images from multiple viewing directions.3D-GS explicitly represents the scene as a set of 3D Gaussian points. Each 3D Gaussian point 𝐺 𝑖 is characterized with attributes {𝑝 𝑖 , 𝑞 𝑖 , 𝑠 𝑖 , 𝜎 𝑖 , 𝑐 𝑖 , 𝑆𝐻 𝑖 }, where 𝑝 𝑖 ∈ R 3 is the space position, 𝑞 𝑖 ∈ R 4 is the quaternion representing rotation, 𝑠 𝑖 ∈ R 3 is the scaling on each axis, 𝜎 𝑖 ∈ R is the opacity, 𝑐 𝑖 ∈ R 3 is the diffuse color, and 𝑆𝐻 𝑖 is the spherical harmonic function to express anisotropic colors. The dimensions of 𝑆𝐻 are depended on the order used. The shape of each 3D Gaussian point is controlled by a covariance matrix Σ = 𝑅𝑆𝑆 𝑇 𝑅 𝑇 , where 𝑅 is the rotation matrix transformed from quaternion 𝑞, and 𝑆 is the scaling matrix transformed from 𝑠. Hence, each 3D Gaussian point can be expressed as:EQUATIONDuring the rendering procedure, each 3D Gaussian is projected onto an image plane in camera space to shape a 2D Gaussian. To determine the color of each pixel, the Gaussians that are contained in one pixel are sorted by depth, and the pixel color ĉ is estimated according to 𝛼-blending:EQUATIONwhere 𝐺 pixel is a set of Gaussians that are contained in this specific pixel, ĉ𝑖 is the learned color of the Gaussian, and α𝑖 is the learned opacity multiplied with the Gaussian.Existing approaches that capture Gaussian point cloud dynamics are generally based on the Lagrangian perspective. The Lagrangian methods track individual particles over time, focusing on their trajectories as they move through space, which is fairly intuitive.In contrast, the Eulerian perspective focuses on specific locations in space, observing how particles move through these locations over time. To approximate such an Eulerian motion field, Holynski et al. [14] propose a static motion field in 2D image space, where the value of each pixel defines its immediate velocity that remains constant over time. We follow this method and adapt it to describe the deformation of soft non-rigid objects (such as branches, flags, ropes, etc.) in 3D space. Formally, the motion of a particle from one frame to the next through Euler integration is described as follows:EQUATIONwhere ì 𝐸 is the static Eulerian motion field and 𝑋 (𝑡) is the position of a particle at time 𝑡. 
method 4.1 overview: 
 Given a set of multi-view images of a static scene, our goal is to create a seamlessly loopable and natural-looking 3D cinemagraph. The overview of our method is illustrated in Fig. 2 . We start by creating a 3D Gaussian point cloud using 3D-GS [17] with an additional eccentricity regularization (Sec. 4.2). As not all objects in the scene are suitable for deformation, we segment the parts that are likely to exhibit loopable motion with SAGA [5] . The dynamic 3D Gaussians are then projected into a feature space via an autoencoder and clustered according to both position and feature information through our designed SuperGaussian approach (Sec. 4.3). Next, we derive a global feature for each cluster and calculate the similarity among clusters based on these global features. The similarity information is used to estimate a velocity field, which is further refined with an MLP as the final Eulerian motion field (Sec. 4.4). Finally, we achieve a loopable video based on our estimated Eulerian motion field using bidirectional animation technology [14, 19, 30] (Sec. 4.5). We will elaborate on the details in the following subsections.3D Gaussians Generation. We obtain the flexible representation of the initial static scene, which is composed of 3D Gaussian points, using 3D-GS [17] as described in Sec. 3.1. We choose 3D-GS because a 3D Gaussian point is essentially an ellipsoid, the shape of which can efficiently and accurately characterize intricate geometries when the scene is static. However, the dynamics of a scene's deformation can inevitably modify the positional relationships among these Gaussian points, and the excessively sharp ellipsoids may introduce glitch artifacts, thereby compromising the visual coherence of the scene. To address this, inspired by prior works [26, 50] , we improve the representation robustness and visual fidelity of the scene by introducing a constraint on shape during training. Specifically, we incorporate a regularization term targeting at ellipsoidal Given multi-view images of a static scene, we initially create a 3D Gaussian point cloud using 3D-GS with an eccentricity regularization term. Next, we identify the point cloud region that the user wishes to deform using a 2D Mask. The 3D Gaussians are then projected into the feature space via an autoencoder and undergo clustering using SuperGaussian. Subsequently, we derive a sparse velocity field based on self-similarity, interpolate to acquire a dense velocity field and refine the final Eulerian motion field through an MLP. Finally, we can generate a seamlessly loopable video by leveraging bidirectional animation techniques in 3D space and incorporating specified camera parameters.eccentricity when training the 3D Gaussian scene:EQUATIONwhere 𝑠 𝑖 is the scaling on each axis and G ≔ {𝐺 𝑖 } is the set of 3D Gaussian points of the scene as described in Eq. (1).3D Gaussians Optimization. During the training process, the 3D Gaussian point cloud is rendered to a novel view image through the rasterization pipeline explained in Sec. 3.1. The error is then calculated as the difference between the rendered image and its corresponding ground-truth image. Following [17] , we adopt the absolute error L 1 and the structural similarity index L D-SSIM as the difference measure. Mathematically, the total loss for 3D Gaussians optimization is defined as:EQUATIONwhere L shape is our introduced regularization loss on ellipsoidal eccentricity, 𝛽 is a weighting factor that balances L 1 and L D-SSIM , and 𝜂 is another weighting factor that balances the error loss and the regularization loss.Compared with the latest works on cinemagraph generation, the benefits of representing the scene with 3D Gaussians and optimizing it using Eq. ( 5) are three-fold. First, we can take as input multiple images from different viewing directions instead of just one single image, which is advantageous for reconstructing the intricate 3D geometries of the observed scene. Second, we present an auxiliary regularization term on the shape of 3D Gaussians to mitigate the artifact issue, which will be empirically validated in Sec. 5.4. Third, we can efficiently construct the subsequent Eulerian motion field by exploiting the distance relationships among 3D Gaussians. 3D Gaussians Separation. After learning the reconstruction of the 3D Gaussian point cloud, we manually annotate the objects that are anticipated to have motion effects within images in the training set. Next, we utilize SAGA [5] , an interactive segmentation approach for 3D Gaussians, to create a mask for the point cloud. This mask is introduced to segregate the 3D Gaussian point cloud into static and dynamic components. Concretely, let 𝐼 represent the multi-view images of the training set and 𝐴 the manual annotations on these images, the binary mask M is obtained by:EQUATIONFor each Gaussian point 𝐺 𝑖 , if its corresponding mask value 𝑚 𝑖 is 1, it will participate in the subsequent construction of Eulerian motion field; otherwise, it remains stationary. This separation process offers a flexible way to concentrate exclusively on modeling the dynamic 3D Gaussians. For brevity, we reuse the notation G to denote dynamic 3D Gaussians in the next subsections.The point cloud of 3D Gaussians offers an unstructured representation of a scene; however, the scene it depicts usually exhibits a structured geometry. An illustrative example is observing a flag waving in the wind: if a corner of the flag moves in a specific direction, the entire flag is likely to exhibit similar motion patterns. This local coherence of motion originates from the physical interconnection within the flag. Generally, if one point of an object moves, the surrounding points may also show a similar trend of movement due to the local consistency of geometry.In consideration of the analysis above, we borrow the concept from supervoxel to preserve the local consistency of geometry. The basic idea of supervoxels involves clustering 3D points according to spatial proximity and feature similarity. Inspired by previous works on supervoxel segmentation [25, 36] , we introduce a clustering method for 3D Gaussians, termed SuperGaussian. Concretely, let G ≔ {𝐺 𝑖 |𝑖 = 1, • • • , 𝑁 } be the set of 3D Gaussians in a point cloud and C ≔ {𝐶 𝑘 |𝑘 = 1, • • • , 𝐾 } the set of clusters, where 𝑁 is the total number of points and 𝐾 is total number of clusters. We first partition the scene into voxels based on a voxel resolution 𝑅, and then randomly select a seed Gaussian point within each of the non-empty voxels (i.e., voxels that contain at least one Gaussian point). Let SG(•) represent the SuperGaussian model that assigns a clustering label to a Gaussian point, and SG * the optimized Super-Gaussian. Clustering is achieved through the optimization of the following objective function:EQUATIONwhere 𝑘 ′ is the corresponding index of the seed Gaussian point for cluster 𝑘. The metric function 𝐷 (•, •) is defined as:EQUATIONwhere 𝑓 𝑖 and 𝑓 𝑗 are the features of the Gaussian points 𝐺 𝑖 and 𝐺 𝑗 from an autoencoder [10, 40] , 𝑝 𝑖 and 𝑝 𝑗 are the positions of the Gaussian points, 𝜇 is a weighting factor that balances the importance of features and positions, and 𝑅 stands for the resolution of supervoxels. With the seed points selected as centers, we gradually search outwards, applying the metric function 𝐷 (•, •) to identify the appropriate cluster for each point. This iterative process continues until all 3D Gaussian points are successfully assigned to a cluster. SuperGaussian can be seen as a variant of k-means, but enjoys the following advantages. On one hand, the seeding procedure is implemented by selecting seed Gaussian points in each non-empty supervoxels, so that the seeds are almost uniformly distributed across the scene. On the other hand, we incorporate the attributes of the learned 3D Gaussians into the distance measure, which ensures that the clustering algorithm will converge in just a few iterations. We experimentally find that only one iteration is enough to achieve a satisfactory clustering result, which makes SuperGaussian even more efficient.Motion field estimation in natural scenes is a challenging task due to the scarcity of comprehensive datasets, particularly for point clouds. A significant limitation arises from the difficulty in predicting subsequent scene flow based solely on static point cloud data. In response to this challenge, we propose a hypothesis: Similar objects generally have similar movement trends. The rationale behind this hypothesis stems from the observation that objects within natural scenes often exhibit collective behavior or tend to interact with one another in predictable ways. For instance, when a breeze blows through a forest, leaves on nearby trees tend to move in unison, following a similar direction. Building upon this hypothesis, we devise an efficient estimation method for the Eulerian motion field according to the similarity relationships among clusters.Sparse Velocity Field Estimation. We derive an initial velocity field by moving each cluster to its nearest neighbor. To achieve this, we first need to identify a global feature for each cluster, on which a similarity metric can be performed. Considering the disorder of Gaussian points in each cluster, we adopt maximum pooling, a permutation-invariant and symmetric function, to extract the global feature 𝑓 𝐶 𝑖 for each cluster 𝐶 𝑖 . Next, we obtain a similarity matrix S ≔ {𝑠 𝑖 𝑗 } by computing the cosine similarity between two global features:EQUATIONThe most similar cluster 𝐶 ĩ for each cluster 𝐶 𝑖 is then identified with the highest cosine similarity:EQUATION)To create the velocity field, we define a center point with position p𝑖 for each cluster, which is calculated by taking the average of positions of all Gaussian points within the cluster. The velocity field v sparse ≔ {𝑣 𝑠 𝑖 } 𝐾 𝑖=1 is then achieved by moving one cluster to another with the highest similarity:EQUATIONNote that the initial velocity field is fairly sparse due to the limited number of clusters. This sparse velocity field is prone to overfitting as they lack the granularity necessary for smooth representation across the entire spatial domain. This limitation may become more prominent when the motion field extends into unfamiliar areas, which can greatly increase the risk of divergence and discontinuity.Next, we will introduce our solution to this issue.Dense Velocity Field Estimation. We opt for Kriging interpolation [35] to estimate a dense velocity field from the sparse one calculated in Eq. (11) . Kriging is a geostatistical interpolation method that is widely used for estimating the value of a variable at an unmeasured location based on the values of neighboring positions with known values. Concretely, we compute the velocity at each position of a 3D Gaussian point via Kriging interpolation, which results in a dense velocity field v dense ≔ {𝑣 𝑑 𝑖 } 𝑁 𝑖=1 . The dense velocity 𝑣 𝑑 𝑖 at position 𝑝 𝑖 is a weighted sum of the observed sparse velocities v sparse , where the weights are derived from a variogram. In practice, we employ a standard spherical model to solve the variogram. The Kriging interpolation procedure excels in preserving the smoothness of the velocity field, and we will empirically demonstrate the effectiveness of it in Sec. 5.4.Eulerian Motion Field Estimation. As the Eulerian motion field captures velocities at all spatial positions, it is crucial to ensure smoothness to prevent scene tearing during deformation.To this end, we adopt an MLP to estimate the Eulerian motion field ì 𝐸 𝐺 for enhanced smoothness. The inputs to the MLP are the spatial positions { p𝑖 } 𝐾 𝑖=1 and {𝑝 𝑖 } 𝑁 𝑖=1 , and it predicts the velocities at the corresponding positions. The training process of MLP is fully supervised by the sparse and dense velocities v sparse and v dense . During inference, we can estimate the velocity 𝑣 = ì 𝐸 𝐺 (𝑝) ≔ MLP(𝑝) at any given position 𝑝 in 3D space via the Eulerian motion filed.Given the estimated Eulerian motion field ì 𝐸 𝐺 , we can efficiently ascertain the motion of each 3D Gaussian point. However, the convergence of each point's movement within the Eulerian field may not be guaranteed over time. This is because an Eulerian field is not necessarily an irrotational field, which may lead to unexpected scene tearing. To address this challenge, inspired by the bidirectional animation technology [14, 19, 30] in image space, we make an effort to escalate it from 2D space to 3D space, ensuring that each point stays within a reasonable range of motion and will ultimately revert to its initial position.For each 3D Gaussian point 𝐺 𝑖 , we introduce a vector 𝜓 ∈ R 3 to regulate the magnitude of motion on each axis. The vector is simultaneously controlled by the lengths of sides of the scene's axis-aligned bounding box ℎ ∈ R 3 , the number of video frames 𝑇 , and a hyperparameter 𝜔 that controls the amplitude of motion, i.e., 𝜓 = 𝜔 𝑇 •𝑒 -ℎ . The position 𝑝 𝑖 (𝑡) of the point 𝐺 𝑖 at time 𝑡 is calculated through Euler integration from 0 to 𝑡:𝑝 𝑖 (𝑡) = 𝑝 𝑖 (0) + 𝑡 -1 ∑︁ 𝜏=0 𝜓 ⊙ ì 𝐸 𝐺 (𝑝 𝑖 (𝜏)) ,where 𝑝 𝑖 (𝜏) = 𝑝 𝑖 (𝜏 -1)EQUATIONHere ⊙ signifies the Hadamard product. To achieve forward and backward animation in 3D space, we calculate the spatial positions 𝑝 𝑖 (𝑡) and 𝑝 𝑖 (𝑡 -𝑇 ) of each 3D Gaussian point 𝐺 𝑖 at frames 𝑡 and 𝑡 -𝑇 using Eq. ( 12). Then we derive its final position p𝑖 (𝑡) at time 𝑡 through linear interpolation:EQUATIONwhere 𝛼 = (1 -𝑡 𝑇 ). In this way, we obtain a temporal sequence of positions for the 3D Gaussian point cloud. Given the camera parameters, including its position and viewing angle, we employ the rendering pipeline outlined in Sec. 3.1 to render each frame, which ultimately forms a seamlessly looping video. 
experiments 5.1 datasets: 
 We utilize a combined synthetic dataset to comprehensively evaluate our proposed method. Part of the dataset is from the static NeRF synthetic dataset [32] . Due to the scarcity of dynamic datasets depicting natural scenes, we also produce a brand new dataset that showcase dynamic nature scenes using Unity. This dataset follows the structure of the NeRF synthetic dataset and is accompanied by a corresponding synchronized ground truth video, which has a resolution of 900 × 900 pixels and consists of 48 frames. Noted that the video is non-loopable. Our experiments are completed on a single NVIDIA GeForce RTX 4090 with the PyTorch framework [38] . We empirically set 𝛽 = 0.2 and 𝜂 = 0.9 in Eq. ( 5), and the total number of training iterations is 50, 000. The autoencoder that projects 3D Gaussian points into feature space is designed based on PointNet [40] , where the architecture of the decoder is symmetric to that of the encoder. During the clustering of 3D Gaussian points, we adjust the scene resolution 𝑅 according to 𝜆 max(ℎ) to balance granularity and detail preservation, where 𝜆 = 0.04 and 𝜇 = 0.5 in Eq. ( 8). The MLP utilized to depict the Eulerian motion field has two hidden layers of size 128 and 64 respectively, with positional encoding applied to the input. For the amplitude control of motion, we set 𝜔 = 1.2. Regarding final video rendering, we set the video duration 𝑇 to 48 frames, with a resolution specified as 900 × 900.To the best of our knowledge, we present the first work that creates cinemagraphs in the authentic 3D space. For fair comparison with state-of-the-art work on synthesizing cinemagraphs in 2D image space, here we render our cinemagraphs in 2D image space as well. However, we remark that our proposed method is more advanced, in the sense that our method is performed in 3D space and is capable of rendering from any viewpoint, which is impossible for any of previous related works.Quantitative Evaluation. We perform a quantitative evaluation on the self-produced synthetic dataset mentioned in Sec. 5.1. To rigorously evaluate the effectiveness of our approach, we conduct a quantitative comparison between optical flow maps generated by our method and that of a reference video. Optical flow maps serve as a fundamental tool in video analysis, offering insights into the dynamic changes occurring both temporally and spatially within a video sequence. We commence the evaluation process by analyzing the optical flow starting from the second frame onwards. The optical flow between each frame and its preceding frame is computed and averaged to obtain a comprehensive optical flow map. We assess the quality of results using three popular image quality assessment metrics, i.e., PSNR, SSIM, and LPIPS [54] . The quantitative comparison results of the average optical flow maps are presented in Table 1 . It is evident from the table that our method's optical flow maps closely resemble the reference optical flow maps, suggesting that our approach generates more realistic motion. We also use Fréchet Video Distance (FVD) [46] to evaluate the quality of the generated video. The comparison results can be seen in Table 2 , which further demonstrates the advantage of our method.Qualitative Evaluation. Visual comparisons between our proposed method and 3D Cinemagraph [19] are shown in Fig. 3 . Despite being labeled as 3D Cinemagraph, their method primarily caters to fluid scenes and lacks precise geometric information in 3D space. Consequently, when handling geometrically continuous objects, their results can manifest severe artifacts, ultimately disrupting the scene's structure. In contrast, our method excels in preserving the object's geometric continuity, evident in the natural distortion of objects like a flag without tearing. Notably, as seen from the optical flow maps shown in the last row, their method typically features objects moving uniformly in one direction, whereas our approach demonstrates objects engaging in a periodic reciprocal motion, which is more aligned with real-world scenarios.User Study. We conducted a user study involving 110 participants in answering which method produces videos that are more visually 3 , showing that most participants preferred the results derived from our method.In this section, we conduct ablation studies to systematically analyze the impact of various components of our proposed method.Eccentricity Regularization. As illustrated in Fig. 4 , eccentricity regularization significantly reduces the occurrence of artifacts such as burrs and glitches in the scene during deformation. Sharp 3D Gaussians at the edges of the flag are prone to protrude during deformation, making it difficult for the originally connected points to remain in close proximity, consequently leading to the formation of burrs. By incorporating eccentricity regularization, the shape of the 3D Gaussians is constrained to closely resemble a sphere, thereby alleviating this phenomenon.Interpolation Methods for Dense Velocity Vectors. The choice of interpolation function for dense motion vectors directly affects the smoothness and accuracy of the motion field, which in turn influences the quality of the rendering result. The ablation results of interpolation is shown in Fig. 5 . As can be seen, the objects are more complete and the motion of objects is more continuous when using Kriging interpolation, compared to no interpolation or RBF interpolation.Impact of Voxel Resolution on Clustering. Voxel resolution directly impacts the granularity of spatial representation, thereby affecting the accuracy and granularity of cluster formation. As illustrated in Fig. 6 , excessive resolution results in overly coarse scene division (Fig. 6a ), leading to unrelated objects being grouped together, whereas overly small resolution causes excessive partitioning (Fig. 6c ), potentially leading to loss of some higher-level information. We empirically adopt an appropriate voxel resolution that strikes a balance between scene segmentation and the preservation of scene information (Fig. 6b ).Motion Amplitude Control. The magnitude of motion for each 3D Gaussian point is controlled by 𝜓 in Eq. ( 12), with 𝜔 serving as a hyperparameter to regulate 𝜓 , and the impact of 𝜔 is shown in Fig. 7 . A higher value of 𝜔 corresponds to a larger motion range for each 3D Gaussian point, resulting in more pronounced dynamics across the entire scene. However, it should be noted that excessively large values of 𝜔 may disrupt the continuity of the scene. 
conclusion: 
 In this paper, we introduce LoopGaussian, a novel framework for generating authentic 3D cinemagraphs from multi-view images of static scenes. By leveraging 3D Gaussian Splatting and inherent scene self-similarity with an Eulerian velocity field, our method enables natural, loopable motion trajectories without extensive pre-training. LoopGaussian surpasses previous methods that are restricted to 2D image space, as we reconstruct the 3D geometry of the observed scene. Besides, our method enables rendering from any viewpoint and ensures consistency across multiple perspectives. Experiments demonstrate the effectiveness of our method in simulating motion for soft, non-rigid objects.  
 
 Appendix:  


Decision: The paper presents a novel technique for creating animations from static scenes. The paper is well-written and the method is adequately described. Overall, it is a good contribution to ACM MM.
The presented LoopGaussian method is innovative (R1, R2, R3). The paper presents a new technique to animate static images using 3D Gaussian modeling and Eulerian motion fields. The LoopGaussian method is well explained and clearly presented. Using off-the-shelf components is well-advocated and justified (R1, R2, R3). The presented method is tested through adequate experiments. The results are convincing, and the animations are of good quality and visually pleasing. A comprehensive ablation study is also presented.
Most comments by the reviewers were addressed in the rebuttal, however, a couple of suggestions to further improve the paper were also made by the reviewers.  A comparison with more exiting similar techniques would justify the claims as suggested by reviewer R1. Moreover, if there are any limitations of the presented method, they must be discussed (R1, R2).

Reviews: review: paper_summary: paper_summary: This paper aims to generate cinemagraphs based on 3D scene, where the 3D scene is represented by multi-view images forming a 3D Gaussian point cloud. Cinemagraphs are a combination of motion and stillness, commonly used for dynamic presentations of 2D images. This paper's unique contribution is generating dynamic effects based on 3D static scene generation technology. 

Additionally, I believe this paper not only implements the dynamic techniques commonly used in 2D images within a 3D scene but also addresses the issue of animating 3D Gaussian point clouds. This could have a significant impact and provide inspiration in the field.
strength_weakness: strength_weakness: 1.Rich and precise vocabulary with clear and concise expression.
2.Provides new ideas for animating static 3D scenes.
3.Proposes a new method for scene presentation by creating cinemagraphs in 3D scenes.
limitations: limitations: 1.The paper devotes too much description to the Gaussian spray technique. More space could be allocated to describing the contributions of this paper.
2.There are few comparisons with existing methods.
3.The paper does not discuss its own limitations.
paper_summary: paper_summary: The paper presents LoopGaussian, a novel method for creating 3D cinemagraphs by leveraging 3D Gaussian modeling and Eulerian motion fields. Cinemagraphs combine still photography with subtle motion to create visually captivating experiences. Traditional methods are limited to 2D image space and lack depth information. LoopGaussian addresses this limitation by reconstructing 3D Gaussian point clouds from multi-view images and employing shape regularization to prevent artifacts. The method then uses an autoencoder tailored for 3D Gaussian to project the point clouds into feature space, followed by clustering using SuperGaussian. By deriving an Eulerian motion field, the method animates the 3D Gaussian points, generating high-quality, seamlessly loopable 3D cinemagraphs. Experimental results demonstrate the effectiveness of this approach.
strength_weakness: strength_weakness: 1. The proposed LoopGaussian method is highly innovative, effectively extending the concept of cinemagraphs from 2D to 3D space using 3D Gaussian modeling and Eulerian motion fields. This approach represents a significant advancement in the field.
2. The methodology is detailed and well-explained, with each step of the process clearly described. The use of shape regularization, the tailored autoencoder for 3D Gaussian, and the SuperGaussian clustering are well-justified and effectively implemented.
3. The experimental results are comprehensive and convincingly demonstrate the effectiveness of the proposed method. The generated 3D cinemagraphs are high-quality and visually appealing, showcasing the practical applicability of LoopGaussian.
limitations: limitations: 1. This paper introduces a new dataset, however, the specific information about this data set is too little, such as what kind of scenes it mainly contains, how many scenes there are, etc.
2. What's the limitation of the proposed method?
3. What's the computational cost of the proposed method?
paper_summary: paper_summary: This paper reconstruction a static scene with 3DGS and use a regularization term to encourage ellipsoidal eccentricity.
Then the authors propose the concept of SuperGaussian to enhance the structure of Gaussians representaiton.
To animate the Gaussians, this paper distills the estimated sparse velocity field and dense velocity field into an MLP to help estimating the Eulerian motion field. Finally, the animation is obtained through interpolation.
strength_weakness: strength_weakness: This paper issues an interesting problem, which is like a 4D generation but is highly consistent with original scene.
Merely idea itself is novel enough for acceptance.
It's easy to agree that the geometric information leads to geometric consistency.
Methods are self-contained. And ablation study has a rich variety.
limitations: limitations: Lack of quantitative ablation study:
Although ablation study is rich, it lacks quantitative results for evaluaton.

Potential limited comparison in experiment:
The comparison is limited, with the compared method only takes one images as input, while this paper needs multi-view images for reconstruction.
Although this is not compulsory, it's better to compare with more recent methods with also multi-vew images as inputs, such as PhysGaussian.

Lack of video results in supplementary:
Since this paper proposes a methods with a loopable video as results. This is really uncompelling when no video results are submitted