MiraGe: Editable 2D Images using Gaussian Splatting

## Abstract

Abstract Implicit Neural Representations (INRs) approximate discrete data through continuous functions and are commonly used for encoding 2D images. Traditional image-based INRs employ neural networks to map pixel coordinates to RGB values, capturing shapes, colors, and textures within the network’s weights. Recently, GaussianImage has been proposed as an alternative, using Gaussian functions instead of neural networks to achieve comparable quality and compression. Such a solution obtains a quality and compression ratio similar to classical INR models but does not allow image modification. In contrast, our work introduces a novel method, MiraGe, which uses mirror reflections to perceive 2D images in 3D space and employs flat-controlled Gaussians for precise 2D image editing. Our approach improves the rendering quality and allows realistic image modifications, including human-inspired perception of photos in the 3D world. Thanks to modeling images in 3D space, we obtain the illusion of 3D-based modification in 2D images. We also show that our Gaussian representation can be easily combined with a physics engine to produce physics-based modification of 2D images. Consequently, MiraGe allows for better quality than the standard approach and natural modification of 2D images.

## 1 Introduction

Figure: Figure 1: MiraGe encodes 2D images with parameterized Gaussians, enabling high-quality reconstruction and real-life-like modifications. Selected part of image can be transformed in 3D space, creating a 3D effect, with a physics engine controlling movement and interactions.
Refer to caption: x1.png

Recent research has increasingly emphasized human perception and the understanding of the world through this lens (Lu, ; Davoodi et al., ). In line with this trend, we introduce a model that encodes 2D images by simulating human interpretation. Specifically, our model perceives a 2D image as a human would view a photograph or a sheet of paper, treating it as a flat object within a 3D space. This approach allows for intuitive and flexible image editing, capturing the nuances of human perception while enabling complex transformations (see Fig. 1).

Gaussian Splatting (3DGS) framework models the structure of a 3D scene using Gaussian components (Kerbl et al., ). In the 2D domain, GaussianImage (Zhang et al., ) has shown promising results in image reconstruction by efficiently encoding images in the 2D space, with a strong focus on model efficiency and reduced training time.
Regrettably, the GaussianImage lacks the capability for human-driven adjustments of scene objects, which is a crucial aspect of 3DGS.
In our work, we address this by introducing the MiraGe(^1^11[https://github.com/waczjoan/MiraGe/blob/main/README.md](https://github.com/waczjoan/MiraGe/blob/main/README.md)) model, which encodes 2D images through the lens of human perception, bridging the gap between 2D image representation and 3D spatial understanding (see Fig. 2).

Figure: Figure 2: MiraGe employs 3D flat parameterized Gaussians in 3D space to encode 2D images, representing each flat Gaussian as three points, forming a cloud of triangles called a triangle soup. This representation enables real-time manipulation of the 3D triangle/point clouds, allowing for flexible, real-world modifications. The model seamlessly integrates with a physics engine, enhancing its applicability in dynamic environments.
Refer to caption: extracted/5895312/imgs/2dgaussians_schema_1_drawio.png

Building on the foundational idea that humans intuitively can perform transformations on photographs–primarily through affine transformations and bending them beyond the 2D plane–we introduce a novel approach using flat Gaussians with GaMeS parameterization (Waczyńska et al., ). This allows our model to facilitate image editing within 2D and 3D spaces. In particular, our framework enables changes in the image’s perspective with minimal effort. Perspective adjustments, typically complex, are simplified by direct editing in the third dimension (see Fig. 3).

Figure: Figure 3: MiraGe simplifies intuitive editing of image, allowing transformations such as adjusting the tilt of a hand with minimal complexity. This is achieved by modifying the object along the third dimension.
Refer to caption: extracted/5895312/imgs/odchylaniereka_drawio_1.png

In addition to classical edits, our model has the unique capability of interfacing with physics engines, enabling applications that enhance the realism and immersiveness of animations (Jiang et al., ). MiraGe treats the physics engine as a black box and offers three distinct methods for controlling Gaussians, i.e., 2D, Amorphous and Graphite. For 2D representation (2D-MiraGe) we used Taichi_elements, for 3D representation (Amorphous-MiraGe, Graphite-MiraGe) we use Blender. This flexibility makes our model highly applicable to various fields, such as computer graphics for populating spatial interfaces, where realistic, physics-factual 2D animations can be incorporated (Tadeja et al., ).

Embedding 2D images in 3D space allows for seamless integration of 2D and 3D objects, enabling the creation of dynamic backgrounds or interactive elements within animated scenes. This versatility extends to applications such as virtual reality, where 2D images can function as backdrops (Yin et al., ). This capability opens up new avenues for creative composition, offering a powerful toolset for users. The novelty of this work lies in its ability to enable easy, intuitive 3D transformations and integrations within a traditionally 2D framework, expanding the possibilities for both image editing and animation (see Fig. 4).

Since high-quality image reconstruction is critical in animation, we compared MiraGe with other models, showing our state-of-the-art performance in the image reconstruction task. Our results demonstrate that the model operates in real-time, though this comes with a trade-off in terms of model size (see Fig. 5).

Figure: Figure 4: Two images were encoded using the MiraGe model on distinct planes within a 3D space. This setup allows for seamless integration of the encoded images, resulting in a collage-like composition. Moreover, the model facilitates editing capabilities, as illustrated here with modifications to the background image (the rear plane).
Refer to caption: x2.png

It is worth highlighting that flat 3D Gaussians can be utilized for 2D images in four distinct scenarios, with modifications that emphasize how controlling the Gaussians during training affects the perspective of viewing each image (see Fig. 6).

The following constitutes a list of our key contributions:

- •
We introduce the MiraGe model, which represents 2D images using flat Gaussian components, achieving state-of-the-art reconstruction quality.
- •
MiraGe enables the manipulation of 2D images within 3D space, creating the illusion of 3D effects.
- •
We integrate MiraGe with a physics engine, enabling physics-based modifications and interactions for both 2D and 3D environments

Figure: Figure 5: MiraGe outperforms other models in the image reconstruction task. The following columns present ground truth image, GaussianImage reconstruction and MiraGe reconstruction. The bottom row graphics show the difference between the ground truth image and the outcomes of subsequent methods.
Refer to caption: extracted/5895312/imgs/reconstruction_comparison_drawio.png

## 2 Related Works

Our work builds on several key research areas, including image reconstruction techniques, Gaussian-based representations and Gaussian animation frameworks.

One rapidly growing area in image reconstruction is Implicit Neural Representations (INRs), which have attracted significant attention for their ability to model continuous signals, such as images, through neural networks (Klocek et al., ). INRs encode spatial coordinates and map them to corresponding values, such as RGB color, allowing for highly compact and efficient representations (Xie et al., ). This has led to the development of several specialized models for image INR, such as SIREN (Sitzmann et al., ) with the novelty of sine used as a periodic activation function to tackle the problem of complex image signals. Fourier feature mapping was proposed in (Tancik et al., ) as another answer to the difficulty of aligning the multilayer perceptron (MLP) predictions with high-frequency pictures. Interestingly, authors of WIRE (Saragadam et al., ) have leveraged continuous complex Gabor wavelets to capture visual signals with decent quality. The growing field of research resulted in further improvements of already existing solutions, e.g. in (Liu et al., ), certain limitations of SIREN, namely the arising capacity-convergence gap, were successfully alleviated with the idea of variable-periodic activation functions. Yet another worth noting work from this area is (Müller et al., ) with INR solution designed to effectively perform on modern computer architecture utilizing a simple data structure concept of hashmap to offer speed-oriented image representation with high fidelity of the outcomes.

Alternative approaches to INRs were presented in GaussianImage (Zhang et al., ). Instead of natural networks, the authors propose to approximate 2D images using 2D Gaussian components. In practice, GaussianImage is a 2D version of 3DGS (Kerbl et al., ) that uses 2D Gaussians instead of their 3D version and a simplified rendering procedure. Thanks to such a modification, the GaussianImage is invariant to the order of Gaussian components. Therefore, such a model is numerically efficient.

Figure: Figure 6: Flat 3D Gaussians can be utilized for 2D images in four distinct scenarios. Modifications demonstrate how the perspective of viewing a 2D image can change in each case. We establish a baseline using the GaMeS framework with Gaussian initialization on the XZ plane. Our results show that controlling Gaussian positions during learning enables various applications. The model employs a mirror setup during training, with the Amorphous configuration achieving the best results for image reconstruction and 3D analysis. The 2D model represents images on a single plane, allowing 2D physics engines like Taichi_elements to be used, but it does not support 3D modifications. In contrast, the Graphite model operates across multiple planes, making it ideal for 3D spatial reasoning and image combination.
Refer to caption: x3.png

GaussianImage represents each pixel color as a weighted sum of 2D Gaussians. The training procedure is similar to 3DGS without pruning. The authors show that such representation gives a similar reconstruction quality to classical INR models and is able to obtain a high compression ratio and fast rendering.

The representation and editing of objects using Gaussians is a well-explored topic in 3D graphics. In this field, meshes can be modified to simulate Gaussian editing (Guédon & Lepetit, ; Huang et al., ), or Gaussians can be directly parameterized and manipulated to achieve specific outcomes (Waczyńska et al., ; ). This approach enables flexible and continuous deformations, offering an intuitive method for controlling object shapes and rendering properties, which has proven particularly useful in tasks like texture mapping, surface smoothing, and dynamic simulations.

Gaussians enable precise and flexible editing of objects, providing continuous control over shapes and transformations. Moreover, integrating physics engines enhances these capabilities, allowing for more sophisticated and physically consistent modifications, such as simulating realistic interactions, deformations and movements in 3D environments. (Xie et al., ; Borycki et al., ).

## 3 MiraGe: Editable 2D Images using Gaussian Splatting

Here, we describe in detail the inner workings of our MiraGe model. We start by presenting classical 3DGS. Next, we present GaMeS-based (Waczyńska et al., ) parametrization of flat Gaussians. In the end, we present our MiraGe and how it relates to prior works.

Figure: Figure 7: MiraGe use GaMeS (Waczyńska et al., ) representations of flat Gaussian by triangle soup. Therefore, we can use real-life modification by moving points.

#### 3D Gaussian Splatting

3DGS models 3D scene by a set of Gaussian components with color and opacity:

$$ $\mathcal{G}=\{(\mathcal{N}({\bf m}_{i},\Sigma_{i}),\sigma_{i},c_{i})\}_{i=1}^{ p},$ $$

defined by their mean (position) ${\bf m}_{i}$, covariance matrix $\Sigma_{i}$, opacity $\sigma_{i}$, and color $c_{i}$, which is represented using spherical harmonics (SH) (Fridovich-Keil et al., ).

During the rasterization stage, the 3DGS produces a sorted Gaussian list based on the projected depth information. Then, the $\alpha$-blending method is used to create the image. We refine the Gaussian parameters, color, and opacity in the training phase according to the reconstruction cost function. Training the Gaussian component is challenging. Consequently, the authors implement additional strategies for reducing and multiplying components. Gaussians with low opacity are removed, while those that change rapidly during optimization are multiplied. These strategies make the 3D Gaussian approach very efficient and capable of generating high-quality renders.

Figure: Figure 8: We integrate MiraGe with Material Point Methods (MPM) to achieve realistic alterations of 2D images. The initial column presents the original image, the subsequent two columns display renders captured midway through the simulation, and the final column shows the outcome of the full simulation. The colored lines in the last column trace the paths of randomly chosen points from the simulation.
Refer to caption: x5.jpg

#### GaMeS Parametrization of Gaussian Component

In MiraGe, we use flat Gaussian components in 3D space.
In such a model we use Gaussian components with a covariance matrix $\Sigma$, factored as:

$$ $\Sigma=RSSR^{T},$ (1) $$

where $R$ is the rotation matrix, and $S$ is a diagonal matrix containing the scaling parameters. However, we force one of the scale parameters to be zero.
Consequently, we obtain a collection of flat Gaussian:

$$ $\mathcal{G}=\{(\mathcal{N}({\bf m}_{i},R_{i},S_{i}),\sigma_{i},c_{i})\}_{i=1}^ {p},$ (2) $$

where $S=\mathrm{diag}(s_{1},s_{2},s_{3})$, with $s_{1}=\varepsilon$, and $R$ is the rotation matrix defined as
$R=[{\bf r}_{1},{\bf r}_{2},{\bf r}_{3}]$, with ${\bf r}_{i}\in R^{3}$.
In such a case, we can use GaMeS (Waczyńska et al., ) parametrization to represent flat Gaussian by triangle-face mesh. This mapping is denoted by $\mathcal{T}(\cdot)$. When applied, this parametrization generates a set of triangles labeled as triangle soup.

To outline the GaMeS parameterization, consider a Gaussian component $\mathcal{N}({\bf m},R,S)$, characterized by the mean ${\bf m}$, the rotation matrix $R=[{\bf r}_{1},{\bf r}_{2},{\bf r}_{3}]$ and the scaling matrix $S=\mathrm{diag}(\varepsilon,s_{2},s_{3})$. Then its face representation $\mathcal{N}(V)$ is based on a triangle:

$$ $V=[{\bf v}_{1},{\bf v}_{2},{\bf v}_{3}]=\mathcal{T}({\bf m},R,S)$ (3) $$

with the vertices defined as:
${\bf v}_{1}={\bf m},\;\;{\bf v}_{2}={\bf m}+s_{2}{\bf r}_{2},\;\;\textrm{and}
\;\;{\bf v}_{3}={\bf m}+s_{3}{\bf r}_{3}.$
Conversely, given a face (triangle) representation $V=[{\bf v}_{1},{\bf v}_{2},{\bf v}_{3}]$, we can recover the Gaussian component:

$$ $\mathcal{N}(\hat{\bf m},\hat{R},\hat{S})=\mathcal{N}(\mathcal{T}^{-1}(V))$ (4) $$

through the mean $\hat{\bf m}$, the rotation matrix $\hat{R}=[\hat{\bf r}_{1},\hat{\bf r}_{2},\hat{\bf r}_{3}]$, and the scaling matrix $\hat{S}=\mathrm{diag}(\hat{s}_{1},\hat{s}_{2},\hat{s}_{3})$, where the parameters are defined by the following formulas:

$$ $\hat{\bf m}={\bf v}_{1},\;\;\hat{\bf r}_{1}=\frac{({\bf v}_{2}-{\bf v}_{1}) \times({\bf v}_{3}-{\bf v}_{1})}{\|({\bf v}_{2}-{\bf v}_{1})\times({\bf v}_{3} -{\bf v}_{1})\|},$ (5) $$

$$ $\hat{\bf r}_{2}=\frac{({\bf v}_{2}-{\bf v}_{1})}{\|({\bf v}_{2}-{\bf v}_{1})\| },\;\;\hat{\bf r}_{3}=\mathrm{orth}({\bf v}_{3}-{\bf v}_{1};{\bf r}_{1},{\bf r }_{2}),$ (6) $$

$$ $s_{1}=\varepsilon,\;\;\hat{s}_{2}=\|{\bf v}_{2}-{\bf v}_{1}\|,\;\;\textrm{and} \;\;\hat{s}_{3}=\langle{\bf v}_{3}-{\bf v}_{1},\hat{\bf r}_{3}\rangle.$ (7) $$

Here $\mathrm{orth}(\cdot)$ denotes a single step of the Gram-Schmidt process (Björck, ).
Accordingly, the corresponding covariance matrix of a Gaussian distribution is given as $\hat{\Sigma}=\hat{R}\hat{S}\hat{S}\hat{R}^{T}$.

Figure: Figure 9: MiraGe can be integrated with Blender, by using flat 3D Gaussians in 3D space. The initial column presents the original image, the subsequent two columns display renders captured midway through the simulation, and the final column shows the outcome at the simulation’s conclusion. The colored lines in the last column trace the paths of 10 randomly chosen points from the simulation.
Refer to caption: extracted/5895312/imgs/teser_drawio_1_2_drawio.png

The parametrization enables control over the Gaussians’ position, scale, and rotation by manipulating the underlying triangle mesh. Applying transformations to the triangle directly alters the corresponding Gaussian, as illustrated in Fig. 7.

#### MiraGe

Figure: Figure 10: MiraGe enables modifying 2D images, such as adjusting the scene’s elements’ sizes.
Refer to caption: x6.jpg

In this work, we present an approach that leverages the concept of flat Gaussian distributions in 3D space to model a single 2D image as input. Our methodology is grounded in human visual perception. This perspective allows us to reframe the problem: instead of merely processing a pixel matrix, we interpret the images as objects with a fixed spatial configuration in a 3D environment.

We put the 2D image on the $XZ$ plane where the center is situated at axes origin $(0,0,0)$ with the fixed distance from the camera origin. In practice, the distance from the plane is a hyper-parameter. In our approach, we model flat objects within 3D space, where the camera distance parameter effectively controls the perceived scale of the object. This relationship allows for intuitive adjustments of object size based on the desired visual effect. For instance, increasing the camera distance can naturally expand the apparent size of background elements like distant mountains (Fig. 4), making it easier to represent them as larger objects without additional modeling complexity. While this feature is beneficial, it is not strictly necessary for most applications.

Figure: Figure 11: Comparison of PSNR obtained on a butterfly image from DIV2K dataset by MiraGe in comparison with GaussianImage (Zhang et al., ) and Gaussian Splatting (Kerbl et al., ). The lines with markers represent how the PSNR was changing during training. Different colors represent models trained with different numbers of Gaussians during initialization. The dashed black and orange lines represent the best results obtained during training by GaussianImage and GS, respectively. Vertical lines represent iteration, where MiraGe obtained better results than GaussianImage, the time in min:sec format above each line is the training time until this iteration. The experiment was performed on the rtx 4070 GPU.
Refer to caption: x7.png

We propose a method that situates the Gaussians within the $XZ$ plane, ensuring that the entire image remains visible under perspective projection. To achieve this, the possible range of $x$-values and $z$-values is calculated using the camera field of view. We first calculate the deviation from 0 0 on the $X$ axis using the similarity of triangles $\text{dev}_{z}=\text{cam}_{\text{dist}}\cdot\tan(0.5\cdot\text{Fov}_{\text{
vert}})$, where $\text{cam}_{\text{dist}}$ and $\text{Fov}_{\text{vert}}$ are camera distance from the $XZ$ plane and camera field of view respectively. The deviation in the $X$ axis can be then computed by multiplying this value by the camera aspect ratio.

Consequently, the initialization of Gaussians is consistently performed on the $XZ$ plane; however, we have opted to permit their movement within the 3D space. Drawing inspiration from three distinct models, we introduce three conceptual approaches for manipulating the spatial positioning of Gaussians.

#### Amorphous

The baseline approach how to control Gaussians is based on the classical GaMeS parametrization, initialized randomly on the $XZ$ plane, with the mean parameter’s $y$ coordinate set to zero:

$$ $\mathcal{G}=\{(\mathcal{N}([m_{1},0,m_{3}],[{\bf r}_{1},{\bf r}_{2},{\bf r}_{3 }],\mathrm{diag}(\varepsilon,s_{2},s_{3})),\sigma_{i},c_{i})\},$ (8) $$

where ${\bf m}=[m_{1},0,m_{3}]$ $S=\mathrm{diag}(s_{1},s_{2},s_{3})$, with $s_{1}=\varepsilon$, and $R$ is the rotation matrix defined as
$R=[{\bf r}_{1},{\bf r}_{2},{\bf r}_{3}]$, with ${\bf r}_{i}\in R^{3}$.

It should be highlighted that we only initialized the Gaussian component on the $XZ$ plane. During training, Gaussians can move amorphously in 3D space. We use the classical loss function $L_{1}$ combined with a D-SSIM term:

$$ $\mathcal{L}=(1-\lambda)\mathcal{L}_{1}(I,GS(I))+\lambda\mathcal{L}_{D-SSIM}(I, GS(I)),$ $$

where $I$ is the input image and $GS(I)$ is the constraint obtained by the Gaussian renderer. While this solution enables the modeling of images using a collection of triangles, often referred to as “triangle soup,” it proves insufficient for high-quality representations. During editing, significant artifacts emerge (Fig. 6–Baseline).

#### 2D

Building on the promising outcomes of GaussianImage, we strategically anchored all Gaussians onto the $XZ$ plane. This configuration allows us to effectively translate the flat image geometry into a spatial framework, bridging perceptual intuition. We set the mean of these components to have zero in the second coordinate. Moreover, we use the projection of flat Gaussians on a 2D plane. Unfortunately, orthogonal projection can produce artifacts. Therefore, we use a rotation of Gaussian components to lay on the $XZ$ plane. Since we use flat Gaussians to extract such rotation we can use a rotation matrix between two vectors to align the vector in 3D (Markley, ). We use the notation $\mathrm{Rot}(a,b)$ for the rotation matrix.

MiraGe on 2D plane is define by set of 3G parameterized Gaussian components:

$$ $\mathcal{G}=\{(\mathcal{N}([m_{1},0,m_{3}],[{\bf r}_{1},{\bf r}_{2},{\bf r}_{3 }]\mathrm{Rot}({\bf r}_{3},{\bf e}_{2}),\mathrm{diag}(\varepsilon,s_{2},s_{3}) ),\sigma_{i},c_{i})\},$ (9) $$

where ${\bf e}_{2}=[0,1,0]$, ${\bf m}=[m_{1},0,m_{3}]$, $S=\mathrm{diag}(s_{1},s_{2},s_{3})$, with $s_{1}=\varepsilon$, and $R$ is the rotation matrix defined as
$R=[{\bf r}_{1},{\bf r}_{2},{\bf r}_{3}]$, with ${\bf r}_{i}\in R^{3}$.

#### Graphite

Figure: Figure 12: Comparison of FPS obtained on a butterfly image from DIV2K dataset by MiraGe in comparison with GaussianImage (Zhang et al., ) and Gaussian Splatting (Kerbl et al., ). The experiment was performed on the rtx 4070 GPU.
Refer to caption: x8.png

Unfortunately, 2D-MiraGe produces artifacts when we use modification in 3D space (see the third row in Fig. 6). Such an effect is coursed by the Gaussians, which appear randomly according to the camera position. To solve such a problem and obtain the possibility of 3D modifications, MiraGe allows the Gaussians to leave the $XZ$ plane:

$$ $\mathcal{G}=\{(\mathcal{N}([m_{1},0,m_{3}]+\gamma{\bf e}_{2},[{\bf r}_{1},{\bf r }_{2},{\bf r}_{3}]\mathrm{Rot}({\bf r}_{3},{\bf e}_{2}),\mathrm{diag}( \varepsilon,s_{2},s_{3})),\sigma_{i},c_{i})\},$ (10) $$

where $\gamma$ is trainable parameter of translation scale along the vector ${\bf e}_{2}=[0,1,0]$, ${\bf m}=[m_{1},0,m_{3}]$, $S=\mathrm{diag}(s_{1},s_{2},s_{3})$, with $s_{1}=\varepsilon$, and $R$ is the rotation matrix defined as:
$R=[{\bf r}_{1},{\bf r}_{2},{\bf r}_{3}]$, with ${\bf r}_{i}\in R^{3}$. Such a model allows for the order of Gassians according to camera positions.

By leveraging parameterized Gaussians, we achieved precise manipulation of 2D images directly within their native 2D space, enabling targeted edits of segmented regions and transformations of complete scenes in easier way. While this approach demonstrated substantial promise, we observed significant artifacts when extending manipulations into the 3D domain, particularly along the $Y$-axis.

To address this challenge, we employ a novel approach utilizing two opposing cameras positioned along the $Y$ axis, symmetrically aligned around the origin and directed towards one another. The first camera is tasked with reconstructing the original image, while the second models the mirror reflection. The photograph is thus conceptualized as a translucent tracing paper sheet, embedded within the 3D spatial context. The reflection can be effectively represented by horizontally flipping the image, denoted as $\mathcal{M}(I)$. This mirror-camera setup enhances the fidelity of the generated reflections, providing a robust solution for accurately capturing visual elements.

The MiraGe is initialized according to equation Eqn. 8 and utilizes a cost function: $\mathcal{L}(I)+L(\mathcal{M}(I))$. We simultaneously model both the image and its mirrored reflection, as shown in the second row in Fig. 6.

Figure: Figure 13: MiraGe allows us to produce realistic modifications of small details like changing human facial expressions.
Refer to caption: extracted/5895312/imgs/czlowiek_drawio.png

After thorough experimentation, we find that our model, Amorphous-MiraGe, utilizing a mirror camera, achieves state-of-the-art reconstruction results. This model demonstrates significant advantages over alternative methods in terms of both performance and outcome quality.

#### Editability

The ability to manipulate Gaussians based on their spatial positioning empowers MiraGe to effectively edit 2D images. When utilizing a mirror camera, the quality of the resulting images is sufficiently high, enabling the parameterization and animation of Gaussians to significantly reduce artifacts. Our findings demonstrate that our model facilitates the animation of both segmented objects and entire scenes. Users can create manual animation, or leverage automated processes using physics engines like Taichi_elements or Blender (8, 9). To incorporate MiraGe with the 2D physics engine, we use 2D-MiraGe (Fig 8).

We argue that the Graphite-inspired model allows the creation of attractive compositions made of multiple images that effectively present the positive attributes of the layered structure, like Graphite, through the strategic positioning of Gaussians.

## 4 Experiments

We split the experimental section of our paper into two main parts. First, we demonstrate that our approach achieves high-quality 2D reconstruction by comparing it with existing models. Second, we highlight the versatility of our method in image editing both full scenes (Fig. 10) and selected objects (Fig. 1, 7), presenting examples of user-driven modifications and demonstrations involving physical simulations (Fig. 8, 9).

Figure: Figure 14: MiraGe model allows modifications in 3D space, but the model is limited by 2D images, which was used in training. When we move some elements from the foreground, we are not able to see the background since the model only reconstructs objects. After modification, we can see artifacts on the hind paw of the presented animals.

#### Reconstruction quality

To assess image quality, we utilized two established metrics, i.e., PSNR and MS-SSIM (Wang et al., ). These metrics measure the distortion between the reconstructed images and the ground truth.

Our image reconstruction assessment utilizes two widely-recognized datasets. Specifically, we employ the Kodak dataset(^2^22[https://r0k.us/graphics/kodak/](https://r0k.us/graphics/kodak/)), which includes 24 images at a resolution of $768\times 512$, alongside the DIV2K validation set (Agustsson & Timofte, ), which involves $2\times$ bicubic downscaling and comprises 100 images with sizes ranging from $408\times 1020$ to $1020\times 1020$.
The dataset was selected to facilitate direct comparison with the work of GaussianImage.

As a baselines we use competitive INR methods GaussianImage (Zhang et al., ), SIREN (Sitzmann et al., ), WIRE (Saragadam et al., ), I-NGP (Müller et al., ), and NeuRBF (Chen et al., ).

In Tab. 1, we demonstrate the performance outcomes of different methods on the Kodak and DIV2K datasets. We see that our proposition outperforms the previous solutions on both datasets. The quality measured by both metrics shows significant improvement compared to all the previous approaches. Fig. 11 illustrates a general trend observed during training in contest of image reconstruction. The selection of hyperparameters, including the number of iterations, was inspired by the principles of 3DGS. We provide ablation studies and extensive numerical analyses in the appendix for further insights.

It is important to note that although our model takes longer to train, it quickly achieves better results than GaussianImage. The trend we observe is illustrated in Fig. 11, which also includes the number of initial Gaussians, indicating how densely the space has been filled. We see a clear upward trend in performance as the density of the Gaussian initialization increases.

Due to our particular focus on animation, we analyzed FPS trends to benchmark real-time performance. Fig. 12 shows that while our model introduces a higher number of parameters, leading to a decrease in FPS compared to GaussianImage, it maintains the ability to render animations in real time

#### Manual modification

MiraGe allows for manual manipulation of 2D images. By leveraging GaMeS parameterization, each Gaussian component is represented as a triangle. Vertices can then be independently adjusted and moved within 3D space, enabling flexible image modification (Fig. 2, 7).

We demonstrate examples of modifications using datasets such as DIV2K, Kodak, and Animals(^3^33[https://www.kaggle.com/datasets/alessiocorrado99/animals10](https://www.kaggle.com/datasets/alessiocorrado99/animals10)). Additionally, we generated our own 2D images using DALL-E 3 to illustrate the benefits of our method. We can obtain modifications of small details like changing fingers position (Fig. 1), human facial expressions (Fig. 13) or dog poses (Fig. 14). As MiraGe can trained in a 3D context, we can implement modifications in the third dimension to create the illusion of 3D transformation (Fig. 3, 6).

It is crucial to note that when we displace elements from the foreground, the background remains unseen because the model only reconstructs the objects. This is demonstrated in Fig. 14, where artifacts are apparent on the hind paw of the depicted animal.

**Table 1: Quantitative comparison with various baselines in PSNR and MS-SSIM. MiraGe gives state of the art results. Model-$x$ denotes that the model was initialized with $x$ Gaussians.**
|  | Kodak dataset | DIV2K dataset |  |  |
| --- | --- | --- | --- | --- |
|  | PSNR $\uparrow$ | MS-SSIM $\uparrow$ | PSNR $\uparrow$ | MS-SSIM $\uparrow$ |
| WIRE (Saragadam et al., ) | 41.47 | 0.9939 | 35.64 | 0.9511 |
| SIREN (Sitzmann et al., ) | 40.83 | 0.9960 | 39.08 | 0.9958 |
| I-NGP (Müller et al., ) | 43.88 | 0.9976 | 37.06 | 0.9950 |
| NeuRBF (Chen et al., ) | 43.78 | 0.9964 | 38.60 | 0.9913 |
| 3DGS (Kerbl et al., ) | 43.69 | 0.9991 | 39.36 | 0.9979 |
| GaussianImage-70k (Zhang et al., ) | 44.08 | 0.9985 | 39.53 | 0.9975 |
| GaussianImage-100k* (Zhang et al., ) | 38.93 | 0.9948 | 41.48 | 0.9981 |
| MiraGe-70k (our) | 57.41 | 0.9998 | 53.22 | 0.9996 |
| MiraGe-100k (our) | 59.52 | 0.9999 | 54.54 | 0.9998 |

#### Physics application in MiraGe

Using 2D-MiraGe we can express Gaussian components with a 2D point cloud. Therefore, we can use MPM (Hu et al., ) based physics engine implemented, for example, in Taichi$_$elements. This high-performance physics engine supports multiple materials, including elastic objects and sand. We use inspiration from GASP (Borycki et al., ) and train simulation on 2D points then use physical deformation on triangle soup. In Fig. 8, we present simulation results obtained using Taichi_elements. As we can see, we can add physical properties to 2D objects. On the other hand, using Amorphous-MiraGe or Graphite-MiraGe we can use Blender and modify directly parameterized flat 3D Gaussian (Fig. 9).

## 5 Conclusion

In this paper, we introduce MiraGe that uses flat 3D Gaussian components to model 2D images. MiraGe gives state-of-the-art reconstruction quality and simultaneously allows image manipulation. Furthermore, we can modify photos on a plane (Fig. 8) and in 3D space (Fig. 10, 9). In consequence, we obtain the illusion of 3D-based modifications. Furthermore, we can combine our solution with a physics engine to obtain realistic motion in the image. Conducted experiments show that MiraGe is applicable in many different scenarios and produces high-quality simulations.

#### Limitation

It is crucial to note that the model is not generative, so improper adjustment of Gaussian positions can cause gaps in the image (e.g. a missing dog’s paw 14). While the model can produce realistic changes, significant modification may introduce visual artifact. Moreover, our model requires encoding more parameters than GaussianImage to achieve high-quality image reconstruction for animation. Addressing this trade-off will be a focus of our future work.

#### Social impact

The model can be applied to generate novel image transformations used to dataset augmentation, facial recognition (Sanil et al., ), distortion correction, and visualizing architectural designs (Gerstweiler et al., ), as shown in Fig.
10. In the field of medical imaging, it is utilized for refining anatomical models and enhancing the accuracy of surgical simulations and diagnostic tools (Lin et al., ). Additionally, in 2D computer games, the model facilitates more realistic animations by incorporating physics-based effects (Mohd et al., ).

## 6 appendix

Here, we provide a comprehensive overview of the implementation details. Furthermore, we present supplementary experimental results, such as extended performance evaluations and ablation studies focusing on camera settings.

### 6.1 Implementation details

The source code for this project will be made publicly available on GitHub. Our code was developed based on the GaMeS framework, and it is distributed under the GS Vanilla license. Computational experiments in main paper were conducted using NVIDIA GeForce RTX 4070 Laptop version and NVIDIA GeForce RTX 2080. Appendix time comparisons were reported using NVIDIA GeForce RTX 2080.

Figure: Figure 15: MiraGe allows for the modification of larger scenes. We can selectively alter specific areas and introduce smooth movements or material adjustments. In this example, the bottom of the blanket is shown in motion. This, along with other modifications, is available in the supplementary files as videos.
Refer to caption: x10.png

Figure: Figure 16: MiraGe allows for manual image edits and for using a physics engine for real-life-like image modifications.The left image illustrates a Gaussian representation achieved through a triangle mesh triangle soup, while the accompanying point-based depiction provides finer details, offering a more refined visual comparison.
Refer to caption: extracted/5895312/imgs/kura_drawio_2.png

Building upon the GaMeS framework, we initialized the Gaussian distributions to lie perpendicular to the $XZ$ plane. In our model, where all Gaussians are constrained to a 2D plane at rendering time, we consider only the rotation angle, denoted as $\phi$, as the primary rotation parameter. To facilitate the rendering of Gaussians positioned on the $XZ$ plane, $\phi$ serves as the primary learning parameter. The corresponding quaternions of rotation are computed as follows: for rotation about the x-axis $q_{x}=[cos(\frac{\phi}{2}),sin(\frac{\phi}{2}),0,0]$, and for the z-axis $q_{z}=[cos(\frac{\pi}{2}),0,0,sin(\frac{\pi}{2})]$. Since no rotation occurs about the y-axis, the quaternion remains $q_{y}=[1,0,0,0]$. These quaternions are then combined through multiplication to form a new rotation matrix, ensuring precise alignment of the Gaussians on the $XZ$ plane.

### 6.2 Supplementary Numerical Findings from the Primary Paper

We conducted an extensive analysis of the MiraGe model due to its unique ability to control the behavior of Gaussians. Three distinct settings for Gaussian movement were explored:

- •
Amorphous the first allows Gaussians to move freely in 3D space,
- •
2D: the second restricts their movement to align parallel to the $XZ$ plane
- •
Graphite the third confines all Gaussians to the $XZ$ plane, effectively creating a 3D representation.

A qualitative analysis was performed, considering the impact of the mirror camera (Tab. 2), as well as the effect of varying the number of initial Gaussians on the overall model behavior (Tab. 3).

A clear tendency is to be noted in Tab. 2: adding the mirror camera view improves significantly the representation’s fidelity of every proposed Gaussian method. This behavior can be detected with the help of any of the measured metrics, i.e., PSNR, MS-SSIM and LPSIS. The drawback of improving the image quality is a longer training time required. The ablation study presented in Tab. 3 similarly suggests that our model scales well with the number of Gaussians used during model initialization. The striking example here is an average 62.12 PSNR score achieved by Amorphous method on Kodak dataset. The price paid in time of training grows here slower, i.e., increasing the number of starting Gaussians by an order of magnitude results in more extended though comparable training period length.

### 6.3 Extension of examples modification and Artifacts

Figure: Figure 17: Example of artifacts generated during animation, typically due to imperfect rendering. The model was trained on a white background, leaving residual white Gaussians along the border of the camel’s muzzle, leading to artifacts. In this instance, the Graphine-MiraGe performed best in handling the head-turning movement
Refer to caption: extracted/5895312/imgs/wielblad_drawio_1.png

MiraGe enables manual image editing and incorporates a physics engine for image modifications (Fig. 16). Animating a full scene can be non-trivial. It is crucial to remember that if certain Gaussians are shifted without considering their dependencies on others, the image will be disrupted. Therefore, the relationships between the Gaussians must be carefully modeled. We demonstrate this concept with the example of children playing with a blanket (Fig. 15). Despite the movement of the blanket (as seen in the supplementary video), the image remains uninterrupted and coherent."

Integrating the representation into Blender can introduce automatic adjustments that may result in visual artifacts (Fig. 17), particularly when training on images with a white background. These modifications can lead to unrealistic renderings that are challenging to detect through automated means and currently require subjective evaluation by a human observer.

**Table 2: Ablation study of the effect of adding the mirror camera as augmentation technique on training time and the output image quality measured in widely recognized metrics: PSNR, MS-SSIM, LPSIS. Experiment was performed on Kodak dataset with 100k Gaussians on start.**
| Kodak dataset |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Gaussian control method | Camera Setting | PSNR $\uparrow$ | MS-SSIM $\uparrow$ | LPSIS $\downarrow$ | Training Time(s) $\downarrow$ |
| Amorphous | One camera | 51.56 | 0.9996 | 0.0050 | 448.73 |
| Mirror cameras | 59.52 | 0.9999 | 0.0005 | 639.66 |  |
| Graphite | One camera | 42.49 | 0.9948 | 0.2984 | 398.54 |
| Mirror cameras | 46.90 | 0.9983 | 0.1238 | 739.66 |  |
| 2D | One camera | 42.75 | 0.9950 | 0.2931 | 552.80 |
| Mirror cameras | 48.82 | 0.9987 | 0.0071 | 942.78 |  |

**Table 3: Measuring the influence of the initial number of Gaussians on the quality of the Image reconstruction. Experiment was performed using mirror camera view for every table entry.**
| Kodak dataset |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Gaussian control method | Initial Gaussians | PSNR $\uparrow$ | MS-SSIM $\uparrow$ | LPSIS $\downarrow$ | Training Time(s) $\downarrow$ |
| Amorphous | 10k | 50.66 | 0.9987 | 0.3531 | 584.84 |
| 50k | 55.54 | 0.9997 | 0.0033 | 634.65 |  |
|  | 100k | 59.52 | 0.9999 | 0.0005 | 639.66 |
|  | 150k | 62.12 | 0.9999 | 0.0002 | 676.10 |
| Graphite | 10k | 40.39 | 0.9940 | 0.0599 | 651.32 |
| 50k | 44.90 | 0.9973 | 0.2024 | 732.91 |  |
|  | 100k | 46.90 | 0.9983 | 0.1238 | 739.66 |
|  | 150k | 48.16 | 0.9987 | 0.0105 | 801.18 |
| 2D | 10k | 39.75 | 0.9886 | 0.0769 | 857.30 |
| 50k | 45.03 | 0.9955 | 0.2789 | 876.56 |  |
|  | 100k | 48.82 | 0.9987 | 0.0071 | 942.78 |
|  | 150k | 50.54 | 0.9992 | 0.0031 | 955.86 |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: The selection process when editing is not shown (e.g. what tools/algorithms are used to select patches of vertices to move around?)

Fig. 12 needs more context and intuition as to why the FPS drops after training for longer for the proposed method, but not for Gaussian Image/GS?
soundness: 3
strengths: *I think this is a novel and creative combination of existing methods, and the additional “mirror” loss acts as an effective regularizer

*Evaluation of reconstruction quality is sound and strongly favors the proposed approach
summary: This paper combines GaMeS framework (which maintains a mesh representation of a 3DGS such that mesh edits propagate to the GS), with the idea of GaussianImage (which uses GS to represent 2D images). They analyze and fix several issues that arise, including moving from 2D to 3D gaussians to allow for 3D editing. An additional point of novelty is that both the image and its mirror are optimized when training the GS, improving quality dramatically. The authors present qualitative results showing complex editing of 2D images enabled by their method.
weaknesses: *I did not see training/inference time and memory in the quantitative evaluation table

*The editing capability does not have a comparative evaluation. Because this is a claimed contribution, I think the authors should find baselines and demonstrate why the proposed approach is superior for interactive image editing. Some of the following references could be considered in the evaluation:

[1] Pan, Xingang, et al. "Drag your gan: Interactive point-based manipulation on the generative image manifold." ACM SIGGRAPH 2023 Conference Proceedings. 2023.
[2] Shi, Yujun, et al. "Dragdiffusion: Harnessing diffusion models for interactive point-based image editing." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
[3] Jacobson, Alec, et al. "Bounded biharmonic weights for real-time deformation." ACM Trans. Graph. 30.4 (2011): 78.
[4] Wang, Yu, et al. "Linear subspace design for real-time shape deformation." ACM Transactions on Graphics (TOG) 34.4 (2015): 1-11. 


*Visualizations of the mesh are not shown when editing, and how the mesh edit propagates to the Gaussian splat. As an ablation, it would help to show why we can’t live with the edited (textured) mesh and need to map the edits to the Gaussian Splat.

*Evaluation of reconstruction quality is missing comparative qualitative examples. Because the PSNRs/SSIMs are generally quite high, it may be tough to appreciate these improvements qualitatively.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: * Can the authors describe in more details on how the exact editing was performed for the different results? Eg: facial expression change, scaling the painting/mouth
soundness: 4
strengths: * The paper is well written and neatly presented. Both the problem and the solution are well-motivated. 
* It explores an interesting direction to edit images in 3D using flat Gaussians, compared to exisitng generative based methods.
* The paper explores different approaches for controlling Gaussians- Amorphous, 2D, and Graphite. The formulation for Graphite inspired from mirror reflections is quite interesting. 
* Evaluation setup seems sound and the authors include an ablation as well. The mirror cameras show a good improvement in scores. 
* The method performs very well on reconstruction and the editing results look promising while keeping the original structure in place as it's not generative
* The implementation is efficient by leveraging physcis engines such as Taichi
summary: MIRAGE proposes to represent images using flat Gaussians by using the notion of mirror reflections to create 3D awareness. The method achieves good reconstruction and editing of images in 3D space. The authors propse 3 slightly different techniques for controlling Gaussians- Amorphous, 2D, and Graphite. The authors evaluate the approaches on the Kodak and Div2 dataset.
weaknesses: * Manipulating Gaussians directly to perform edits seems quite tedious and even challenging for users (compared to instruction based editing in InstructPix2Pix for example)
* While the method is promising, the edits performed have small delta with the original as it's mostly with scaling and shifting Gaussians. This limitation is a natural con of lacking a generative prior.
* The time taken to train the Gaussians on a single image is rather long 450~950 seconds.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: As mentioned in the weaknesses section, I would appreciate it if the authors could provide more persuasive motivation and clearly outline the contributions of the proposed method.
soundness: 2
strengths: The authors provide numerous figures and videos for visualization, and the reconstruction performance is significant.
summary: The paper aims to represent 2D images with high reconstruction quality in an editable fashion. To this end, it positions 2D images in 3D space and manipulates them accordingly. More specifically, the paper integrates GaussianImage and GaMes parameterization, along with the proposed mirror loss.
weaknesses: First of all, although not significant, the paper has been submitted in the camera-ready format rather than the under-review format. Therefore, I apologize for not being able to pinpoint the exact locations of each word and sentence.

It would be highly beneficial for the paper to provide more concrete reasons for why this low-level “manual editing” is an important issue, as manually editing Gaussians is quite trivial. Furthermore, there is an inconsistency regarding this matter. In the introduction, it states that GaussianImage lacks the capability for manual editing, but in the related works section, it contradicts this by claiming that Gaussians enable precise and flexible editing. More detailed explanations of the limitations of GaussianImage in manual editing, as well as a clearer highlighting of the strengths of this paper, would better clarify its intentions.

Additionally, more thorough and extensive experiments could be conducted, as Table 1 alone is insufficient to demonstrate how much each component affects performance. Specifically, an ablation study on GaMes parameterization and mirror loss concerning representation performance would clearly demonstrate the role of each component. A quantitative comparison between Amorphous, 2D, and Graphite on various metrics would help readers understand the benefits of each method more clearly. Moreover, quantitative and qualitative results on other manual editing methods would further enhance the paper.

(Minor) Additionally, there are some minor misused or misleading words and phrases that could be easily corrected without disrupting the main storyline. For example, “natural networks” in the Related Works section seems to refer to neural networks. The phrase “Training the Gaussian component is challenging” in Section 3 could be more clearly articulated. A clearer version might state like "the optimal number of Gaussians required to represent a given input (image) is not known a priori, and it is non-trivial to adjust the number of Gaussians.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. Questions: I don't queit get the necessity of intriducing the mirror camera. Could you provide more detailed explanation on this?

2. Suggestions: It is recommended that the authors show more convincing editing visual results that could prove that your method has a good understanding of underlying 3D structure. By the way, I think the paper is hard to follow for me, which take me a while. I suggest the authors to improve the readability of the paper.
soundness: 2
strengths: The proposed method achieving state-of-the-art reconstruction quality of 2D image using Gaussian Splatting, also enables the manipulation of 2D images within 3D space, creating the 3D effects.
summary: This paper introduces a novel method, MiraGe, which uses mirror reflections to perceive 2D images in 3D space and employs flat-controlled Gaussians for precise 2D image editing.
weaknesses: 1. Lack of understanding of 3D structure: This paper elevates 2D image editing to 3D space, but it overlooks the importance of capturing the underlying 3D structure in the 2D image. As a result, the proposed method is limited in its ability to handle editing tasks that involve invisible parts or complex transformations. For example, while the method can achieve editing by dragging existing parts, it fails to handle editing tasks that require manipulating hidden parts, such as the hidden leg of a cat (as shown in Figure 2). Additionally, most of the manipulation effects are limited to affine transformations or local rigid editing, which can also be accomplished using standard 2D editing tools like Photoshop. Without a robust representation of the 3D structure, it is difficult to guarantee that the complex editing capabilities offered by the proposed method in 3D space will always translate to reasonable results when reflected in the 2D image.

2. The authors should provide a more detailed explanation of why the mirror camera is necessary for the proposed method and how it contributes to the overall performance of the method. Additionally, the comparison between the second row (Amorphous + mirror camera) and the fourth row (Graphic + mirror camera) in Figure 6 suggests that they achieve almost the same effects, which does not clearly illustrate the necessity of Graphite.

3. I think the paper is hard to follow for me, which take me a while. I suggest the authors to improve the readability of the paper further. The paper layout and formatting is a little messy, and the writing style is not very clear. The authors should consider revising the paper to make it more accessible to readers. And the experimental results are not very convincing and has insufficient experimental length. Concretely, this paper lack the visual comparison with baseline methods and 2D editing methods.

Meta-review: The submission received mixed reviews. The reviewers appreciate the innovation of using flat Gaussians for 3D-like editing while demonstrating strong reconstruction results. The major concerns from g8tX and 9FhM were on the accuracy of claims on 3D editability and physical simulation-readiness, as well as the practical necessity of Gaussian representations for image editing. The AC carefully read through the paper, the reviewers' comments, the authors' rebuttal and the discussions. While the paper presented an interesting approach with well-executed experiments, the AC agrees with g8tX and 9FhM that more extensive discussions on the practicality and enabled capabilities should be provided to better justify the contributions. The AC also believes that the work has deviated away from the domain of learning representations / neural networks / AI, and the manuscript may be more suitable in computer graphics or vision conferences (e.g. SIGGRAPH or CVPR). As such, the AC recommends rejection at this time.

Decision: Reject