Towards Realistic Example-based Modeling via 3D Gaussian Stitching

## Abstract

Abstract. Using parts of existing models to rebuild new models, commonly termed as example-based modeling, is a classical methodology in the realm of computer graphics. Previous works mostly focus on shape composition, making them very hard to use for realistic composition of 3D objects captured from real-world scenes. This leads to combining multiple NeRFs into a single 3D scene to achieve seamless appearance blending.
However, the current SeamlessNeRF method struggles to achieve interactive editing and harmonious stitching for real-world scenes due to its gradient-based strategy and grid-based representation.
To this end, we present an example-based modeling method that combines multiple Gaussian fields in a point-based representation using sample-guided synthesis.
Specifically, as for composition, we create a GUI to segment and transform multiple fields in real time, easily obtaining a semantically meaningful composition of models represented by 3D Gaussian Splatting (3DGS). For texture blending, due to the discrete and irregular nature of 3DGS, straightforwardly applying gradient propagation as SeamlssNeRF is not supported. Thus, a novel sampling-based cloning method is proposed to harmonize the blending while preserving the original rich texture and content. Our workflow consists of three steps: 1) real-time segmentation and transformation of a Gaussian model using a well-tailored GUI, 2) KNN analysis to identify boundary points in the intersecting area between the source and target models, and 3) two-phase optimization of the target model using sampling-based cloning and gradient constraints. Extensive experimental results validate that our approach significantly outperforms previous works in terms of realistic synthesis, demonstrating its practicality. More demos are available at homepage .

## 1. Introduction

As we all know, 3D scenes typically contain multiple 3D objects composed of various parts. Example-based modeling  (Funkhouser et al., ) is a technique that involves combining different parts from different objects to create new ones. This is a common tool in Computer Graphics (CG) modeling, where objects are designed in a non-realistic CG fashion.
In this paper, we consider realistic example-based modeling, where all parts are captured from the real world, as shown in Fig.  1. This task becomes prominent with the emergence of Neural Radiance Fields, which enables photorealistic 3D reconstruction and rendering.

Among the various approaches designed for 3D modeling from multiple neural fields, a portion of the research (Gao et al., ; Liu et al., ) is devoted to the inverse rendering process to achieve consistent lighting and shadowing. But these methods rarely consider a situation where the harmonious and seamless effect is required for merging or unifying two or more neural fields.
SeamlessNeRF (Gong et al., ) is the first work to tackle seamless merging, attempting to address the consistency problem by propagating gradients on synthesis cases. Nonetheless, due to its implicit grid-based representation, SeamlessNeRF can neither achieve fine-grained editing (e.g. the face in the Santa case in Fig. 2) under real-world cases nor provide an interactive workflow in real-time. Additionally, its gradient-based strategy can produce significant artifacts (see Fig. 9) and fails to propagate structural characteristics when the condition becomes more complex (e.g., the bottle in the left-upper corner in Fig. 1). Therefore, achieving a harmonious and photorealistic stitching result on real-world data remains an unsolved challenge that needs further exploration.

To address the limitations mentioned above, we propose a new method for interactive editing and stitching multiple parts using explicit shape representation in 3D Gaussian Splatting. Our method has two significant advantages. First, its point-based representation enables fine-grained editing, allowing for detailed appearance optimization and the removal of artifacts. Second, its rasterizer pipeline provides a real-time interactive editing environment.
Due to the discrete and irregular nature of 3D-GS, it is not feasible to conduct gradient propagation as SeamlessNeRF. Thus,
we introduce a novel sampling-based optimization strategy that can seamlessly propagate not only color tones but also structural characteristics.
Our evaluation benchmarks are primarily derived from real-world scenes, demonstrating our superior ability to handle complex cases.

More specifically, our pipeline takes multiple scenes as input, containing source and target objects represented by 3DGS. We then carefully segment these objects and apply rigid transformations in order to create a semantically meaningful composite in 3D space. An intersection boundary region between the objects is also identified before blending.
The next is the key step in our process which aims to optimize the appearance of the target objects so that their texture and color match those of the source object.
We achieve this by using a two-phase optimization scheme: the first phase involves sampling-based cloning (S-phase), and the second phase involves clustering-based tuning (T-phase).
During the S-phase, the target field is optimized using a heuristic sampling strategy that considers the structural characteristics at the boundary. Additionally, an efficient 2D gradient constraint is applied to preserve the original texture content of the target field.
However, optimizing solely with S-phase may lead to the appearance of artifacts or unintended color features that do not fit with the overall composite.
Therefore, we address this issue with T-phase, where we utilize a pre-calculated feature palette derived from the source field through aggregation and clustering. Subsequently, this palette is applied to tune the target field.
It is important to note that the two-phase optimization is a joint procedure, where losses from the S-phase are always maintained while losses from the T-phase are added later during optimization.

In summary, our method makes the following contributions:

- •
The first work to use 3D-GS for realistic and seamless part compositing, enabling real-world example-based modeling.
- •
A novel sampling-based optimization strategy is proposed, with which not only the texture color but also the structural characteristics can be propagated seamlessly.
- •
A user-friendly GUI is carefully designed to support an interactive workflow of the modeling process in real time.

Figure: Figure 2. Overview of our framework. Our novel pipeline provides an interactive editing experience and has real-time previewing capabilities to visualize the optimizing process, allowing for the seamless and interactive combination of multiple Gaussian fields.
Refer to caption: x1.png

## 2. Related Work

### 2.1. Example-based Seamless Editing

Seamless editing, particularly in the context of example-based image and texture synthesis, is a well-studied editing technique in computer graphics and image processing.
As for textures, example-based texture synthesis (Wei et al., ; Efros and Leung, ) intends to seamlessly create textures at any size from exemplars, which has been widely employed in contemporary graphics pipelines and game engines. In 2D image synthesis, patch-based synthesis techniques have been widely researched to seamlessly combine visually inconsistent images (Pérez et al., ; Darabi et al., ).
Meanwhile, Kwatra et al. () introduced “Texture Optimization,” which transfers photographic textures to a target image for example-based synthesis. To facilitate structural image editing tasks, “PatchMatch” (Barnes et al., ) found approximate nearest-neighbor correspondences between patches in images for seamless image region reshuffling.
In terms of seamless editing in 3D objects, Rocchini et al. () and Dessein et al. () propose methods for stitching and blending textures on 3D objects, respectively, while Yu et al. () use the Poisson equation to implicitly modify the original mesh geometry via gradient field manipulation.
Additionally, example-based modeling can also generate novel models from parts of existing models (Funkhouser et al., ), allowing untrained users to create interesting and detailed 3D designs, such as city building (Merrell, ), things arrangements (Fisher et al., ), mesh segmentation (Katz et al., ), and merging (Kreavoy et al., ). Recently, deep learning methods have leveraged generative models to generate diverse instances from a single exemplar (Wu and Zheng, ; Li et al., ) or a cluster of examples (Zhang et al., ). Definitely, the example-based methodology is a valuable tool for creating diverse and novel content, which can reduce the workload for the artists or can be leveraged by procedural content generation programs. In our work, we combine this valuable idea with the advanced technique of 3DGS to create content directly from the real world.

### 2.2. Neural Scene Composition

Neural scene composition primarily involves the synthesis of multiple neural objects represented by neural fields, such as free-viewport video (Lin et al., ; Zhang et al., ; Wang et al., ), autonomous driving (Ost et al., ; Kundu et al., ; Tancik et al., ; Fu et al., ; Zhou et al., ; Yang et al., ) and scene understanding (Kerr et al., ; Shuai et al., ; Yang et al., ; Wu et al., ).
And for those composition tasks with multiple pre-trained models, mesh scaffold (Yang et al., ; Yariv et al., ) or texture extraction (Tang et al., ; Chen et al., ) from the neural field are preferred to achieve higher render speed or rather fine-grained control. This type of work acts as a “bridge” between neural and traditional representations in order to improve performance using the classical graphics pipeline.
A small portion of the work focuses on creating a mixed render pipeline for neural 3D scene composition tasks, combining traditional render techniques like ray tracing (Qiao et al., ), shadow mapping (Gao et al., ), and ambient occlusion (Gao et al., ). There are also a few works that focus on creating a compositional scene with generative models like diffusion models (Po and Wetzstein, ).

None of those works except Neural Imposter (Liu et al., ) and SeamlessNeRF (Gong et al., ) focus on example-based modeling by stitching multiple part NeRFs. However, part objects in Neural Imposter are just placed together without any appearance blending, which cannot support a general case of 3D modeling. SeamlessNeRF achieved harmonious results on a small-scale synthesis dataset, making it the first work to discuss seamless example-based modeling with neural techniques today. However, SeamlessNeRF cannot handle real-world cases when the condition becomes more complex, nor can it perform interactive editing, which is commonly required in example-based modeling. On the contrary, our approach overcomes these limitations, performs well in real-world scenarios, and supports interactive editing using Gaussian fields.

### 2.3. 3D Gaussians

3D Gaussian Splatting (Kerbl et al., ) is a point-based rendering method that has recently gained popularity (Yang et al., ; Huang et al., ; Liang et al., ; Tang et al., ; Chen et al., ; Yang et al., ; Li et al., ) due to its realistic rendering and significantly faster training time than NeRFs. Compared to the implicit representation of NeRF, 3DGS is more advantageous for editing tasks. The superior advance lies in the fact that, unlike previous work that embedded an object in a certain neural field (e.g., learnable grid or MLP network), once clusters of Gaussians are optimized, they can be easily fused together and fed into the rasterizer. The 3DGS pipeline was born with an intrinsic property suitable for composition.

## 3. Seamless Gaussians

Our approach starts with segmenting interesting parts from pre-trained Gaussian scenes. After acquiring target and source models represented by Gaussians, we carefully transform them to obtain a semantically meaningful composite. Then we optimize the target objects to achieve a harmonious composite through a two-phase (sampling-based cloning and clustering-based tuning) scheme. All these processes can be run interactively and previewed in real-time with our well-tailored GUI design.

### 3.1. Segmenting and Transforming Gaussians

Segmentation is the first step in example-based modeling, which involves picking out interesting parts as the components of the final artwork. Previous works have performed this task by providing guidance using 2D mask (Cen et al., ; Mirzaei et al., ) or injecting semantic label (Kerr et al., ) into a neural field. Now, benefiting from Gaussian representation (resembling point cloud), segmentation can become more practical at a finer-grained level. In our pipeline, we show that a combination of a simple bounding box and a user brush can work very well for a clean mask (see Fig. 14). For instance, we can mask the sculpture with a brush to match the shape of Santa’s face (see Fig. 2).

Transformation aims at placing multiple interesting parts $\mathcal{G}_{i}$ represented by Gaussians to form a semantically meaningful composite $\mathcal{M}$, which can be denoted as:

$$ (1) $\mathcal{G}_{i}^{\mathrm{global}}=\normalsize{F}(\mathcal{G}_{i}^{\mathrm{ local}}|\hat{\mathbf{q}}_{i},\mathbf{t}_{i},s_{i}),\quad\mathcal{G}_{i}\in \mathcal{M}$ $$

where $F$ is the rigid transformation applied on one part of Gaussians with rotation $\hat{\mathbf{q}}_{i}$ (represented in quaternion), translation $\mathbf{t}_{i}$, and scale $\mathbf{s}_{i}$, transforming the part from its local space to the global space. Specifically, the partial attributes of each $\mathcal{G}$ should be modified, which includes position $\mathbf{x}$, scaling $\mathbf{s}$, rotation $\mathbf{q}$ (in quaternion), and feature $\mathbf{f}$ (represented as spherical harmonics). The position and scaling can be performed trivially, while the transformed rotation $\mathbf{q}^{\prime}$ and feature $\mathbf{f}^{\prime}$ can be expressed as:

$$ (2) $\displaystyle\mathbf{q}^{\prime}$ $\displaystyle=\mathbf{q}\hat{\mathbf{q}},$ $\displaystyle\mathbf{f}^{\prime}$ $\displaystyle=M_{bands}(\mathbf{f}\>|\>\hat{\mathbf{q}}),$ $$

where $M_{bands}$ means we use a set of matrices to rotate each band of SH coefficients introduced by (Ivanic and Ruedenberg, ).

Figure: Figure 3. For a Gaussian point in the target field, its (a) K-nearest neighbors in the source field can be leveraged to justify whether this point belongs to the intersection boundary region. We use the boundary of (b) as an example to demonstrate the effectiveness of this strategy, as shown in (c).
Refer to caption: x2.png

### 3.2. Boundary Condition by KNN Analyzing

After transformation, certain points in one field approach another field (see Fig. 3), forming intersection boundary regions between all Gaussians. For the sake of simplicity, we will use two Gaussians, source field and target field, to demonstrate our approach.

Before optimization, the boundary points in the target field must be identified, as this is the critical and initial condition for harmonization.
For each Gaussian point in target field $\mathcal{T}$, we search its K-nearest neighbors in source field $\mathcal{S}$, which can be denoted by:

$$ (3) $\left\{b_{i}\right\}_{K}=\mathop{KNN}\limits_{\mathcal{S}}(a),\quad a\in \mathcal{T},b_{i}\in\mathcal{S}$ $$

where $a$ is a point in the target field, and $b_{i}$ is a point in the source field. Whether a point $a$ belongs to boundary $\partial B$ can be identified as $a\in\partial B$ iff.:

$$ (4) $\frac{1}{K}\sum\limits_{i}^{K}|b_{i}-a|<\beta\quad\mathrm{and}\quad o(a)>\tau,$ $$

where $o(a)$ is the opacity of that Gaussian point, $|b_{i}-a|$ is the Euclidean distance between $b_{i}$ and $a$. $\tau$ and $\beta$ are thresholds and we empirically set $\tau$ to 0.95, $\beta$ to $0.05\times L$. $L$ is the size of the composite. (e.g. measured by the bounding box).
An additional method for a better boundary condition on real-world data is that we discard outliers in both fields (e.g. some Gaussian points are far from the others, which may occur in some scenes).

We calculate referenced features for these boundary points in order to confirm the boundary condition. For each $a\in\partial B$, its target feature is:

$$ (5) $\hat{\mathbf{f}}(a)=\frac{1}{K}\sum\limits_{i}^{K}\mathbf{f}^{\prime}(b_{i}), \quad a\in\partial B,b_{i}\in\mathop{KNN}\limits_{\mathcal{S}}(a)$ $$

where $\mathbf{f}^{\prime}(b_{i})$ is the feature of $b_{i}$ after transformation.
To achieve this boundary condition, we optimize boundary points toward their target features:

$$ (6) $\mathcal{L}_{feature}=\sum\limits_{a\in\partial B}\left\lVert\mathbf{f}^{ \prime}(a)-\hat{\mathbf{f}}(a)\right\rVert_{2}^{2},$ $$

where $\mathbf{f}^{\prime}(a)$ is the feature of $a$ and we directly apply this loss on SH coefficients.

Figure: Figure 4. ablation study on the color loss in the S-phase. Without color loss, the propagation is inefficient and will not begin. The cases shown above have been running for more than twice as long, but they are still trapped in insufficient propagation. It is because, without color loss, only a small number of points’ features need to be updated at first, as opposed to shared weights in an MLP applied to all points. That minor “forces” cannot drive the overall minimization of the gradient loss.
Refer to caption: extracted/5818686/figures/wont_start2.png

Figure: Figure 5. ablation study on the effectiveness of gradient loss for different weights. Experiments show that higher weights can help to preserve more content while preventing harmonization.
Refer to caption: x3.png

Figure: Figure 6. ablation study on sampling-based cloning (S.) and clustering-based tuning (T.). Here, “Both” means the full scheme.
Refer to caption: x4.png

Figure: Figure 7. Ablation study on the impact of mapping function $\phi$ in the S-phase. Random effects make composition more realistic.
Refer to caption: x5.png

### 3.3. Sampling-based Cloning

We propose sampling-based cloning as our “S-phase” in optimization. The core idea is how to seamlessly propagate the style in boundary through the remaining points in the target field while preserving its rich content. In contrast to a regular grid suitable with a gradient-based strategy in SeamlessNeRF (Gong et al., ), Gaussian points are irregularly and discretely distributed in 3D space. As a result, alternative approaches need to be explored. A straightforward idea is that given a point in target field $\mathcal{T}$, one can calculate the feature difference between that point and its neighbors in $\mathcal{T}$, resembling “Laplacian coordinates”. Then, one can use that “difference” as the regularizer while minimizing $\mathcal{L}_{feature}$.
However, this naive approach may fail even before propagation begins (see Fig. 4).
Furthermore, the boundary’s structural characteristics (such as the bottle-bell intersection in the right-upper corner of Fig. 9) necessitate seamless cloning, which significantly improves the stitching quality.

Hence, we propose an effective sampling strategy to explicitly propagate features for each remaining point outside the boundary. The core idea lies in the way of searching several “driven points” for a candidate. The color of the candidate is driven by those points. For each $a\in\mathcal{T}-\partial B$, the optimizing target of its color in direction $\mathbf{d}_{a}$ is:

$$ (7) $\hat{\mathbf{f}}(a,\mathbf{d}_{a})=\frac{1}{K}\sum\limits_{i}^{K}\mathbf{f}^{ \prime}(b_{i},\mathbf{d}_{b}),\quad a\in\mathcal{T}-\partial B,\>b_{i}\in \mathop{KNN}\limits_{\partial B}(\phi(a))$ $$

where $\mathbf{f}(a,\mathbf{d}_{a})$ means sampling SH color in view direction $\mathbf{d}_{a}$ (from point $a$ to camera), the same as $\mathbf{f}(b,\mathbf{d}_{b})$.
The camera centers are uniformly sampled from the surface of a sphere centered on the composite object’s origin.
It is important to note that the sampling strategy $KNN(\phi(a))$ maps the locations of nearby candidate points $a$-s to the correlated neighboring “driven points” and inherits the continuity of the textures from those “driven points”.
We use $\phi(x)=x+sin(\gamma\cdot\delta x)$ to add random effect by disturbing KNN searching (see Fig. 7), where $x$ is the position of $a$, $\delta x$ is the distance between $a$ and its nearest $b_{i}$ in boundary, and $\gamma$ is empirically set to 10. A larger $\gamma$ is suitable for higher structural frequencies.
In this way, we can synthesize structurally aware stitching results. With Eq. (7), we add a color loss to the S-phase:

$$ (8) $\mathcal{L}_{color}=\sum\limits_{a\in\mathcal{T}-\partial B}\left\lVert\mathbf {f}^{\prime}(a,\mathbf{d}_{a})-\hat{\mathbf{f}}(a,\mathbf{d}_{a})\right\rVert_ {2}^{2},$ $$

so that the color of those candidates can be optimized towards their target to achieve our explicit feature propagation.

To preserve the original rich content in $\mathcal{T}$, we present a more efficient gradient loss calculated in the local space of $\mathcal{T}$, leveraging the guidance in 2D screen space:

$$ (9) $\displaystyle\mathcal{L}_{grad}$ $\displaystyle=\sum\limits_{x\in I}\left\lVert\nabla_{x}I^{\mathcal{T}}(p)-\hat {\nabla}_{x}I^{\mathcal{T}}(p)\right\rVert_{2}^{2},$ $\displaystyle I^{\mathcal{T}}(p)$ $\displaystyle=\mathcal{R}(\mathcal{G}_{\mathcal{T}}^{local},p),$ $$

where $p$ is the randomly sampled camera in the local space of target field $\mathcal{T}$, $I$ is the rendered color image by rasterizer $\mathcal{R}$ of 3DGS. We pre-calculate $\hat{\nabla}_{x}I$ for each camera with the Sobel operator (Sobel et al., ) before the optimization starts. We found that supervising gradients in screen space is more efficient than the straightforward one, as shown in Fig. 11.

Figure: Figure 8. Results for more real-world data from the BlendedMVS (Yao et al., ) and Mip360 (Barron et al., ) datasets, demonstrating that our method can produce realistic effects in real-world scenarios.
Refer to caption: extracted/5818686/figures/main_results.png

### 3.4. Clustering-based Tuning

While S-phase optimization is effective in preserving local color consistency, relying solely on it may lead to misaligned global appearance, such as uneven brightness, hues, and saturation (See Fig. 6). Therefore, we propose using a clustering extracted color palette to perform global tuning, which we refer to as the ”T-phase” in optimization. This approach enhances the overall harmony of the composite by performing dynamic matches to a palette. To implement the T-phase, we first aggregate and cluster the color of the source field from various angles:

$$ (10) $\{\mathbf{c}_{i}\}_{N},\{w_{i}\}_{N}\leftarrow\mathcal{A}(\mathcal{G}_{ \mathcal{S}}^{global}),$ $$

where $\mathbf{c}_{i}$ is the color (cluster center) in palette, $w_{i}$ is the sample percentage occupied by the center, and $\mathcal{A}$ stands for our aggregation algorithm.
Our approach, inspired by Li et al.’s work (Li et al., ), uses a streaming method to accelerate color aggregation. We start with three bins, collect color samples from a random view, and calculate the new color center for each bin by averaging the original center and new samples collected in it. The number of bins expands to accommodate far-off samples. Centers expire after 20 iterations with no sufficient votes. We repeat this process until all color centers are stable.

Once the aggregation process finishes, those color centers will form a palette (see Fig. 2). We employ the following loss in our T-phase as a pixel-wise summation:

$$ (11) $\displaystyle\mathcal{L}_{tune}$ $\displaystyle=\sum\limits_{\mathbf{c}\in I^{\prime}}w_{\chi_{c}}\left\lVert \mathbf{c}-\mathbf{c}_{\chi_{c}}\right\rVert_{2}^{2},~{}I^{\prime}\leftarrow\{ I^{\mathcal{T}}_{x}(p)|\alpha(x)>0.95\},$ $\displaystyle\chi_{c}$ $\displaystyle=\mathop{\arg\min}\limits_{1\leq i\leq N}\left\{\left\|\mathbf{c} -\mathbf{c}_{i}\right\|_{2}-w_{i}\right\},$ $$

where $p$ is the randomly sampled camera in the global space, and $\alpha$ is the alpha mask corresponding to $I^{\mathcal{T}}$. Both $\alpha$ and $I^{\mathcal{T}}$ are rendered by rasterizer $\mathcal{R}$. $\chi$ represents the target bin’s index, and it is determined by both the distance from color centers and the probability density of bins. Our final total loss function can then be expressed as:

$$ (12) $\mathcal{L}_{total}=\mathcal{L}_{feature}+\mathcal{L}_{color}+\lambda_{1} \mathcal{L}_{grad}+\lambda_{2}\mathcal{L}_{tune},$ $$

where both $\lambda_{1}$ and $\lambda_{2}$ are empirically set to 2 in our experiments.

## 4. experiment

To test the effectiveness and generality of our approach, we conducted experiments on a variety of fascinating 3D objects.
We interactively built 21 composite results, comprising a total of 39 part models: 17 from BlendedMVS (Yao et al., ), 4 from Mip360 (Barron et al., ), 16 from SeamlessNeRF datasets, and 2 created by ourselves in a graphics engine.
For more results or the implementation details, please refer to our supplementary materials.

### 4.1. Qualitative Comparison

We compare our method to SeamlessNeRF (Gong et al., ), the first and most recent work that approaches our goal. Fig. 9 depicts three comparison cases. In the first case (clay & bread), SeamlessNeRF failed to achieve high-level geometry editability and struggled with artifacts caused by implicit representation. In the second case (bottle and bell), SeamlessNeRF failed to maintain a harmonious seamless effect due to applying the gradient-based strategy on the complex boundary. In the third case, SeamlessNeRF failed to propagate sufficient color tones due to the complex gradients in the boundary. In addition, we show that the 2D-guided style-transfer method (Nguyen-Phuoc et al., ) cannot produce a seamless stitching effect, as shown in Fig. 10. On the contrary, ours can handle all of these situations while producing harmonious results.

### 4.2. Quantitative Comparison

Currently, there is neither a specialized dataset providing ground truth nor an established metric to assess the realism of a 3D model’s appearance, making it challenging to evaluate the effectiveness of our approach quantitatively. Nevertheless, we force an evaluation utilizing VQA(Video Quality Assessment) methods, as outlined by Wu et al. (), and explored the use of 2D projection in video display for assessment purposes. Our results, presented in Tab. 1, demonstrate that our average score surpasses the baseline. For a comprehensive understanding of the quantitative experiments, please refer to our supplementary materials.

**Table 1. Quantitative comparison between ours and the baseline.**
|  | ours | SeamlessNeRF |
| --- | --- | --- |
| VQA average score $\uparrow$ | 0.784 | 0.753 |

Figure: Figure 9. Comparisons between our approach and the baseline methods (Gong et al., ). SeamlessNeRF failed in all of these real-world scenarios.
Refer to caption: x6.png

Figure: Figure 10. We show that these style-transfer results (Nguyen-Phuoc et al., ) fail to achieve our effect. Here, we re-implement SNeRF’s strategy (Nguyen-Phuoc et al., ) based on Gaussians to produce results above.
Refer to caption: extracted/5818686/figures/compare_snerf.png

### 4.3. Ablation Study

#### Effectiveness of 2D Gradient Loss.

Fig. 5 depicts the effect of gradient loss at various weights. Higher weights can help to preserve more content while obstructing harmonization. Fig. 11 demonstrates that 2D gradient loss with Sobel operator is significantly more effective than the simple one mentioned in Sec. 3.3.

#### Functionality of S-phase and T-phase.

We demonstrate the efficacy of our two-phase scheme in Fig. 6.
The S-phase aids in seamless boundary formation, while the T-phase aids in global harmonization when only the S-phase is present.

#### Effectiveness of Sampling Strategy for View-dependent Effects.

We ablate the sampling strategy in the S-phase (see Fig. 12) to show that view-dependent effects can be properly propagated using this strategy instead of random sampling.

Figure: Figure 11. ablation study on two kinds of gradient loss. The 2D gradient supervision (upper row) is more effective than the straightforward one since it focuses on the surface instead of the whole space.
Refer to caption: extracted/5818686/figures/ablation_knnsobelgrad.png

Figure: Figure 12. Ablation study on keeping view-dependent effects by sampling strategy in the S-phase. With that strategy used in Equ. (7), the upper-view color of paint on bell is properly propagated.
Refer to caption: x7.png

### 4.4. Editor and Application

To enable a practical and user-friendly workflow, we created an interactive GUI editor that can control and visualize any procedure in the entire process in real-time, including Gaussian segmentation and transformation, boundary identification, and optimization (see Fig. 16 and refer to the supplementary video for more details). Our framework can generate high-fidelity and seamless results across a wide range of real-world scenarios, providing distinct advantages in the direct creation of imaginative 3D models from reality.

## 5. Conclusions and Limitations

We have developed a highly efficient and effective interactive framework for creating realistic 3D models. The method involves stitching Gaussian components seamlessly to create a harmonious 3D model that is an accurate representation of the real world. Our approach has been tested on real-world datasets and has proved to be capable of handling complex cases with a user-friendly interface. This presents a promising avenue for example-based modeling directly from the real world.

#### Limitations and Future Work.

Currently, our work is unable to transform Gaussian models in a non-rigid manner, which may make it difficult to develop more imaginative cases. To enable a more flexible composition, we can use deformation methods such as ARAP (Igarashi et al., ) in the future. Furthermore, achieving a consistent lighting effect can help improve composition quality under intense lighting.

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 3
presentation: 2
questions: When does the method break? Can you characterize the (or a) class of shapes where the assumptions made work well and what happens if the data does not match these assumptions?
soundness: 3
strengths: The idea of using differential coordinates (as in Poisson image editing, or, put alternatively, editing in higher-frequency bands of the appearance signal only) in the context of editing "Gaussian Splatting" scenes is quite appealing; if it wasn't for other reservations, I would consider this a sufficient argument for accepting the paper (I should note, however, that I am not an active researcher in this area and thus might overlook prior work; but if this is new, I would think it is a really nice and useful idea for interactive editing of such data).

The results obtained are convincing (see below for qualifications) and the interactive demo of the system shown in the supplementary material shows that this could be practically applied.

Further, the general area of making use of scene representations obtained from matching "real-world" photos by making them editable is certainly important for making the whole research area useful to artistic applications in computer graphics.
summary: The paper proposes a method for interactively editing radiance fields encoded in point clouds ("Guassian Splatting"). It consists of two components: First, point clouds have to be segmented, which seems to be done mostly manually with seam-detection supported by nearest-neighbor detection across two point clouds to be stitched. It appears to me that this is not the main focus of the paper. The second component is an adaptation of Poisson image editing to radiance fields, where local differences in radiance information (encoded as spherical harmonics) is propagated along with boundary constraints. To improve realism, additional constraints are added, such as attracting the computed colors towards clusters of "colors" (actually, appearance / radiance information) in the target shape (to which a new piece is stitched), look-ups of nearby colors in the target point cloud for smoother transitions, and ad-hoc mid-frequency "noise" (function $\Phi$) for increased variability.

The resulting merged objects look quite convincing and appear more natural than competing results in the examples selected in the main paper. The appendix provides a more detailed evaluation, including quantitative results.

Limitations are not discussed (aside from the obvious limitation to merging rigid pieces, to which the paper limits itself a priori).
weaknesses: First of all, I have the impression that this paper is out of scope of ICLR. In terms of its approach and methodology as well as the problem it addresses, I would see this firmly within "computer graphics and interactive techniques"; it has only tangential impact on machine learning and representation learning. It might be close enough to warrant consideration, but I would see a graphics venue as a far better fit (for example, in computer graphics people would be much more interested in how this could lead to better software systems for creating better content, rather than evaluating the suitability and performance of the algorithms and data structures proposed for fitting and representing data in general).

That said, I would also seem room for improvement in positioning the paper. It seems to me that the stitching algorithm (differential appearance merging) is the main idea, while segmentation does not offer much novelty over the state of the art (see for example the long line of work of direct point cloud editing methods starting with Pauly et al.'s PointShop3D [Siggraph 2002, 2003]. More sophisticated segmentation algorithms, such as using graph-cut methods on point clouds are also known and for example already available in open source software (PCL/point cloud library). While these do not directly handle radiance data, it would be not a big step to integrate such ideas to reduce manual effort. Similarly, a long line of automatic outlier detection and removal methods could also improve this step easily (see for example "tensor voting" as a very basic approach). My point here is that the contribution of this paper is rather moderate, so it would make sense to position the paper with more emphasis on the second step (differential merging) which I personally found to be really neat.

Another issue I see (and again, probably more in writing/positioning than conceptually) is that the method mixes basic ideas (such as differential coordinates for appearance) with ad-hoc heuristics such as sinusoidal perturbations of color look-ups to create more appealing results. Differentiating between artistic and foundational ideas would make the paper stronger, in particular in the context of a rather technical venue aiming at fundamental machine learning research rather than graphics and interaction.

Finally, I am also a bit disappointed by the very short discussion of limitations in the main paper, which basically just states that the paper only considers rigid pieces; I would think that there are certainly trade-offs and mismatches in prior assumptions in the presented approach, independent of whether the geometry to be merged was also deformed prior to merging, that could be discussed more clearly. Understanding "when it breaks" is very important in order to put such ideas to use in new context.

Overall, I like the key idea, and if it is really novel, this is a strong point, but limitations in presentation and evaluation (in particular, discussion of limitations) make me skeptical; for the context of ICLR I would thus at this point be hesitant in terms of making a positive recommendation for this submission.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: see the weaknesses.
soundness: 3
strengths: The paper is well-written, logically clear, and readable.

The proposed method is reasonable and improves the Nerf-based method by seamlessly integrating 3DGS with the real-world model.
The paper provides experimental results to support its claims. Visualization results demonstrate the effectiveness of the method.
summary: The paper presents a novel method for realistic example-based modeling using 3D Gaussian Splatting (3DGS). It addresses the limitations of current methods like SeamlessNeRF by enabling interactive editing and seamless stitching of 3D models from real-world scenes. Key innovations include a real-time GUI for model segmentation and transformation, KNN analysis for boundary detection, and a two-phase optimization strategy for texture blending.
weaknesses: 1.  In Figure 13, the color propagation results are not clear. Could the authors provide examples with stronger color contrast or offer a detailed explanation of the color propagation, especially how the method handles various texture complexities?
2. The appendix briefly mentions time consumption comparisons, but could the authors elaborate on the computational demands of their method in the main text, perhaps comparing it to other existing methods in more detailed scenarios?
3. The paper mentions a GUI editor that facilitates the interactive modeling process. Could the authors provide more details or a demonstration of how the GUI editor is used, especially any features that help users manage complex scenes or models?
4. The success of the method heavily relies on the quality and complexity of the initial 3D models. Poor initial models might lead to suboptimal results, which needs further discussion in the paper.
5. The discussion of limitations is insufficient. It is recommended to add detailed pictures and discussion details of the failure of the paper's method.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Using color as a consistency supervision is sufficient to some extent, but it only achieves local statistical alignment. Have the authors considered incorporating semantic labels, such as SAM or DINO, for additional stylization supervision or evaluation?

2. Regarding the S-phase, it resembles a stylization process. Why didn't the authors follow a conventional stylization pipeline instead of using a statistical method, which may lack generalizability?

3. as for Fig.15, why implement SA3D instead of Segmeng any 3D Gaussians / Gaussian Grouping for segmentation since the latter are newer than former.
soundness: 3
strengths: 1. This paper proposes an effective method for composing pretrained 3D Gaussians, ensuring geometric accuracy and successfully addressing stylization between different objects.

2. A novel sampling-based optimization strategy is introduced to maintain the consistency of texture color between two 3D Gaussians, demonstrating the authors’ deep insight into object composition.

3. Authors develop a user-friendly GUI for composition and editing, which seems like a useful technical contribution for the proposed method.
summary: This paper proposes a method for stitching 3D Gaussians, introducing KNN, T-phase, and S-phase to effectively compose and stylize 3D Gaussians. Experimental results in certain cases demonstrate that this approach outperforms baseline methods, such as seamless NeRF.
weaknesses: 1. Please use the same cases as those in seamless NeRF (e.g., Figures 4, 5, 6, and 7 from seamless NeRF) to ensure a fair comparison.

2. Providing only VQA scores and images/videos is not sufficient for a convincing evaluation. I recommend that the authors introduce a more robust evaluation metric, rather than relying solely on videos/results.

3. A user-friendly GUI is primarily a technical contribution rather than a theoretical one, even though it is emphasized in Section 1. While there is text supporting the novelty of the GUI, more emphasis is needed. Additionally, please include more detailed guidance in the supplementary material.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: * For larger scenes with multiple objects, how does the runtime performance scale? Are there specific optimizations that could be applied to maintain the method's real-time processing speed in more complex compositions?
* How could the proposed method be adapted for objects with highly specular or translucent materials? The experiments in the paper primarily focus on diffuse materials—would adjustments be needed to handle complex reflectance and transparency properties?
soundness: 3
strengths: * The proposed approach addresses a key gap in example-based modeling by enabling realistic and seamless stitching of 3D objects from real-world scenes, which previous techniques like SeamlessNeRF struggle with. This makes the method highly applicable for real-world applications that require detailed, cohesive compositions.
* Extensive experimentation shows that this approach achieves high-quality, harmonious blends even in challenging real-world cases, where traditional neural field blending methods fail.
summary: This work proposes a new method for realistic 3D object composition by combining parts of existing models. Unlike prior approaches that focus on shape composition and struggle with real-world 3D objects, this method uses 3D Gaussian Splatting to achieve seamless blending of textures and structures between objects. A novel sample-guided synthesis approach allows for real-time segmentation and transformation of these objects. Additionally, a two-phase optimization process—sampling-based cloning and clustering-based tuning—ensures both local texture harmonization and global appearance consistency. Extensive experiments demonstrate that this method outperforms existing techniques like SeamlessNeRF, offering more realistic synthesis and interactive editing capabilities in real-world scenes.
weaknesses: * Although the paper’s experiments on real-world data show improvements over SeamlessNeRF, it could further benefit from a more diverse dataset, as the current scope primarily includes simple object compositions without significant lighting variation or occlusion. Expanding the dataset to more complex scenes or settings with varied lighting conditions and object types could better showcase the generalizability of the approach.
* The paper primarily demonstrates its effectiveness on objects with diffuse materials, which may limit its applicability to scenes involving highly specular or translucent objects. Complex materials with reflective or transparent properties could introduce additional challenges for the proposed method.