Title: Track, Inpaint, Resplat: Subject-driven 3D and 4D Generation with Progressive Texture Infilling
Venue: NeurIPS 2025
Accepted: yes
Keywords: Subject-driven Generation

Personalization

3D and 4D Generation
Abstract: Current 3D/4D generation methods are usually optimized for photorealism, efficiency, and aesthetics. However, they often fail to preserve the semantic identity of the subject across different viewpoints. Adapting generation methods with one or few images of a specific subject (also known as Personalization or Subject-driven generation) allows generating visual content that align with the identity of the subject. However, personalized 3D/4D generation is still largely underexplored. In this work, we introduce TIRE (Track, Inpaint, REsplat), a novel method for subject-driven 3D/4D generation. It takes an initial 3D asset produced by an existing 3D generative model as input and uses video tracking to identify the regions that need to be modified. Then, we adopt a subject-driven 2D inpainting model for progressively infilling the identified regions. Finally, we resplat the modified 2D multi-view observations back to 3D while still maintaining consistency. Extensive experiments demonstrate that our approach significantly improves identity preservation in 3D/4D generation compared to state-of-the-art methods. Our project website is available at https://zsh2000.github.io/track-inpaint-resplat.github.io/.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTIRE (Track, Inpaint, REsplat)  method for 3D/4D generation with semantic consistency across views.\nThis method maintains the consistency of the target through three stages of processing. The tracking strategy adopts a backward tracking optimization scheme that supports any 3D representation form to improve tracking accuracy. The repair strategy uses a progressive repair strategy to solve the consistency problem when the target perspective and the source perspective differ too much. The final reconstruction stage uses multi-perspective consistency optimization to improve the visual consistency of the final result.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The three-stage approach solves several core contradictions in visual consistency, and creatively uses backward tracking to improve the effect.\n- From the experimental results, compared with the existing SOTA 3D solutions (including closed-source solutions), it is indeed better in terms of visual consistency.\n- In theory, this method can handle a variety of different 3D representations well.\n\nWeaknesses:\n- The experimental phase did not experiment with different 3D representations, so it is not possible to judge which representation the method is most suitable for and which representation it will have problems with.\n- The method part does not explain much about backward tracking, and does not give the specific implementation of backward tracking and the problems that may arise (why a solution like cotracker can use backward tracking, and why backward tracking is used in the field of 3D generation)\n- Figure 1 is somewhat misleading. In theory, the whole process can be automatically run in stages, but Figure 1 looks like it requires more human thinking and intervention.\n\nQUESTIONS:\n- For tracking, which 3D expression works best and which works worst? Why? Please add some ablation experiments\n- For tracking, backward tracking. What are the disadvantages of backward tracking, and what are the common situations where backward tracking fails and normal tracking succeeds?\n- For the inpaint phase, it is necessary to explain the correlation between the current progressive design and loss design, and add experiments with different degrees of progressiveness.\n- Please optimize the description of Figure 1 so that it does not appear to be an automatic annotation or determination method based on LLM.\n-\n\nLIMITATIONS:\n- The multi-stage solution will accumulate errors. At the same time, there is no overall optimization loss or optimization goal between the stages, or in the end, resulting in the three stages being equivalent to three independent algorithms.\n- The algorithm based on progressive rotation restoration is prone to misalignment in details and will have obvious segmentation lines. You can try to give some experimental results of leopards with spots or giraffes with more patterns.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method for generating a 3D model from a single RGB image such that the identity of the subject is preserved. This is done in a progressive manner by first creating a coarse 3D representation using a pre-trained 2D-to-3D model; rendering this representation from novel viewpoints; tracking regions of the rendered views that are not visible in the source image; inpainting these regions using an identity-aware diffusion-based inpainter; and finally, combining all rendered+inpainted views into a single Gaussian representation after enforcing consistency among views.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The method shows fairly good results using pre-trained and fine-tuned components.\n2. The idea of fine-tuning an identity-preserving inpainting model for view synthesis is novel and interesting.\n\nWeaknesses:\n1. The motivation for the task the paper is solving, and what exactly \"identity preservation\" means is not clearly defined in the introduction. This is especially important since \"personalization\" and \"identity preservation\" are terms more commonly used by generative models which use text prompts to generate subjects in novel settings (\"an sks dog in an astronaut suit\"). But such text-based control doesn't seem to be present in the current model. \n2. The qualitative evaluation is limited solely to pictures of cats and dogs. In these cases it is difficult to judge how effectively identity is preserved. Furthermore, the title of the paper implies applicability to general objects, not just animals. Therefore, this should be supported with additional qualitative results on diverse objects. \n3. The paper claims to work for 4D reconstruction based solely on the assumption that all its steps translate to video. But no results on 4D reconstruction are shown. This claim should be toned down. \n4. The evaluation section does not support the claims of the paper. Specifically, the qualitative results of the method, while good, are not clearly better than past work. In fact, they may arguably be worse than Trellis and Hunyuan3D. The quantitative results are also not better. If, as the authors claim, the quantitative metrics are incapable of capturing the improvement of the method, then they should explore or devise metrics that better capture the contribution of identity preservation. Perhaps the authors can refer to the \"VBench\" paper to get a comprehensive list of metrics for generative tasks. Furthremore, the user study does not ask participants to judge identity preservation, which is the main contribution of the paper.\n5. Section 3.1 provides far too much detail about diffusion models that is not directly related to the task at hand. Diffusion models are common enough now for this information to be known beforehand by readers. If absolutely necessary, it should be moved to a supplemental. The additional space can be used for more qualitative results as suggested above.\n\nQUESTIONS:\n1. The paper does not provide any details about how the iniital 3D model is generated. In particular, what method is used to lift the image from 2D to 3D? \n2. It is not clear to me how the inpainting model can be successfully finetuned on an identity with a prompt such as \"a photo of sks\" which does not include a class (\"a photo of an sks dog/cat/vase\") of the subject. Some more details and justification for the choices made here would be very helpful.\n3. Why don't the authors compare all baselines in the quantitative evaluation stage? Specifically, the SOTA methods Trellis and Hunyuan3D are not included in Table 1.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a pipeline for enhancing the appearance preservation (personalization) in image(video) to 3D(4D). First a 3D(4D) is initialized using ImageDream (and Wonder3D), then three steps are performed: Track - visibility masks are created for the 3D/4D representation with respect to the source view using a video tracker. Inpaint - the masks are progressively inpainted using a personalized inpainting model (similar to RealFill). Resplat - the multi-view images are resplatted using LG(4)M to obtain a 3D(4D) gaussian splatting representation. The pipeline can be seen as an extension for LG(4)M. Experimentally, they compare their method with 6 img-3D models and 3 vid-4D models and also conduct a user study.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The method improves the shadowy bias of L(4)GM (Figs. 4-5), and corrects some geometry artifacts (Fig. 6), without heavy training, relying only on light inpainter fine-tuning and off-the-shelf models. It can be applied on top of any image-to-multiview (the initialization) and multiview-to-3D (the resplatting) models.\n2. Since 3D/4D generation typically relies on small and scarce 3D/4D datasets (excluding slow SDS-like methods), this approach effectively combines geometry from methods trained on 3D/4D data with appearance from strong 2D generative models.\n3. Progressive personalized inpainting, combined with visibility masks from a video tracker, is a solid and interesting idea that could be useful in other contexts.\n\n**Major weaknesses:**  \n1. Appendix F states that the authors use Wonder3D for 3DGS color features and ImageDream for geometry features, due to ImageDream’s poor color quality. This makes it unclear whether appearance improvements stem from the inpainting module - the main contribution, or simply from using Wonder3D as the image-to-multiview backbone for color. An ablation without inpainting is needed to isolate its impact.\n2. From my understanding, progressive inpainting should not introduce inconsistencies in an image-to-3D setting. Could the authors provide examples of such inconsistencies and clarify how the 30% denoising steps in the image-to-multiview model addresses them? Additionally, since each video frame at timestep t is processed independently, wouldn’t this lead to temporal inconsistencies across frames?\n\nMinor Weaknesses:\n1. Regarding reproducibility: most inpainting pipelines use a strength parameter, and I believe “30% of the denoising schedule” corresponds to setting this parameter to 0.3. It would be helpful to clarify this explicitly in the appendix.\n2. Line 202: “A photo of sks” — It would be more appropriate to denote the rare token as $\\hat{V}$ in the main text, and provide the exact token in the appendix. Additionally, in prior work, rare tokens are often prepended to a general class name. Have you considered extracting a general class (e.g., via BLIP or an LLM) to guide the inpainter more effectively?\n\nTypos:\n           Eq. 7, I believe that term should include a squared norm, please verify.\n\nQUESTIONS:\n+ Could you clarify whether the observed appearance improvements are due to the proposed inpainting module or simply the use of Wonder3D for 3DGS color features? An ablation without inpainting would help isolate its effect and increase the score.\n\n+ Can you provide specific examples of inconsistencies that arise without progressive inpainting?\n+ Given that each frame at timestep t is processed independently in the pipeline, how do you address potential temporal inconsistencies across frames in video sequences?\n+ Does the “30% of the denoising schedule” correspond to setting the inpainting strength parameter to 0.3? If so, could this be clarified explicitly in the appendix?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tries to address the problem of 3D and 4D generation while preserving the subject identity. They first use off the shelf point tracker to find out regions that need to be inpainted and then rerun 3D reconstruction to obtain better 3D/4D generation. They compare the proposed pipeline with some existing 3D/4D reconstruction methods and show that it can indeed improve the unnatural back side views.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\nThe paper is overall well written, it is easy to follow the whole pipeline. \n\n**Weakness**\n- Significance: the problem this paper tries to address seems to be very specific to LGM -- the backside views do not look natural and faithful to frontview, same for LG4M. The root cause of this is actually comes from ImageDream which generates blueish images in the backside views. I am not sure this is a really meaningful problem, as this might be largely due to data issue. hence can be trivially solved with better image renderings and fine tune the model. \n- Originality. The whole pipeline is basically combining several blocks together and making the system work. There is not much technical contribution. \n- Quality. The evaluation is not solid. it is not convincing that the image to 3D is actually better than TRELLIS or HY3D. only a few qualitative examples are shown.\n\nQUESTIONS:\n- I am mainly concerned with the limited scope of this paper.. Authors motivates this method by showing that some methods generate unrealistic backside views which are not faithful to the input views. The example comes from LGM, but what is the real reason for this? As far as I know, this is due to the multi-view diffusion stage of ImageDream. A simple way to address this is to fine tune ImageDream with better images. If full model fine tuning is too expensieve, LoRA based fine tuning could also be considered. In this regard, the pipeline is designed to address a problem specific to one particular model. In order to actually show this method is general, can the authors demonstrate: \n1. The backside artifact/identity not preserving problem is a general problem that lots of single image to 3D/4D generation models have? e.g. if the root cause of LGM is due to ImageDream, then one can simply address it by using another multi-view diffusion models. e.g. Zero123++. In fact Gaussian reconstruction model (GRM, Xu et al.) can take multi-views output from different models. \n2. If the problem does exist in lots of models, can the authors show that the proposed pipeline is general to fix this type of problems? e.g. can this method also take other multi-view/3D generation models as input and improve their artefacts? \n\n\n\n- Experiment evaluation is not solid. The paper claims to address both 3D and 4D generation. Both tasks are not properly evaluated and the results are not convincing. \n1. For 3D, why is there only qualitative result? The selected examples from TRELLIS and HY3D do not look like the actual performance one would see for object reconstruction. Quantitative evaluation with at least 100 examples is needed, i.e. via CLIP score or user study, or more advanced 3D generation metrics (e.g. MET3R, CVPR'25). \n2. For 4D, why is SV4D not compared is the user study, and also table 1? This is related to the first point, does SV3D/SV4D actually have the problem of artifacts in the backside views?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 1, "ORIGINALITY": 1, "DATE": "2025-07-04"}