GG-Editor: Locally Editing 3D Avatars with Multimodal Large Language Model Guidance

Authors: yunqiu  xu
linchao  zhu
yi  yang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-01 07:41:56

Abstract: Text-driven 3D avatar customization has attracted increasing attention in recent years, where precisely editing specific local parts of avatars with only text prompts is particularly challenging. Previous editing methods usually use segmentation or cross-attention masks as constraints for local editing. Although these masks tightly cover existing objects/parts, they may limit editing methods to create drastic geometry deformations beyond the covered contents. From a different perspective, this paper presents a GPT-guided local avatar editing framework, namely GG-Editor. Specifically, GG-Editor progressively mines more reasonable candidate editing regions via harnessing multimodal large language models which already organically assimilate common-sense human knowledge. In order to improve the editing quality of the local areas, GG-Editor explicitly decouples the geometry/appearance optimization, and adopts a global-local synergy editing strategy with GPT-generated local prompts. Moreover, to preserve concepts residing in source avatars, GG-Editor proposes an orthogonal denoising score that orthogonally decomposes editing directions and introduce an explicit term for preservation. Comprehensive experiments demonstrate that GG-Editor with only textual prompts achieves realistic and high-fidelity local editing results, significantly surpassing prior works. Project page: https://xuyunqiu.github.io/GG-Editor/.

Main paper: 
 introduction: 
 Text-driven 3D avatar generation [26, 27, 41, 71, 74, 78] and editing [5, 20, 49, 60, 73] are crucial for various applications and industries, which enable the creation of customized digital humans using a few words. Despite notable advancements in 3D editing [8, 19, 21, 57, 79] , locally editing some specific parts of 3D contents remains challenging, due to the ambiguity in language and the complexity of 3D space. In training, the editing-related regions should be specified and properly manipulated, while the irrelevant contents are maintained.Prior local editing methods usually locate the candidate editing regions via manual designation (e.g., sketch) [51] , cross-attention attribution [59, 80] or open-vocabulary grounding/segmentation [33, 63] with manually selected text queries. Manually specifying an editing region or selecting a suitable text query for grounding it could be cumbersome and inflexible. Furthermore, it is difficult to ground some concepts that are not in current images with text queries, as shown in Fig. 3a . The editing regions indicated by attention/segmentation maps usually tightly cover existing objects within given images. These object/part masks may not be a good constraint for editing, since there could be a misalignment between the candidate editing regions and the object/part masks. For instance, the optimal region to add a hat is the area above the head, rather than the head region itself. Consequently, it is hard to introduce some non-rigid edits (e.g., drastic geometry deformations), if the assigned editable region is the head region itself.Motivated by these observations, we try to mine more reasonable editing regions using multimodal large language models (LLMs) [1, 3, 4, 7, 29, 42, 45, 47, 52, 66, 69] which already organically assimilate common-sense human knowledge. In this paper, we exploit the extraordinary text parsing and spatial reasoning capability of multimodal LLMs (i.e., GPT-4V [1] ), and present a GPT-guided 3D avatar editing framework, namely GG-Editor. In contrast to the existing LLM-guided generation methods [39, 40, 77] , our editing method also requires a good understanding of the existing visual contents and reasoning in 3D space. To enable multimodal LLMs to handle 3D inputs and locate reasonable regions, we decouple the region seeking process into multiple steps: representative view selection, coarse grid region selection and iterative fine region mining. In addition, we inject some domain knowledge regarding avatars and devise various visual prompting strategies to enhance the grounding capability of GPT. As a result, GG-Editor progressively mines some local regions corresponding to the given editing prompts, and can use the mined regions for subsequent editing.We incorporate the mined local editing regions into a geometryappearance decoupled learning scheme [9, 53] , where the geometry and appearance of the pre-trained source avatars are edited sequentially. We notice that using a standard human-centric camera pose sampling system may be less effective for local editing, as the local editing regions could be small or occluded in the rendered full-body images. To provide high-quality edits with more geometry and texture details, GG-Editor employs a global-local view synergy editing strategy that simultaneously renders images from global and focal views. Nevertheless, the semantics of the local-view images could be deviate from the given prompt describing the global avatar. To cope with such semantic misalignment issue, we leverage GPT to analyze the source/target prompts, and then generate local prompts tailored for the local view.Many prior 3D editing works [38, 80] optimize models using score distillation sampling (SDS) [55] with only target prompts. We believe that it is also important to exploit the information residing in source avatars and corresponding prompts. Drawing inspiration from an image editing approach [23] , we treat the original contents as the reference and calculate the delta scores that steer the editing toward a less biased direction. We also observe that such delta score function may sometimes bring over-editing results that largely deviate from the source avatars. Thus, we orthogonally decompose the condition directions, and present a new orthogonal denoising score (ODS) loss that contains an explicit term to adjust the preservation of the original contents. In this way, GG-Editor brings well compositional and high-fidelity edits.To the best of our knowledge, GG-Editor is the first multimodal LLM-guided framework for text-driven 3D avatar editing. We showcase it on multiple avatars with various editing prompts. Comprehensive experiments validate its superiority in locally editing avatars. The main contributions can be summarized as follows:• We introduce a new GPT-guided framework for zero-shot text-driven 3D avatar editing, which first integrates commonsense human knowledge and progressively mines reasonable candidate regions for local editing. • We devise a global-local view synergy editing strategy to improve the local editing results by training models with additional local renderings and GPT-generated local prompts. • We present ODS loss that orthogonally decomposes the editing directions and introduces an explicit term to adjust the preservation of the source concept. 
related works: 
 Controllable Text-Driven 3D Content Editing. Most current text-driven 3D editing methods globally manipulate (e.g., style transfer) the whole scenes [21, 31, 34, 49, 65] or objects [10, 19, 20, 50, 57] . Implementing local editing and maintaining the unrelated areas is more challenging, which requires models to have a more finegrained understanding of 3D contents and editing prompts. To improve the local editing controllability, some methods manually assign multi-view semantic sketches [51] or editable regions [6, 12, 36, 38] as auxiliary constraints. However, manually adding such constraints in a 3D space could be cumbersome and inflexible.Numerous methods [11, 15, 22, 33, 63, 73, 76] try to generate masks as constraints for local editing, using off-the-shelf openvocabulary segmentation or grounding methods [32, 43] . A few works [13, 14] attempt to locate the semantically related local regions on mesh surface using CLIP guidance [56] . Another line of work [28, 59, 79, 80] utilizes cross-attention mask-based techniques to obtain the editable regions. While these mask-based methods show promising results regarding rigid editing (e.g., changing appearance), they usually struggle to bring drastic geometry deformations, as the editable regions have been aligned with existing objects/parts indicated by a manually selected text query. This paper, from a different perspective, attempts to mine reasonable editing regions with only text prompts and presents a local editing method effectively utilizing the obtained editable regions. LLM-Guided Visual Content Generation. LLMs, like GPT [1] and BLIP [37] series, have exhibited outstanding efficacy in many text-related tasks. Lv et al. [46] and Sun et al. [64] utilize LLMs to generate Python scripts commanding 3D software (e.g., Blender) for text-to-video and 3D generation. Gao et al. [18] use GPT to generate scene graphs assisting compositional 3D scene generation. Another branch of research explores LLMs to generate various types of textgrounded layouts (e.g., boxes and polygons) as constraints for textto-image [16, 39, 70, 75] , video [40, 44] and 3D generation [67, 77] . Unlike prior works that employ LLM to generate layouts that reflect the spatial relationships and motions described by given texts, we aim at exploiting multimodal LLMs to better understand both the textual and 3D visual inputs and then infer reasonable local regions for 3D editing. 
preliminary: 
 Geometry-Appearance-Decoupled 3D Representation. Learning 3D representations with explicit disentanglement of geometry and appearance has shown its effectiveness in 3D reconstruction [53] and text-to-3D generation [9] . In light of these findings, we adopt a similar two-stage scheme for editing.In the first stage, DMTet [61] is utilized as the geometry representation 𝜃 geo , which can efficiently render high-resolution meshes with differentiable rasterization [35] . DMTet models 3D shapes using a deformable tetrahedral grid and an implicit SDF [54] . The SDF values and the position offsets of deformable tetrahedral vertices are learned using a MLP in DMTet. In training, the explicit mesh can be extracted through the differentiable marching tetrahedral layer.Once the geometry model 𝜃 geo is trained, an extra physicallybased rendering (PBR) material model [48] is adopted to learn the appearance representation. The material model 𝜃 mat is parameterized using a MLP, which outputs diffuse value 𝑘 d , roughness and metallic value 𝑘 rm , and normal perturbation value 𝑘 n for any point on the mesh surface extracted from 𝜃 geo . When both representations are optimized, we can produce textured meshes that are compatible with standard 3D tools and game engines.Score Distillation Sampling. SDS proposed by Poole et al. [55] has become a popular way to distill the diffusion priors for textdriven 3D generation and editing. Formally, given a diffusion model 𝜙 and images x = 𝑔(𝜃, 𝑝) generated with a differentiable renderer 𝑔(•) and a camera pose 𝑝, SDS minimizes the difference between the added Gaussian noise 𝜖 and the predicted noise 𝜖 𝑠 𝜙 :EQUATIONwhere 𝑦 indicates the text condition and z 𝑡 is obtained by adding noise 𝜖 to x corresponding to the 𝑡-th timestep of the diffusion process. 𝑤 (𝑡) denotes a weighting function determined by the time step 𝑡, and 𝜖 𝑠 𝜙 z 𝑡 ; 𝑦, 𝑡 is the classifier-free guidance (CFG) [25] :EQUATIONwhere ∅ is a null condition. The conditioned prediction 𝜖 𝜙 (z 𝑡 ; 𝑦, 𝑡) of the noise is extrapolated away from the unconditioned prediction 𝜖 𝜙 (z 𝑡 ; ∅, 𝑡) by an amount controlled by a scalar 𝑠. During training, the diffusion model 𝜙 is frozen and the gradients are backpropagated to the parameterizable 3D representation 𝜃 . 
methodology: 
 Given a 3D avatar along with a text prompt describing the original visual content (i.e., source prompt), our goal is to locally edit the avatar using another target prompt specifying the content after editing. It requires imposing proper manipulation on specific local regions corresponding to the editing prompts and retaining other editing-extraneous regions. Furthermore, the edited avatars should look realistic and keep cross-view consistency.To achieve the above objectives, we propose GG-Editor a new GPT-guided avatar editing framework (see Fig. 2 ). Given an avatar with source and target prompts, we utilize multimodal LLMs (i.e., GPT-4V) to analyze and gradually seek reasonable regions for editing. We devise a coarse-to-fine pipeline with various prompting strategies to alleviate the hallucination issue and obtain more accurate regions. The mined candidate editing regions are integrated into a geometry-appearance decoupled pipeline for local editing. For higher-quality local edits, a global-local synergy editing strategy is employed to optimize models with additional local views with GPT-generated local prompts. Moreover, we introduce an orthogonal denoising score, which performs 3D editing effectively and introduces an explicit term controlling the preservation of the source concept.Unlike segmentation-based editing methods using accurate masks that cover existing contents, we aim to seek reasonable editing regions beyond given avatars with multimodal LLM guidance. Locating 3D local editing regions indicated by source/target prompts is quite difficult, though the powerful GPT is leveraged. We split the overall process into several steps and introduce a coarse-to-fine pipeline, gradually mining the candidate editing regions.Representative View Selection. To enable GPT to understand the 3D contents, we first project 3D avatars to 2D space. We empirically render four images from the orthogonal views (i.e., front, back, left and right views) that are informative enough to represent the input meshes. Then, we request GPT to select a pair of images from the rendered images, where the selected image pair should be view-orthogonal and can better present the candidate edits.Drawing inspiration from chain-of-thought [68] , we also employ reasoning before answering prompting strategy to improve the robustness of multimodal LLMs. Specifically, we prompt GPT to first provide the descriptions of the given images and candidate edits, as well as the reasoning for the decision making before returning the selected views.Coarse Region Mining by Grid Selection. With the selected two renderings, we attempt to locate the regions relevant for candidate editing. However, we notice that GPT usually struggles with producing accurate coordinates, especially at fine granularity, as shown in Fig. 3b . Thus, we devise several visual prompting [62, 72] strategies to unleash its locating capability.We encode visual prompts to help GPT identify different regions within images. Concretely, we divide the images into different grids and assign different identifiers to each grid. We ask GPT first to analyze the required image modifications based on given source/target prompts and then select several grids within the encoded images that are most relevant. The selected grids roughly indicate the regions for editing.Fine Region Mining via Iterative Verification. The fine region mining is formulated as an iterative proposal selection process. At each round of selection, one optimal box is selected as the reference, and then we jitter the coordinates of the reference box to generate multiple box proposals around the reference and avatar for the next round. At the beginning of selection, we initialize the reference box proposal based on the selected grids.To enable GPT to be aware of different regions, we encode visual prompts in the rendered images by indicating these boxes with different colors, as shown in Fig. 3c . We also provide GPT with some key guidelines as well as the process steps for selecting a good proposal for editing. At each round of selection, GPT first interrupts the editing task and analyzes some key factors, based on the source/target prompts. GPT describes the regions covered by each box and scores each region with the corresponding reasoning before returning the final selection. Since the selected proposals could be noisy, we introduce a chain-of-verification strategy to alleviate the hallucinations of GPT. Concretely, the selected boxes in different rounds are stored, and we verify the previously selected boxes every 𝑁 ver rounds. After multiple rounds of selection, we can obtain a reasonable candidate region for local editing.Optionally, if the mined region does not meet the user's specific requirements, we can also interactively chat with GPT to further rectify the box to determine a better editing region. After obtaining the coordinates of two dimensions (e.g., X-axis and Y-axis) from the first view, we perform a similar coarse-to-fine mining process on the other view using the fixed Y-axis coordinates to obtain the coordinates in another dimension (e.g., Z-axis). We project all mined coordinates back to 3D space and generate a 3D bounding box indicating the editable regions.Geometry and Appearance Editing with Local Constraints. As in the geometry-appearance decoupled framework discussed in Sec. 3, we sequentially manipulate the geometry and appearance of the given avatars. We first initialize the geometry and appearance models of the target avatar using the pre-trained source avatar: {𝜃 With a randomly sampled camera pose, we simultaneously render both the source and target avatars as well as the 3D bounding box indicating the candidate editing regions. Specifically, we calculate the editable map m, based on the rendered depth map d box ∈ [0, 1] and object mask o box ∈ [0, 1] of the 3D box and the source avatar's depth map d src :EQUATIONwhere 1 {d box ⩽d src } is the indicator function, being 1 if d box ⩽ d src and 0 otherwise. To maintain the original contents, in both geometry and appearance editing stages, we impose reconstruction losses on the non-editable regions:EQUATIONwhere x ∈ {n, c} denotes the normal/shading in geometry/appearance editing stages respectively. Similar to SDS loss in Eqn.(1), we can distill the prior knowledge from a pre-trained text-to-image diffusion model [58] to the editable regions m for editing target avatar's geometry/appearance:EQUATIONwhere 𝑦 tgt is the target prompt, and z tgt 𝑡 ∈ {z 𝑡 (n tgt ), z 𝑡 (c tgt )} represents the noisy normals/shadings. 𝜃 tgt ∈ {𝜃 tgt geo , 𝜃 tgt mat } indicates the parameterizable 3D representation of avatar's geometry/texture.As the editable regions are determined, we present a global-local viewpoint synergy strategy to improve the editing quality of the local regions. Besides randomly sampling camera poses 𝑝 glb around the whole body as in [9] , we further set up another spherical coordinate system centered on the 3D box and sample some focal views 𝑝 loc around the local editing regions. At each optimization step with Eqns. ( 4) and ( 5), we simultaneously render normals/shadings and editable masks from global and local views (see Fig. 4 ) to enrich geometry/texture details within local editing regions.However, using shared prompts for both global and local views may not be optimal in training, because the semantics of local parts may differ from that of global views. As shown in Fig. 2b , using a prompt "Hulk holding a shoe" does not accurately describe the contents covered a local view that covers only the hand regions of the Hulk. To cope with this issue, we harness the reasoning capability of GPT to parse global source and target prompts, and infer the local prompts for training. Specifically, given the source and target prompts, we ask multimodal LLMs to analyze the candidate editing areas as well as the interactions with the avatars, and then generate more appropriate prompts (e.g., "hand holding a shoe") for local views.SDS loss is initially designed for text-to-3D generation and has become a common practice in many 3D editing works [38, 59, 80] . However, we believe it may not be optimal for editing tasks. As shown in Fig 2c , SDS loss leads samples from different views to one concept center defined by target prompts. We argue that it is also important to exploit the information residing in the original inputs (i.e., source avatar and corresponding prompt) for 3D avatar editing.Drawing inspiration from an image editing method [23] , we use the images rendered from the source avatar as the reference to help the optimization. We add the identical noise 𝜖 to source and target inputs, and calculate the delta denoising score (DDS) loss:EQUATION)Following the assumption introduced by Katzir et al. [30] , we decouple the CFG score 𝜖 𝑠 𝜙 (z 𝑡 ; 𝑦, 𝑡) in SDS loss into three components:EQUATIONwhere 𝛿 C is the condition direction leading the generated image towards the text condition 𝑦. 𝛿 D and 𝛿 N are domain correction and denoising direction respectively, which are not directly related to the editing prompts. Since the editing is from a pre-trained avatar that can render in-domain images, 𝛿 D component is not effectively required and can be dropped. The noisy residual 𝛿 N -𝜖 is relatively negligible and also can be dropped. Consequently, Eqn. ( 6) can be reformulated as:EQUATIONThough optimizing models using the delta of source and target condition directions improves the editing effectiveness, it is still prone to bring edits that deviate significantly from the source avatars. To tackle this issue, we try to disentangle the editing direction, inspired by Perp-Neg [2] . Specifically, we orthogonally decompose the condition directions and introduce an explicit term to preserve source-concept contents, as shown in Fig. 2c . Suppose the projection and perpendicular of 𝛿 tgt C on 𝛿 src C are:EQUATIONwe present a new orthogonal denoising score (ODS) function and use it in both geometry and appearance editing stages:EQUATIONwhere 𝜆 proj is an adjustable term for source concept preservation. VoX-E GG-Editor (ours) Figure 5 : Qualitative comparisons to three baselines on four different cases. The results show that our GG-Editor achieves realistic local editing results corresponding to the given prompts while better preserve the irrelevant regions. In addition, GG-Editor also shows high-quality results with richer geometry and texture details in the local editing regions. The source prompts are colored in gray. 
experiments: 
 Implementation Details. We optimize our models with four NVIDIA RTX 3090 GPUs. For each avatar, we optimize the geometry and appearance models for 3K iterations (∼60 mins) and 2K iterations (∼40 mins) respectively. We adopt Stable Diffusion v1.5 as the diffusion prior for both geometry and appearance stages. AdamW optimizer is utilized with a learning rate of 10 -3 and 10 -2 for the two stages respectively. DMTet grid resolution is set to 128. The overall batch size is set to 4, where 2 for the local view. 𝑁 ver and 𝜆 proj is empirically set to 5 and 0.2. The source avatars can be generated using Fantasia3D [9] or reconstructed using nvdiffrec [53] .Evaluation Metrics. Following previous 3D editing works, we adopt CLIP similarity (CLIP sim ) [56] , CLIP directional similarity (CLIP dir ) [17] , Frechét Inception Distance (FID) [24] and peak signalto-noise ratio (PSNR) for quantitative evaluation. CLIP sim measures the alignment between the target avatars and the target prompts. CLIP dir evaluates the alignment between the changes in both the avatars and text prompts. FID validates the edit magnitude, and PSNR quantifies the ability to preserve the source contents. Since the quality assessment of editing results could be subjective, we also conduct user studies for evaluation. Concretely, we provide 360 • videos of source avatars and multiple target avatars edited by different methods, and ask users to select the best based on the local editing quality and the similarity to the source avatars. We compare our GG-Editor to several recent advanced methods i.e., Fantasia3D [9] , DreamEditor [80] and Vox-E [59] . As Fantasia3D is a text-to-3D generation method that can be initialized with custom meshes, we adapt it to editing by fine-tuning pre-trained source avatars with target prompts and lower learning rate.Qualitative Comparisons. As shown in Fig. 5 , without local region constraints, Fantasia3D-FT is prone to manipulate the entire avatars. DreamEditor and VoX-E can sometimes locate reasonable regions for local editing, but they usually show unrealistic results with limited geometry changes. Our GG-Editor successfully mines reasonable local editing regions and imposes faithful editing in relevant regions to the textual prompts, while the irrelevant regions are properly retained. We present more results of our GG-Editor in Fig. 6 . Various high-quality normal and shaded RGB renderings validate the effectiveness and generalization of our local editing method. Please find more examples in the supplementary material. Quantitative Comparisons. We report the quantitative results of GG-Editor compared to several state-of-the-art methods in Table 1. Although GG-Editor manipulates a few local regions within source contents, it achieves the best results on CLIP sim , CLIP dir and FID, demonstrating its excellent editing capability. Meanwhile, GG-Editor also obtains the second-best results on PSNR, indicating the strong ability to preserve the source contents. Moreover, GG-Editor receives 81.4% of the votes in user studies, which further validates the advantages of our local editing method from the perspective of human preferences.Effectiveness of Global-Local Synergy. To verify the necessity of the global-local view synergy strategy, we showcase an ablation on a challenging case (i.e., Hulk holding a shoe) in Fig. 7 . We compare it to the model trained using only global views and the model trained with global and local views but without the local prompts. In Fig. 7a , we can find blurry results are achieved when training without the local view. After adding the local views, edited results have more geometry and texture details. However, using a global prompt "Hulk holding a shoe" is not optimal to describe the local region around the hand, which could bring noise in the optimization process. In our global-local synergy, the local region can be specified by the GPT-generated local prompt, which facilitates the local editing. Effectiveness of Orthogonal Denoising Score As shown in Fig. 8 , the shoe optimized using SDS is still unclear and unrealistic, while DDS and ODS effectively add a realistic shoe to the hand. Since the concept of the shoe is quite close to that of the foot, the Hulk's hand becomes a foot through optimization with DDS (see hand regions in the bottom images). In contrast, ODS can better preserve the source concept and retain the hand in the back view. In Fig. 9 , We also analyze the effectiveness of orthogonally terms in ODS loss. Without projection and perpendicular terms, the edited avatars show obvious artifacts and ineffective editing respectively. Effectiveness of GPT-Guided Editing Region Mining. In order to show the superiority of the proposed GPT-guided coarse-to-fine editing region mining approach, we measure the overlap of the mined local editing regions with respect to the human-preferred areas using mIoU. As shown in Fig. 10 , compared to grounding DINO [43] and directly predicting coordinates via GPT, our proposed method can select candidate editing regions that are more consistent with human preferences.As the optimization process of geometry and appearance is explicitly decoupled, it is easy to retexture the avatars with fixed shapes, as shown in Fig. 11 . Besides addition and modification editing, GG-Editor can also performs removal editing as illustrated in Fig. 12 Another characteristic of our proposed method is that GG-Editor can make local edits incrementally. Fig. 13 showcases an example of incremental editing. In each editing step, only the contents within the local editing regions are manipulated, while contents irrelevant to the editing are retained. In addition, GG-Editor directly exports textured avatar meshes that can be used for various downstream applications like relighting and animation in the classic graphics pipeline, as visualized in Fig. 14 . 
limitations and future works: 
 As a pioneer in taming multimodal LLMs for text-driven 3D local editing, GG-Editor presents realistic editing results. However, due to the limited performance of existing text-to-image diffusion models [58] for hand and human-object interaction generation, we find it sometimes fails to edit challenging cases faithfully (see left part of Fig. 15 ). As our method maintains the editing irrelevant contents by directly regressing the normals and colors of source avatars, our method could bring some artifacts around the boundaries of the local editing regions (see middle part of Fig. 15 ). Since GPT-4V lacks accurate localization capacity, it is hard for GG-Editor to localize some tiny regions from avatars precisely, and such mislocalization may result in some unsatisfactory results (see right part of Fig. 15 ). GG-Editor is an early attempt at 3D local editing with multimodal LLM guidance, focusing only on 3D human avatars. In the future, we would like to explore mining local editing regions from more challenging and general scenes. In addition, we will investigate enhancing the locating capability of multimodal LLMs, thereby improving the controllability of 3D local editing. 
conclusion: 
 This paper proposes a new multimodal LLM-guided framework for locally editing 3D avatars, namely GG-Editor. GG-Editor harnesses GPT-4V combined with human common sense knowledge to infer some reasonable local editing regions beyond existing avatars. To enrich the geometry and texture details within local editable regions, we devise a global-local view synergy editing strategy. Integrating it into a geometry-appearance decoupled learning pipeline, GG-Editor achieves high-fidelity local editing results with crossview consistency. Besides, we present ODS loss that orthogonally decomposes editing directions and introduces an explicit term for adjusting source concept preservation. Experiments with multiple avatars and various editing prompts showcase the effectiveness and superiority of our GG-Editor for local avatar editing. 
 
 Appendix:  


Decision: All reviewers give positive ratings after the rebuttal. The AC agrees with the reviewers and recommend acceptance.

Reviews: review: paper_summary: paper_summary: This work uses GPT-4V to guide 3D avatar editing tasks. The geometry and texture of 3D avatar are represented by a DMTet ($\theta_{geo}$) and an MLP, respectively ($\theta_{mat}$). The editing process is divided into two steps:

1. Identifying the modification area: GPT-4V selects the appropriate editing area based on the prompt and maps the 2D editing area back to 3D.
2. Fine-tuning: Ensures consistency in unedited areas and introduces an Orthogonal Denoising Score to guide the fine-tuning process.

This work integrates several proposed methods, including Visual ChatGPT [1], FocalDreamer [2], and Delta Denoising Score [3]. Hence, the core innovation of thie work is not clear. Additionally, there are some weaknesses in Evaluation (see limitations). Nevertheless, the overall clarity and writing quality of the paper are high, leading to a borderline acceptance rating.
strength_weakness: strength_weakness: 1. The article has a clear logic and high writing quality, making the overall approach easy to understand. It also includes a video demo.
2. GPT-4V is used to determine the 3D editing area based on 2D rendering results and well-designed prompts.
3. Compared to previous work, a source prompt and the the delta denoising score are introduced, with modifications tailored to the editing task.
limitations: limitations: 1. Using multimodal LLMs and prompt engineering to analyze images is not new. Visual ChatGPT, EmbodiedGPT [4], and VisionLLM [5] have already excelled in this task. What is your innovation?
2. There is a lack of detailed explanation and experiment regarding the source prompt. How is the source prompt generated? How do different forms of source prompts affect editing results? It seems the purpose of introducing the source prompt is to justify the Orthogonal Denoising Score (ODS) in Section 4.3. What is the performance of your method if removing the the source prompt and ODS ?
3. Line 504 lacks a detailed explanation of the spherical coordinate system.
4. The comparative experiments are incomplete. I noticed that this paper cited FocalDreamer. Adding a comparison with FocalDreamer would enhance the experiment's credibility. Since FocalDreamer is not yet open-source, you could select a common 3D avatar (e.g., Flash Gordon or Turtle) from its project page, apply the same modifications, and present the results.
5. The user experiment lacks specific settings, such as the number of participants and gender ratio. Additionally, user experiments should allow participants to change viewpoints randomly to experience the 3D model, rather than just providing a video.


### References
[1] Wu, Chenfei, et al. "Visual chatgpt: Talking, drawing and editing with visual foundation models." *arXiv preprint arXiv:2303.04671* (2023).

[2] FocalDreamer: Text-Driven 3D Editing via Focal-Fusion Assembly, AAAI 2024

[3] Delta Denoising Score, ICCV 2023

[4] EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought, NIPS 2023

[5] VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks, NIPS 2023
paper_summary: paper_summary: The paper introduces a MLLM-guided framework for text-driven 3D avatar editing. The framework consists of:
1. A novel MLLM-guided pipeline for deciding the reasonable region to edit.
2. A geometry-appearance decoupled editing strategy with global-local view synergy, which improves the local results with additional local renderings and generated prompts.
3. An orthogonal denoising score that decomposes the editing directions to preserve the information in the original source inputs.
Comprehensive experiments showcase the effectiveness of the framework for local avatar editing, achieving high-quality local editing results in either geometry or texture details with cross-view consistency.
strength_weakness: strength_weakness: 1. The research problem of text-to-3D editing is challenging due to the ambiguity in language and the complexity of 3D space. This work would provide some insights into bridging the gap between these two modalities.
2. The idea of leveraging MLLM to decide the optimal editing regions and guide the content generation is novel and the corresponding method is rather intuitive and effective.
3. The experiments and demos are comprehensive and reader-friendly, displaying the effectiveness of your framework in different situations and the failure cases point out the limitations clearly.
limitations: limitations: 1. About more demos: your work focuses on the "editing" of 3D avatars. The term "editing" generally involves three types of modifications: addition, replacement, and removal. In your demos, you have displayed your framework's capability in adding (e.g., Hulk with a cowboy hat / holding a can of coke) and replacing (e.g., Hulk in denim shorts) the semantics in both geometry and texture details, but not result regarding removing is displayed. I would be more convinced if more related demos (e.g., A DSLR photo of Albus Dumbledore with no hat) were provided.
2. About artifacts and failure cases: you have demonstrated some cases of your frameworks where the failures are rather obvious and ridiculous. Since the focus of your work is trying to leverage the capability of MLLM to the extent feasible, I believe this type of failure could be somehow mitigated by the powerful visual reasoning ability of MLLM. Yet you have not explored this field in your work.
3. About the editable regions: the editable regions generated in your framework is a 3D bounding box, which still seems not fine-grained enough in certain situations, for example, "A DSLR photo of Albus Dumbledore with a long magic wand (or walking stick)". A bounding box covering a long stick might take up a very large volume (even though the stick only takes up a small portion of room), resulting in the malfunction of the editing region mining pipeline. More inspection into this problem would make your work more comprehensive.
4. Typo and grammar check: in lines 158-159, "To coping with..." is more appropriate with "In coping with..." or "To cope with...".
paper_summary: paper_summary: The paper introduces GG-Editor, a framework designed for locally editing 3D avatars, guided by multimodal large language models like GPT-4V. It allows precise edits to specific parts of the avatar using text prompts while leaving other parts unchanged. This framework can manage geometry deformations and maintain texture details. It identifies appropriate editing regions through a coarse-to-fine process and utilizes a global-local synergy editing strategy combined with an orthogonal denoising score  loss to improve editing quality. Experiments demonstrate that GG-Editor achieves realistic and high-fidelity local edits with only text prompts.
strength_weakness: strength_weakness: 1. A framework called GG-Editor is proposed, which utilizes multimodal large language models to guide the local editing of 3D avatars.
2. GG-Editor can edit specific local areas of a 3D avatar based on text prompts while keeping the rest of the areas unchanged.
3. By employing a global-local synergy editing strategy, GG-Editor can enhance the geometric and textural details of the local editing areas while maintaining overall consistency.
4. An orthogonal denoising score loss is introduced, which aids in decomposing editing directions during the editing process and explicitly controlling the preservation of the original content.
limitations: limitations: 1.	On line 440 of the paper, it states, "if the mined region does not meet the user's specific requirements, we can also interactively chat with GPT to further rectify the box to determine a better editing region." It seems that the process of GPT-Guided Candidate Editing Region Mining still requires human intervention. I feel that this method is somewhat over-engineered, and it might be simpler to manually specify the position of the 3D bounding box.
2.	The Orthogonal Denoising Score is an important innovation of the paper, but there are too few experiments regarding ODS in the paper. More detailed experiments are needed to illustrate the properties of ODS.
3.	The experiments are not comprehensive, and the variety of editing examples is limited.
4.	The paper extensively describes the use of prompt engineering to explore the potential of GPT. However, this approach may make it difficult to apply the method directly to a wide variety of different 3D avatars without human intervention.