CTRL&SHIFT: High-quality Geometry-Aware Object Manipulation in Visual Generation
Authors: Penghui Ruan
Bojia Zi
Xianbiao Qi
Youze Huang
Rong Xiao
Pichao WANG
Jiannong Cao
Yuhui Shi
Venue: ICLR 2026 Poster (2026)
Keywords: Diffusion Model
Image Editing
Video Editing
Primary area: generative models

Abstract: Object-level manipulation—relocating or reorienting objects in images or videos while preserving scene realism—is central to film post-production, AR, and creative editing. Yet existing methods struggle to jointly achieve three core goals: background preservation, geometric consistency under viewpoint shifts, and user-controllable transformations. Geometry-based approaches offer precise control but require explicit 3D reconstruction and generalize poorly; diffusion-based methods generalize better but lack fine-grained geometric control. We present **Ctrl&Shift**, an end-to-end diffusion framework to achieve geometry-consistent object manipulation without explicit 3D representations. Our key insight is to decompose manipulation into two stages—object removal and reference-guided inpainting under explicit camera pose control—and encode both within a unified diffusion process. To enable precise, disentangled control, we design a multi-task, multi-stage training strategy that separates background, identity, and pose signals across tasks. To improve generalization, we introduce a scalable real-world dataset construction pipeline that generates paired image and video samples with estimated relative camera poses. Extensive experiments demonstrate that **Ctrl&Shift** achieves state-of-the-art results in fidelity, viewpoint consistency, and controllability. *To our knowledge, this is the first framework to unify fine-grained geometric control and real-world generalization for object manipulation—without relying on any explicit 3D modeling.*

Published as a conference paper at ICLR 2026
CTRL&SHIFT: SHIFT: HIGH-QUALITY GEOMETRY-AWARE OВJЕCT MANIPULATION IN VISUAL GENERATION
Penghui Ruan" 1.2
Bojia Zi
Xianbiao Qi
Youze Huang
Rong Xiao
Pichao Wang &
Jiannong Cao
Yuhui Shi
'Te Hong Kong Polytechnic University
Southern University of Science and Technology
The Chinese University of Hong Kong
*IntelliFusion Inc.
University Electronic Science and Technology of China
NVIDIA
Source Image
Edited
Source Image
Edited Result
(a) Object Manipulation
(b) Object Removal
(c) Reference Image Inpainting with Comera Control
Figure 1: Results of Cirt&Shift. Strl&Shift demonstrates its superior capability (controllability. plausibility and consistency) on tasks including (a) precise object manipulation, (b) visual object removal, and (c) reference image inpainting with precise camera pose control.
Equal Contribution Project Lead
This work is not related to the author's position NVIDIA.
8 Corresponding Authors

---

Published as a conference paper at ICLR 2026
ABSTRACT
Object-level manipulation-rrllcciinnn or reorienting objects in images or videos while preserving scene realism-is central to film post-production, AR, and cre- ative editing. Yet existing methods struggle to jointly achieve three core goals: background preservation, geometric consistency under viewpoint shifts, and user- controllable transformations. Geometry-base approaches offer precise control but require explicit 3D reconstruction and generalize poorly; diffuusio-bbsed methods generalize better but lack fine-grained geometric control. We present Ctrl&Shift, end-to-end diffusion framework achieve geometry-consistent object manipulation without explicit 3D representations. Our key insight is to decompose manipulation into two otages-object removal and reference-guided inpainting under explicit camera pose control-and encode both within a unified diffusion process. To enable precise, disentangled control, we design a multi- task, multi-stage training strategy that separates background, identity, and pose signals across tasks. To improve generalization, we introduce a scalable real- world dataset construction pipeline that generates paired image and video samples with estimated relative camera poses. Extensive experiments demonstrate that Ctrl&Shift achieves state-of of- ti--the-ade art results in fidelity, viewpoint consistency, and controllability. To our knowledge, this is the first framework to unify fine-grained geometric control and real-world generalization for object vanipulation-withoon relying on any explicit 3D modeling.
INTRODUCTION
Object-level manipulation such as relocating or rotating an object within a single image or video while preserving the surrounding scene-is is a fundamental primitive in film post-production, aug- mented reality (AR), and creative visual editing. For instance, adjusting the placement of prop in movie frame or changing the camera angle of product in an AR preview both demand geometry- aware edits that remain photorealistic under viewpoint changes. Failing to preserve geometric con- sistency leads warped objects, unnatural shadows, background urtifacts-breakingg immersion and rendering the edit unusable in professional workflows.
Despite recent progress in visual editing, controllable and consistent object manipulation remains a persistent challenge. At the heart of this difficulty lies fundamental trade-off: geometry-based methods offer precise control but poor generalization, while diffusion-based methods gener-
alize well but lack fine-grained control over geometry. Geometry -basd approaches (Mildenhall
al., 2020 Kerbl et al. 20223] Michel et al. 2023) Chen et al., 2025 Yenphraphai al.] (024) rely on multi-view optimization or explicit 3D reconstruction to maintain consistency across views. However, these techniques often require synthetic data or per-scene optimization, limiting scalabil- ity and realism. On the other hand, diffusion- based editors (Zhang & Agrawala, 2023)] Wu et al.)
2024 Jiang et al.) 2025)) have demonstrated remarkable generalization to in-the-wild content, en- abling free-form edits via text prompts or trajectories (Choi et al. 2023). Yet, these models struggle to deliver precise control over object pose, making it difficult to specify fine-grained geometric transformations.
This paper introduces Ctrl&Shift, a new framework that breaks this trade-off: it enables geometry-consistent, fine-grained object manipulation with strong generalization real-world con- tent-without requiring explicit 3D reconstruction at inference. This marks a conceptual shift: rather than lifting content into 3D space to perform edits, we inject precise viewpoint control di- rectly into 2D diffusion process. Our core insight is to decompose object manipulation into two too-ttasssohjere removal and snfference-uuider inpainting under explicit camera pose-which can be jointly modeled within a unified diffusion framework. Disentangled Control via Multi- Task Training. The central challenge is enabling the model to respond coherently to multiple condi- tioning signals: background context, object identity, spatial masks, and relative viewpoint changes. To address this, we design a unified conditioning interface and adopt a multi-task training scheme aligned with the natural task decomposition: one task teaches object removal, another reference- conditioned inpainting with camera control, and a third combines both for full manipulation. This structure ensures each signal plays clear, interpretable role during training and inference. From
2

---

Published as a conference paper at ICLR 2026
Synthetic Pretraining to Real-World Generalization. While multi-task learning helps disentangle functional roles, achieving realistic geometry- awwetr control in natural imagery requires high-quality, pose-annotated supervision. Synthetic datasets often fall short in realism and diversity, and exist- ing real-world data rarely includes accurate camera annotations. To close this gap, we develop a scalable pipeline for constructing paired image and video samples from real content, enriched with estimated relative camera poses. Our method leverages 3D object reconstruction, differentiable camera pose estimation, and harmonized rendering synthesize target views under novel camera configurations-enallee
large-scale training on realistic data with geometric supervision.
Contributions. This work introduces new approach controllable object manipulation that fuses the structural precision of geometry- pipelines with the scalability of generative diffusion, all within a unified 2D framework. Our contributions are:
Conceptual Innovation. We propose Ctrl&Shift, an end-to-end framework to achieve geometry- consistent object manipulation without requiring explicit 3D representations inference, by in- jecting relative camera pose control directly into the diffusion process.
Architectural Design. We disentangle background, object identity, spatial masks, and geomet- ric transformations through a multi-task, multi-stage training strategy that mirrors the semantic structure of object manipulation.
Data Construction Pipeline. We introduce scalable method to create real-world paired su- pervision with accurate camera control, by reconstructing object meshes, estimating poses via differentiable rendering, and harmonizing novel views through learned object pasting model.
Systematic Benchmarking, We evaluate Ctrl&Shift across multiple datasets and introduce GeoEditBench, new benchmark for geometry-aware editing. Our model consistently achieves state-of-the the-thr-art results in fidelity, controllability, and viewpoint consistency. Moreover, by avoid- ing 3D modeling at inference while retaining precise geometric control, our approach bridges reometry-based rigor with diffusion flexibility, enabling scalable, controllable editing in the wild.
1.1 RELATED WORK
Diffusion-based Object Editing. Diffusion models have enabled powerful visual editing capabili- ties (Rombach et al. 2022 Saharia et al. 2022; Wu et al. 2025 Zi ct al. 2025b). Text-conditioned approaches offer flexibility but lack precise control over spatial transformations, often leading to ambiguous object manipulation. Trajectory-
methods convert motion signals into spatial con- ditioning via architectures like ControlNet (Zhang & Agrawala, 2023 Jiang et al.) 2025 Wu al., 2024). For instance, Drag Anything (Wu et 2024)) enables interactive point-based manipulation, while FateZero (Qi al. 2023) and Tune-A-Vidde (Wu et al. 2023) extend editing to the tempo- ral domain. Object insertion methods like ObjectAdd (Zhang et al. 2024) and In (Saini et al.) 2024] modify diffusion processes for adding objects without retraining. However, without explicit 3D representations, these methods struggle to provide geometric control, particularly for viewpoint changes, resulting in object identity drift, background corruption, and inconsistent camera.
Georetry-aware Object Editing. Complementary approaches leverage 3D supervision to main- tain multi-view consistency. Single-image 3D lifting methods like Zero-1-to-3 (Liu al. 2023) and SyncDreamer (Liu et al. 2024) synthesize novel views but require harmonization for real scenes. Dataset-driven methods using synthetic data face domain gap challenges (Reizenstein al.)
2021 Deitke et al., 2022). Recent works integrate 3D information into diffusion: GeoDiffusee uses gcometry-based conditioning for precise edits (Sajnani et al., 2025), Diffusion Handles enables handle-based 3D-aware manipulation (Pandey et al. (023). and OBJect 3DIT provides language- guided multi-view consistent editing Miccel et al. (2023). ObjectMover leverages video priors for object movement, while BlenderFusion uses 31D engines for manipulation (Yu et al.) (0255] Chen ct al. 2025). However, these methods typically require explicit 3D reconstruction and manipulation.
2
METHOD
The primary goal of object manipulation is to shift and rotate the object according to the relative pose signals while maintaining visual coherence. In our framework, we formulate this as an image/ video editing task aask-specffically, given: the source video frames S E RTx3xH xW which provide the
3

---

Published as a conference paper at ICLR 2026
Stoge 1: Object Prier
Input Frames
Reference Imegge
Source Hosk
Estinoted Targst
General Comara Control
Main Branch
Contral
Main Tosk: Object Honipelation
Branch
Torget
Source
Stage Backgrourd Preservation
Night qualty Fine mne-lanng
Infirty
Objest Remava!
Torget
Searce
Control
Branch
Brarch
Awilary Task
Iepainting
Figure 2: The proposed Ctrl&Shift framework employs a multi-task, multi-stage training paradigm integrating object manipulation, removal, and reference inpainting with explicit camera control. Stage focuses on acquiring object priors and camera control; Stage 2 emphasizes background preservation through fine-tuning on high-quality data.
background context; the reference object image 1et E R (extracted from the first source frame), which disambiguates the object identity; the source mask M {0, 1xxxxxxx which delineates the region to be removed; the estimated target mask M . E (0.1)xx|xHxM (as detailed in Section 2.1 which specifies the coarse region for object placement; the relative camera-pont pose descriptor E R (as defined Section 2.2). which encodes the geometric transformation be- tween the source and target viewpoints. Our model translates and rotates the object to the target location/viewpoint and predicts target frames tgt IR T x3xxXXX
Ctrl&shiff adopts ControlNet-stylee DiT chitecture with hidden size dmodal = 1536. The inputs are processed into unified latent space: source frames ~ and reference Ire are en- coded via EVAE, while masks M and M are downsampled via pixel unshuffle gl.. In
Greanshruth
Pelativ
Anfiveeon Jwaye Edletted Torge Hoyk
Eincoder
Preve Concet
Trave Coeatt
Chanet Conoot
the control branch, these features concate- nated channel-wise, temporally replicating rel match the video frames, then projected and injected into the DiT blocks via zero-initialized convolutions. The main backbone operates on noised latents derived from the concate- nated ground-truth video d and reference Iref To enforce geometric control, the relative- pose descriptor f is mapped by a 3-layer MLP Ecam to embeddings that are injected via cross- attention, guiding the viewpoint while preserv- ing identity and background context.
Boock
Control Block
Block
Control Block
DI Blook
Control Block
Figure 3: Overview of the architecture.
2.1 MASK ENCODING
We cast the object manipulation as object removal and reference image inpainting tasks. During training we provide source mask M and estimated target mask M As the true target mask unknown during inference, obtain e from M by: (i) squaring/tightening M via its ne bounding box; (ii) scaling by the distance ratio de to approximate apparent size, (iii) shifting on the image plane by (Arz, Ary); and (iv) truncating outside the frame.
Mask Semanties and Representation. In setting, value of indicates region be edited/painted and indicates a region to be preserved; these semantics differ from image inten- sities (e.g., white vs. black color). To preserve this binary meaning and avoid treating masks as

---

Published as a conference paper at ICLR 2026
appearance, do encode masks with the VAE. Instead, map masks to the VAE latent grid using a space-to-depth (pixel-unsie) unshuff)) operation that exactly matches the VAE stride. Coneretely, for spatial stride and temporal stride t, transform
(M E [0,1] xtx _
M {0, 10.1IXXXXXX
which reduces resolution while increasing channels. This procedure preserves the mask's binary structure, avoids semantic confusion with RGB values, and provides high-fidelity. stride-aligned guidance to the model.
2.2 CAMERA POSE ENCODING
In this subsection, we will introduce how we obtain the relative pose descriptor We position the object at the world origin and use a look-at camera oriented toward the origin with a fixed world-up vector u. = (0, 0) Each view is parameterized by
s (yaw, pitch, d, Tz: rz)
where yaw E (- IT] is the azimuth about the +Y-axit (zero along +X, increasing toward + +2), pitch [-m/2, TT is the elevation above the XZZ-llane, > 0 the camera-origin distance, and (NDC) shifts applied post-projection, with (0,0) at the image center. For source and target views see and gtit, compute world-to-camert extrinsics via
(Fx, ryy E [- are
RRyre: tare) LOOKAT (yaw pitch dr uw):
(Rigt, tegt) LOOKAT (yaw't pitch sgt dftt uw),
where LOOKAT positions the camera (d (osith))))) cos(yaw), d din((pitch) (pitch) sin(yaw)) oriented toward the origin with up-vector uws yielding (R, such that x, Rx t. The relative
transform matrix and the translation vector from source-camera to target-camera coordinates are individually calculated as
Riel Regt Rare:
tyll
figt
where Xegs
Rare + trel- The relative rotation is encoded via its axis-angle representation
(a(R)ol) - vee llog Ryel) E IR3
where log Riel s0(3) is the matrix logarithm and vee s0(3) - R3 extracts the vector form. The relative NDC shifts are computed as
Ar r
Ary = y
sTC
The relative-pose descriptor
(Rr)) trel Arzi Arg) R
Fach component of undergoes Fourier positional encoding, followed by lightweight MIPs, to create tokens (d = 4096). Note that we opt to encode the relative camera pose rather than absolute poses, because defining canonical absolute pose is challenging across diverse objects. Moreover, during inference, specifying the target pose without a clear baseline even more difficult, potentially leading to inconsistent and unintuitive controls. In contrast, relative encoding leverages the camera pose from the input frames as an implicit base, requiring only intuitive, drag-like adjustments.
2.3 MULTI-TASK MULTI-STAGE TRAINING
Ctrl&Shift employs five distinct conditioning signals, integrated through tailored mechanisms: video-like signals are incorporated via ControlNet, while camera control is facilitated through cross- attention mechanisms. Each conditioning signal serves a unique role the manipulation task. To enhance the model's ability disambiguate and leverage these signals effectively, we introduce multi-task training strategy that isolates the functional contributions each signal. Our tasks are defined as follows:

---

Published as a conference paper at ICLR 2026
Dotaset Construction Pipefine
Cana Sace Canera
Olyect Halt
Saare Image/kion
Olject Removr
backgroet Lnege//btt
Rendering
Differentiable Rasterization
Estimation
Object Rendering Posting
Tol
Object Fosting Modil
Torgrt Twager Mldes
Figure 4: Construction of data pairs (X Sth and (X sttt). From X an image- to-ee-tt-msshh model reconstructs the object mesh, and s is estimated via differentiable rasterization. The target pose stat sampled, the object rendered using the mesh, and an object pasting model generates X Our pipeline supports both image and video data synthesis, as the object pasting model is reference- image inpainting model capable of editing both image and video. For video inputs, the image-to- mesh reconstruction, camera pose estimation, and rendering are all performed on the first frame.
Main Task: Object Manipulation. The primary task of object manipulation involves relocating the foreground object from its source position and viewpoint specified target position and viewpoint, while preserving both the object's identity and the background context. Specially, let &VAR be pretrained VAE encoder, and let Ecam denotes the camera pose encoder. We define the conditioning tuple explicitly via the encoding operators:
Cmain (Xxx) (VAR(T"), IM(M")) I, Ecam(f)):
X
igt main
X
where EAAE denotes the VAE encoder mapping images to latent space, II, the space-to-depte oper- ation aligning masks with the VAE latent grid, and &cam is the camera pose encoder. The supervision is provided by the ground-truth target frames X
Auxiliary Task 1: Object Removal. This auxiliary task focuses removing the foreground object from the source frames while reconstructing a coherent background. To achieve this within our unified conditioning interface, we set the reference image a pure white image (indicating no object identity), the estimated target mask to all zeros (indicating no region to inpaint), and the relative camera pose such that the object is positioned outside the image frame, resulting in M = 0F The conditioning vector is
(E"r) E1AE(1), (M, (O), Ecam Eam(())
X
Caux
where denotes white reference image, O an all-zero mask, and b the background frames, encouraging the model to inpaint the background cleanly.
This auxiliary task aims to synthe- size the reference object onto a background under precise camera control, without requiring removal of any region from the input. Within the unified conditioning interface, all zeros (indicating no region to remove) and use the background frames as input. The reference image me is extracted from the first frame of the source input, as in the main task, and the target mask and camera pose guide the object's placement and viewpoint. The conditioning vector is
Auxiliary Task 2: Reference Inpainting with Camera Control.
set the source mask to
Cauxg -EVAE(XX*), EvAE (ITE( (0), TT, (M('))) Ecam(f)):
X
As illustrated on the left side of Figure 2 our model complements the multi-task training framework with two-stage training strategy designed systematically distribute knowledge across its archi- tecture. This regimen ensures that foundational geometric and pose-related priors are established early, while subsequent refinement focuses on real-world fidelity, thereby balancing generalization and photorealism.
This is achieved hy setting the normalized device coordinate (NDC) shifts in f beyond the range [-1, 1), ensuring the projected region exits the frame.

---

Published as a conference paper at ICLR 2026
Stage I: Object Prior & Pose Learning. In Stage I, we pretrain the model on large-scale synthetic dataset comprising 3D object meshes rendered under randomized camera poses (including variations in yaw, pitch, distance, and sereen-space shifts) against uniform white backgrounds. This controlled environment isolates object appearance from complex scene interactions, allowing the model to focus on learning intrinsic object properties. By jointly optimizing both the main branch and the conditioning control branch, this stage instills category-agnostic object priors and robust camera pose representations, ensuring generalization across diverse objects and viewpoints.
Stage I:: Background Preservation & High-Quality Fine-Tuning. In Stage II, we fine-tune the model on curated dataset of high-quality images and videos featuring complex scenes and back- grounds. This stage addresses the limitations of synthetic data by incorporating real-worke data, thereby enhancing generalization to natural, intricate environments. To maintain the object priors and pose understanding acquired in Stage I, we freeze the main branch and update only the condi- tioning control branch, focusing on improved background coherence, and photorealistic rendering.
Ctrl&Shift is trained with flow-matching (Lipman et al. 2023). Specifically, let = EVA X ( denotes the target latent We sample t ~ (1, 1) and E ~ N(O.I) with the same shape as Zo. and define the linear path and target velocity:
2, (1 t) 2o + te, v*(z,t t)
Zg-
A velocity network vo(2,, c, 1) is trained with the flow-matching loss (Lipman et al. 2023
(0) e) Be-uO..). (ooo v² v*(2,
2.4 DATASET CONSTRUCTION
we construct paired examples (X ser) and (X'st, st8t). The source pair a real photograph; the target pair is synthesized by rendering the same object under a novel camera and harmonizing it with the source background.
As illustrated in Figure 4
Object mesh reconstruction. Given a source image Iare (extract from first frame of X and its object mask M we reconstruct watertight, textured mesh M of the foreground object using Hunyuan3D-2 (Zhao et al. 2025).
Source camera estimation. We recover sre by aligning M to the observed silhouette via differen- tiable rendering. Let R(M, s) denote a renderer producing a soft silhouette. We solve
S
srC
arg max loU (ROLOUURRRRMMLA (R(M, s), M
implemented with gradient-based -based optimization (Chnn et al. 2021 Jatavallabhula et al. 2019 Laine
al. 2020), and retain only instances with IoU 2 0.90. This yields reliable source-camers poses.
Target camera sampling and object rendering. We obtain a novel target view stgt by sam- pling (yaw, pitch, d, Ta, Ty) as moderate perturbations of ser We then render the object O RRGE(M, (*t, producing a view-consistent foreground under the desired camera.
Background acquisition and harmonization (object pasting). We first remove the source object with MiniMax-Removee (Zi et al.) (202a) to obtain clean background plate B. To seamlessly compose 'gt into the scene, we train an object-pasting (harmonization) network H (See Appendix. A.3) in self-reconstruuction regime: given (B, Mrre O where O REGE(M, ( the network is supervised to predict x This teaches H. to color-match, re- light, and blend object boundaries. During inference, we provide (B, Mtst O!st) to obtain x ggt = (B, M Otst).
3 EXPERIMENTS
Our model extends the Wan-1.3B backbone Wan et al., 2025) with eight control blocks in the
conditioning branch. Training uses spatial resolution of 640 x 960, with video inputs at 61 x 640 960. In Stage we pretrain for 50k steps using AdamW (learning rate 5 x 10 10-5) and a One-Cycle scheduler on synthetic dataset of ~2M image pairs, generated by rendering 3D object
7

---

Published as a conference paper at ICLR 2026
meshes with randomized camera poses. In Stage II, we fine-tune for 5k steps curated dataset of 100K high-quality image and video pairs. For multi-task training, we balance the main task, auxiliary task 1, and auxiliary task 2 with weighting of 8:1:1. See Appendix
A.2 for details.
Quantitative Evaluation on ObjectMover-A. As shown in Table ! we conduct a zero-shot eval- uation on the ObjectMover-A benchmark (Yu et al. 2025). Following Yu et al. (2025). we assess frame-level performance by calculating the PSNR between the target frame and the edited frame. For object identity preservation, we crop out the object and compute the DINO score (Caron al.) 2021), CLIP similarity (Radford et al. 202T), D, and DreamSim (Fu et al. 2023). As shown in Ta- ble our method outperforms existing approaches by a large margin. demonstrating its superior performance on object translation.
Quantitative Evaluation on GeoEditBench. To evaluate our approach for object manip- ulation in real-world images, we developed GeoBditBench, new dataset with 346 care- fully curated image pairs captured by skilled photographers under controlled settings (See Appendix A.5 for details). We report PSNR, DINO score, CLIP score, and DreamSim met- rics to gauge background preservation and ob- ject identity fidelity. To assess camera pose accuracy, we estimated poses from edited out-
Table 1: Zero-shot Evaluation ObjectMover-A
Method
PSNR 1
DINO
CLIP
DreamSSm J
Drag Anyching
16.36
55.56
84.44
0.411
19.72
45,30
0.514
Paint-by-Example
20.83
55,46
85.23
0.420
Anydoor
21.86
69.32
88.95
0.289
MagicFixup
23.82
78.49
91.06
0.198
ObjectMover
25.27
85.07
93.16
0.142
Ours
28.69
88.07
93,58
0,075
puts and calculated (i) the mean absolute percentage error (MAPE) across pose parameters (yaw, pitch, d, Tx, ry). and (ii) the IoU between the edited object's silhouette and the ground-truth mask. Lower MAPE and higher IoU reflect precise camera control. We compare our method against several baselines. including Drag- Anythingg (Wu et al. 2024). 3DiT (Michel et al. 2023).
VACE (Jiang et al. 2025), Flux-Kontext (Labs et al.j 2025). Qwen-Image-Fidt (Wu et al. 2025
and Mano-Banana (noa) (implementation details in Appendix A.4). as presented in Table 2 Among these, Anything VACE, and Kano-Banand utilize spatial signals (c.g., trajectories, bounding boxes, or target masks) as input, enabling precise object placement and resulting in relatively high DINO, CLIP, and DreamSim scores, as well as improved object IoU. However, these methods ex- hibit higher pose MAPE, reflecting limited control solely on language and coordinate inputs, such as 3DiT and Flux- Kontext, struggle to achieve com- parable spatial accuracy. Our approach achieves superior PSNR, indicating excellent background preservation, alongside high DINO, CLIP, and DreamSim scores, demonstrating robust retention of object identity. Furthermore, attains the lowest pose MAPE and highest object IoU, underscoring its precision in both camera control and object manipulation. Despite these advancements, the re- sults highlight that precise object manipulation real-world images remains a complex challenge, necessitating further research to enhance performance and robustness across diverse scenarios.
object pose. In contrast, methods relying
Qualitative Evaluation. Figure presents a qualitative comparison of our method against state-of- the-art approaches, including VACE (Jiang et al. 2025). Mano-Banany (noa). Flux-Kontext (Labs) et al. 2025), 5), Qwen-Image-Fil
(Wu et al. 2025), DragAnything (Wu et al. al. 2024), and 3DTT (Michel et al. 2023). Draghny and 3DTT exhibit limited generalization to real-world data, constrained by their reliance on specific training datasets. VACE effectively preserves background consistency but fails to directly manipulate the object, instead shifting the entire frame to indirectly place the object at the target location. Nano-Banana and Qwen-Image- Edit produce high-quality images with consistent backgrounds; however, they struggle to accurately incorporate camera pose changes spec-
Table 2: Zero-Shot Evaluation on GeoEditBench
Method
Translation
Rotation
PSNR
DINO-Score f CLIP-Score 1
DreamSim L
Pose MAPE 1
Obj ToU f
Drag Anything
17.65
57.24
70.85
0.205
46.81%:
0.56
3DDT
20.56
39.16
57.76
0.280
51.62%
0.39
75.38
Flux- kontext
21.57
57.97
68.35
0,229
46,76%
0.47
Qwen-Inage-Edde
22.72
61.62
79.77
0.221
39.56%
0.52
Nano-Banana
26.38
78.05
85.63
0.145
24.36%
0.78
Diffusion Handles
24.18
72.23
82.37
0.182
36.36%
0.67
Ours
28.71
85.23
90.44
0.112
17.70%
0.83
8

---

Published as a conference paper at ICLR 2026
Source Image
3D/7
Dragheyching
WICE
Nano Banara
Quen-Image
Foo--oreestt
Figure 5: Qualitative comparisons for object manipulation. displaying relative camera changes and NDC shifts. Our model outperforms state-of-the-arr methods in background preservation, precise camera pose control, and geometric consistency.
ified via text instructions. In contrast, our method achieves precise camera pose control, enabling accurate object relocation and viewpoint adjustments while maintaining background preservation.
Ablation Study: We ablate different stage and task to show the effect of each part of our methods, as shown in 3 Ablation studies on GeoEditBench highlight the contributions of each component in our computer vision pipeline: removing Stage I significantly impairs geometric fidelity, with Pose MAPE increase to 32.50% and Obj IoU declining to 0.68, underscoring its role in camera pose un- derstanding. Excluding Stage 2 diminishes reconstruction quality, evidenced by PSNR dropping to 24.83 and DreamSim rising to 0.195, indicating its importance for background preservation and visual quality. Omitting Aux leads moderate losses in semantic consistency, with Ccor-Score

---

Published as a conference paper at ICLR 2026
falling to 86.32 and DINO-Score 80.47 alongside Pose MAPE increase to 23.80%, suggesting bolsters feature aligument and mid-level visual representations. Dropping Aux 2 most adversely affects object-centric metrics, reducing Obj IoU to 0.65 and elevating Pose MAPE to 28.60% while lowering CLIP-Score to 83.75, revealing its focus on fine-grained supervision for instance-level de- tection and pose tracking. Overall, the full model achieves superior visual coherence and metric balance, validating the synergistic integration of our multi-task multi-stage training paradigm.
Table 3: Ablation Study on GeoEditBennh
Stage
Stage 2
Aux
Aux
Main
PSNR f
DINO-Score 1 CLIP-Score DreamSim
Pose MAPE 1 Obj JoU
v
25.12
78.64
85.21
0,178
32.50%
0.68
x
24.83
79.15
84.67
0.195
20.40%
0,77
4
25.96
80.47
86.32
0.162
23,80%
0.74
4
x 26.54
77.89
83.75
0.149
28.60%
0.65
v
28.71
85.23
90.44
0.112
17.70%
0.81
4 LIMITATIONS AND FUTURE WORK
While our framework achieves precise geometry-aware object manipulation, we acknowledge sev- eral limitations stemming from both the data construction pipeline and the inherent complexity of the task (See Appendix A.8 for details). We discuss these challenges and future directions below:
From Technical Controllability to Practical Usability. While our 8D relative pose descriptor f provides robust geometric control, we recognize that manually specifying these values not user- friendly. A key future challenge is mapping intuitive user interactions, such as 2D mouse drags, rotational gestures, or 3D gizmo manipulation. directly to this descriptor. Since f represents a rigid transformation, it can be analytically derived from such inputs, bridging the gap between precise internal representation and an accessible user experience.
Physical Realism Beyond Geometry. Our framework prioritizes geometric consistency but does not explicitly model physical interactions like lighting, variable shadows, or specular reflec- tions. Instead, we rely on the model to implicitly learn these photometric effects from training data. While often effective, this data-driven approach struggles with physically correct shadows or reflections under drastically different illumination. Integrating explicit lighting estimation or mhysics-based rendering guidance is valuable direction to enhance realism.
Generalization Boundaries via the Data Pipeline. Our method's reliance on 3D mesh recon- struction imposes generalization boundaries: Object Type: The pipeline assumes rigid geometry, limiting applicability to non-rigid objects (e.g., cloth, hair) or topological changes (e.g., smoke, fluids). Material Properties: Transparent highly reflective objects (e.g.. glass, mirrors) are of- ten reconstructed with "baked-in" background textures, causing artifacts during synthesis. Com- plex Occlusions: Our "remove-and-inge inppditnpann strategy assumes the target area is visible or planar and cannot handle complex depth relationships, such moving object behind another scene element, which requires reasoning about occluding background geometry.
Limited Video Manipulation Capabilities: While our method achieves precise geometric con- trol, video manipulation shows lower visual fidelity than image editing. This arises from our rigid 3D reconstruction pipeline's 's inability to capture non-rigid deformations and complex spatiotem- poral dynamics in real-world videos. Future work will leverage 4D representations to improve video manipulation quality.
CONCLUSION
In this work, we introduce Ctrl&Shift, novel inffuusionbbseed framework for precise object-level manipulation in images and videos, enabling relocation and viewpoint control while preserving scene integrity through task decomposition into removal and reference-guided inpainting, multi- task multi-stage training for signal disentanglement, and a curated dataset. Extensive evaluations demonstrate its superior fidelity, temporal coherence. and controllability across real-world scenarios, positioning it as a versatile tool for visual content creation, augmented reality, and film production, with future extensions targeting dynamic scenes and multi-object interactions.
10

---

Published as a conference paper at ICLR 2026
6 ACKNOWLEDGMENTS
This work is partially supported by the Hong Kong General Research Fund under Grant PolyU 15205924 and the National Science Foundation of China under Grants 62250710682 and 61761136008 We also acknowledge the support from the Research Institute for Artificial Intelli- gence of Things at The Hong Kong Polytechnic University.
REFERENCES
Gemini 2.5 Flash Image (Nano Banana) Google Al Studio.
URL https://aistudi::
google. com/ wwtmmoomom/////ctc/oooocc.c..cse mwdeem/ddssssoooeemm...moomo / gemini 2-5 fass-llss-aaal image
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and
Armand Joulin. Emerging Properties in Seff-Superviee Vision Transformers, May 2021. URL
http://arxiv..ott org.ivi.rrraare abs abs//104 hwtp://gg///bbaareoe
14294 1.Ki::200411224 [es].
Jiacheng Chen, Ramin Mehran, Xuhui Jia, Saining Xie, and Sanghyun Woo. BlenderFusion: 3D- Grounded Visual Editing and Generative Compositing, June 2025. URL http::/arxiv.
org/abs/2500 2506. 17450 MrXiv:2506.17450 [cs].
Wenzheng Chen, Joey Litalien, Jun Gao, Zian Wang, Clement Fuji Tsang, Sameh Khamis, Or Litany, and Sanja Fidler. DIB-R++: Learning to Predict Lighting and Material with a Hy- brid Differentiable Renderer, October 2021. URL /RRL:tt////rxxi.oorr http://awwwvv.o//////111...0. arXiv:211 1.000110 [cs].
Jooyoung Choi, Yunjey Choi, Yunji Kim, Junho Kim, and Sungroh Yoon. Custom-Edit Text- Guided Image Editing with Customized Diffusion Models, May 2023. URL, http::/arxiv..
org/ hts//bb/// 2305. 15779 arXiv:2305.15777 [cs].
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Lud- wig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: Universe of Annotated 3D Objects, December 2022. URL arXiv:2212.08051 [cs].
Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data, December 2023. URL [es].
arXiv:2306.09344
Krishna Murthy Jatavallabhula, Edward Smith, Jean- Fran-FFancois Lafleche, Clement Fuji Tsang, Artem Rozantsev, Wenzheng Chen, Tommy Xiang, Rev Lebaredian, and Sanja Fidler. Kaolin: A Py- Torch Library for Accelerating 3D Deep Learning Research, November 2019. URL http:
wlaaxii.rrg/abs////..... 05063
19100063 [cs].
Zeyinzi Jiang, Zhen Han. Chaojie Mao, Jingfeng Zhang, Yulin Pan, and Yu Liu. VACE: All-in-nee Video Creation and Editing, March 2025. URL http:: htap:::///xxv.../aaa http://axxxxv...a//.a// wttp://aa/b/...55555595 arXiv:2503.07598 [es].
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3D Gaussian Splat- ting for Real-Time Radiance Field Rendering, August 2023.
2308. 04079 arXiv:2308.044 .4479 [cs].
Black Forest Labs, Stephen Batifol, Andreas Blattmann, Frederic Boesel, Saksham Consul, Cyril Diagne, Tim Dockhorn, Jack English, Zion English, Patrick Esser, Sumith Kulal, Kyle Lacey, Yam Levi, Cheng Li, Dominik Lorenz, Jonas Müller, Dustin Podell, Robin Rombach, Harry Saini, Axel Sauer, and Luke Smith. FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing Latent Space, June 2025. URL httpp://arxxv..rrg/ab///03000/00 arXiv:2506.1575 [cs].
Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila. Modular Primitives for High-Perfonee Performance Differentiable Rendering. November 2020. URL http:// arxiv.cm wwxivoorg/bb...o /2011 03277 arXiv:201 1.03277 [cs].
11

---

Published as a conference paper at ICLR 2026
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow Matching for Generative Modeling, February 2023. URL httt::/aaxxv..rrg wttp:x/w....//////2220.00002200 /ttp//////////..../////////200.0
arXiv:2210.02747.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-shot One Image to 3D Object, March 2023. URL http:://axiv.orr
11328
arXiv:2303. 1328 [cs].
Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. SyncDremmer: Generating Multiview-consistere Images from a Single-view Image, April 2024.
URL http:://axxiiii org.
03453 urXiv:2309.03453 [cs].
Oscar Michel, Anand Bhattad, Eli VanderBilt, Ranjay Krishna, Aniruddha Kembhavi, and Tanmay Gupta. OBJECT 3DIT: Language-guides guided 3D-aware Image Editing, July 2023. URL http: 7/ //axiv.orgg/abab...e.aao wwtp:////aa///aaa
11073 arXiv:2307.11073 [cs].
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis, August 2020.
URL
wttp/rgg/abb//..////////// 08934 arXiv:2003.0893 [cs].
Karran Pandey, Paul Guerrero, Matheus Gadelha. Yannick Hold Hold-Geoffroy, Karan Singh, and Niloy Mitra. Diffusion Handles: Enabling 3D Edits for Diffiusion Models by Lifting Activations to 3D,
December 2023. URL // wwwpaiiwwwrrr//oooooooooobosssst /231l 021 90 arXiv:2312.02190 [es].
Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. FateZero: Fusing Attentions for Zero-shot Text-based Video Editing, October 2023. URL
hattp://arii.. http://aaaxx...o/a/b/////sss////
09535
arXiv:2303.095: [cs].
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning Transferable Visual Models From Natural Language Supervision, February 2021. URL URL httt::/arxxi org/ abs /2003. C0020 arXiv:2103.00020 [cs].
Jeremy Reizenstein, Roman Shapovalov. Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction, September 2021. httthttp:///axx. http://aaaravoooo///////00000 00512 arXiv:2109.00 .000000..1122 [es].
Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan. Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li. Qing Jiang, and Lei Zhang. Grounded SAM: Assembling Open-Worke Models for Diverse Visual Tasks, January 2024. URL [es].
14159 arXiv:2401.14159
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- Resolution Image Synthesis with Latent Diffusion Models, April 2022. URL http::/arxxi org/ org/abs/ abs/ 2112. 10752 arXiv:2112.10752 [cs].
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang. Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic Text- to-Image Diffusion Models with Deep Language Understanding. May 2022. URL http:
//axxiv.oggaab///// 220p://..///2...../// 110000....887 11487 arXiv:2205.11487 [cs].
Nirat Saini, Navaneeth Bodla, Ashish Shrivastava, Avinash Ravichandran, Xiao Zhang, Abhinav Shrivastava, and Bharat Singh. InVi: Object Insertion In Videos Using Off-the-Shelf Diffusion Models, July 2024. URL httt:tttp///xii// wwwp:/iwworrroooooooooooo / 10958 arX1v:2407.10958 [cs] version: 1.
Rahul Sajnani, Jeroen Vanbaar, Jie Min. Kapil Katyal, and Srinath Sridhar. GeoDiffuser: Geometry- Based Image Editing with Diffusion Models, January 2025. BRLpttp: http://axxvvv./// abs/ 2404. 14403 arXiv:2404. 14403 [cs].
12

---

Published as a conference paper at ICLR 2026
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang. Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng. Shiwei Zhang, Siyang Sun, Tao Fang, Tianxing Wang. Tianyi Gui, Tingyu Weng. Tong Shen, Wei Lin, Wei Wang, Wei Wang. Wenmeng Zhou, Wente Wang, Wenting Shen, Wenyuan Yu, Xianzhong Shi, Xiaoming Huang, Xin Xu, Yan Kou, Yangyu Lv, Yifei Li, Yijing Liu, Yiming Wang, Yingya Zhang, Yitong Huang, Yong Li, You Wu, Yu Liu, Yulin Pan, Yun Zheng, Yuntao Hong, Yupeng Shi, Yutong Feng. Zeyinzi Jiang, Zhen Han, Zhi-Fan Wu, and Ziyu Liu. Wan: Open and Advanced Large-Scale Video Generative h /ttp:///axxvv.e .rg. abs 2503. 20314 4 mnXiv:2503.20314 [cs].
Models, April 2025. URL
Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng-ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, Yuxiang Chen, Zecheng Tang, Zekai Zhang, Zhengyi Wang, An Yang, Bowen Yu. Chen Cheng, Dayiheng Liu, Deqing Li, Hang Zhang. Hao Meng. Hu Wei, Jingyuan Ni, Kai Chen, Kuan Cao, Liang Peng. Lin Qu, Minggang Wu, Peng Wang, Shuting Yu, Tingkun Wen, Wensen Feng, Xiaoxiao Xu, Yi Wang. Yichang Zhang, Yongqiang Zhu, Yujia Wu, Yuxuan Cai, and Zenan Liu. Qwen-Image Technical Report, August 2025. URL http://arxiv. org/ bog//bs//ae / 2508 02324 arXiv:2508.0 02324 [cs].
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-A-Video, One-Shot Tuning of Image Diffusion Models for Text-to- Video Generation, March 2023. URL http:://axii..rrrrg abs///bs//2222 arXiv:2212. 1156 [cs].
11565
Weijia Wu, Zhuang Li, Yuchao Gu, Rui Zhao, Yefei He, David Junhao Zhang, Mike Zheng Shou, Yan Li, Tingting Gao, and Di Zhang. DragAnything: Motion Control for Anything us- ing Entity Representation, March 2024. URL lttt:://axxiv.ot http://rxxvv..r//a//////0 hbtp://bb/244333 07420 arXiv:2403.0743 [cs].
Jiraphon Yenphraphai, Xichen Pan, Sainan Liu, Daniele Panozzo, and Saining Xie. Image Sculpt- ing: Precise Object Editing with 3D Geometry Control. In 2024 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4241-4251, Seattle, WA, USA, June 2024. IEEE ISBN 3503--3300--30--66
doi: 10.1099CCVPS2233.2224..000 URL https: //ieeexplore. leee. http///ccunenttttcsses/
Xin Yu, Tianyu Wang, Soo Ye Kim, Paul Guerrero, Xi Chen, Qing Liu, Zhe Lin, and Xiaojuan Qi. ObjectMover: Generative Object Movement with Video Prior, March 2025. URL http:
//
08037 a:Xiv:2503.08037 [cs].
Lvmin Zhang and Maneesh Agrawala. Adding Conditional Control to Text-to-Image Diffusion 05543 arXiv:2302.05543
Models, February 2023. URL
[cs].
Ziyue Zhang, Mingbao Lin, and Rongrong Ji. ObjectAdd: Adding Objects into Image via a Training- Free Diffusion Modification Fashion, May 2024. URL htttt::://xxii.orr//bbbbb /244
17230 arXiv:2404.1 17230 [cs] version: 2.
Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. UniPC: A Unified Predictor- Corrector Framework for Fast Sampling of Diffusion Models, October 2023. URL http://
arxiv.org/aaao abs /202 220.2322220222 04867 arXiv:2302.0486) [cs].
Zibo Zhao, Zeqiang Lai, Qingxiang Lin, Yunfei Zhao, Haolin Liu, Shuhui Yang, Yifei Feng, Mingxin Yang, Sheng Zhang, Xianghui Yang, Huiwen Shi, Sicong Liu, Junta Wu, Yihang Lian, Fan Yang, Ruining Tang, Zebin He, Xinzhou Wang. Jian Liu, Xuhui Zuo, Zhuo Chen, Biwen Lei, Haohan Weng. Jing Xu, Yiling Zhu, Xinhai Liu, Lixin Xu, Changrong Hu, Shaoxiong Yang, Song Zhang, Yang Liu, Tianyu Huang, Lifu Wang. Jihong Zhang, Meng Chen, Liang Dong, Yiwen Jia, Yulin Cai, Jiaao Yu, Yixuan Tang, Hao Zhang, Zheng Ye, Peng He, Runzhou Wu, Chao Zhang, Yonghao Tan, Jie Xiao, Yangyu Tao, Jianchen Zhu, Jinbao Xue, Kai Liu, Chongging Zhao, Xin- ming Wu, Zhichao Hu, Lei Qin. Jianbing Peng, Zhan Li, Minghui Chen, Xipeng Zhang, Lin Niu, Paige Wang, Yingkai Wang, Haozhao Kuang, Zhongyi Fan, Xu Zheng, Weihao Zhuang, YingPing He, Tian Liu, Yong Yang. Di Wang, Yuhong Liu, Jie Jiang, Jingwei Huang, and Chunchao Guo.
13

---

Published as a conference paper at ICLR 2026
Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generation, February 2025. URL atttp://arxta http://abs//1...2222 12202 arXiv:2501.1220 12200 [cs].
Bojia Zi, Weixuan Peng, Xianbiao Qi, Jianan Wang, Shihao Zhao, Rong Xiao, and Kam-Fai Wong. MiniMax-Remover: Taming Bad Noise Helps Video Object Removal, May 2025a. URL http:
2 24873 arXiv:2505.24873 [cs].
Bojia Zi, Penghui Ruan, Marco Chen, Xianbiao Qi, Shaozhe Hao, Shihao Zhao, Youze Huang, Bin Liang. Rong Xiao, and Kam-Fai Wong. Señorita-2Mr High-Quality Instruction- based Dataset for General Video Editing by Video Specialists, March 2025b. URL /ttp:://axxvv.ortt
abs/ 2502. 06734 arXiv:2502.066 067344 [cs].
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 LLM USAGE
In preparing this manuscript, utilized ChatGPT and Grok, solely for language polishing and minor refinements to improve clarity, grammar, and flow the text. The LLM was provided with sections of the draft and asked suggest revisions, which were then reviewed, edited, and incorpo- rated by the authors as deemed appropriate. All core ideas, research contributions, technical details, and analyses originate from the authors and were not generated or ideated by the LLM. No other LLMs were used in the research process.
A.2 TRAINING DETAILS
Training Details. We build on the Wan-1.3B backbone with eight control blocks. In Stage we train for 50k optimizer steps using AdamW (learning rate 5 x 10-5) with a One-Cycle scheduler. The training resolution is 640 x 960 for images and 61 640 960 for videos. We use 32 A100 GPUs for training, employing mixed precision with 6 and DeepSpeed zero optimization stage During training, we randomly drop the camera pose condition with 0.1 probability; during inference, we apply a classifier-free guidance (CFG) scale of 1.5 with UniPC (Zhao et al. 2023) sampler for 40 steps. Stage fine-tuning runs for steps on mixed image/video data), retaining the same optimization setup unless otherwise specified.
Dataset Construction Details. We curate large-scale corpus for object manipulation through a two-stage process. Source Acqui- sition and Filtering. We start with approxi- mately 400k videos from Pexels, using the first frame of each clip the source image for mesh reconstruction. In parallel, we synthe- size 1000 diverse object-centric images using HunyuanT21. We apply prompt-based filtering exclude non-manipulable categories (e.g., roads, buildings) and discard instances exhibit- ing disconnected components, heavy trunca- tion, or obvious deformations. Mask Extruc- tion and 3D Reconstruction. For each retained source image, obtain object mask us- ing Grounded-SAMe (Ren et al.] 2024) and reconstruct a textured, watertight mesh with Hun (Zhao et al. 2023). This process yields 100k object meshes after filtering. Pair Synthesis for Stage I: For each object mesh we randomly sample 20 different camera views and object placements to render ~2 image pairs for pretraining in Stage High-Quality Subset for Stage For Stage II, we utilize only meshes derived from the Pexels dataset. We estimate the source camera pose via differentiable rasteriza- tion and filter out images/videos where the source camera estimate is unreliable (IoU S 0.90). For the retained samples, render the mesh under new camera view and employ object pasting model to composite the rendered object onto the background image/video. We further conduct an automatic harmonization sanity check using ChatGPT- ChatGPP-Image-1 to exclude poor pastes. The result- ing Stage II dataset comprises roughly ~50K video pairs and ~50K image pairs for high-fidelit fine-tuning.
Source Randered Object Image f WA
photo//dde tnbject) Backgroand ligh quilty. reelutic
Randored Oljject Masi
Randaare Cijact
Smage
f WH
Test Encoder
Ultshuffte
Frome Opro
Dast
Frowe Concue
Chavel Conett
DIT Block
DIT Block
DIT Block
Control Blook
Block
Figure 6: Training of the object pasting model. The model is trained to recover the real-world im- age/video given background frames and rendered object image.
A.3 OBIECT PASTING MODEL
As illustrated in Figure 6 object pasting model adopts architecture similar to that of the object manipulation network. This model is designed to composite the rendered object image onto the background frames, harmonizing it with the surrounding scene by adjusting lighting, shadows,
15

---

Published as a conference paper at ICLR 2026
and other visual attributes while preserving the object's original orientation and position. The inputs include the background frames, the rendered object image, the corresponding rendered object mask, and textual prompt. During training, the rendered object image and mask generated using the estimated source camera pose s with the objective of reconstructing the original source video. This process enables the model learn precise object pasting and seamless integration with the background. At inference, the rendered object image and mask are provided at the target location within the same background, yielding the target frames X We employ Wan-1.3B the backbone, augmented with 16 control layers; the backbone is frozen, and only the control branch is updated during training. The model is trained for 20K steps a learning rate of x 10-5 on 16 A00 GPUs. During inference, we apply cfg of 5.0 with 40 steps using UniPC sampler.
A.4 BASELINE IMPLEMENTATION DETAILS
Implementation Details of 3DIT. We use the pretrained checkpoint of 3DTT, we first translate the object according to our NDC shifts (convert from [-1,1] to [0.1] to match the convention of 3DIT). For rotation, since 3DIT accepts the object rotation angle input, while our R represents the relative camera rotation, we extract the angle as the negative of the yaw from Riel:
θ = atanntwo Rree j0, 2], R,, R,,
which accounts for the opposite direction between camera and object rotations. We adopt the default settings of cfg 3.0 and the input image is resized to 256 x 256 to match 3DiT's resolution and resized back after generation for qualitative and quantitative comparisons.
Implementation Details for DragAnything. We employ the pretrained checkpoint of DragAny- thing and draw linear Gaussian path starting from the source NDC coordinates and ending at the target NDC coordinates. The input is resized to 320 x 576 in accordance with its default settings. We adopt the default settings, with CFG of 3.0, and generate 25 frames video. We then extract the last frame as the edited result for comparison.
Implementation Details for VACE. We utilize the pretrained VACE-1.3B model, adapting its "moving anything" configuration to our setting. Specifically, we compute the bounding boxes for the source and target masks, convert them into video-like signals for input to the VACE control branch, and guide generation using the prompt template: "A [bb}] is moving to the {top/bottom), [left/right} side of the image, with terminal location (Ix), (ry]." .." We adopt the default settings to generate videos at resolution of 81 x 480 x 832 with a CFG scale of 5.0. The last frame from the generated video is extracted for comparison.
Implementation Details of Swen-Imaae.. We utilize the pretrained checkpoint of Qwen-Image- Edit. The model takes the source image as input, guided by a refined prompt template: "Move the (object) to the [top/hottom), (left/right) side of the image at target location {ra]. [. Rotate the object horizontally by [yaw'et yaw" degrees and vertically by {pitch pitch degrees while preserving the background and other objects unchanged." We employ the default settings, with a classifier--ree guidance scale of 4.0, 50 inference steps, and resolution of 832 x 1248.
Implementation Details of Kontext We utilize the official pretrained checkpoint, FLUX.1- Kontext-dev for inference. The model takes the source image input, guided by refined prompt template: "Move the {object] to the (top/bottom), (left/rghh) side of the image at target location {rz), }ry). Rotate the object horizontally by (yaw's' yaw*") degrees and vertically by {pitch' - -pitch sre degrees while preserving the background and other objects unchanged." " We employ the default settings, with classifier-free guidance scale of 5.0, 20 inference steps, and resolution of 960 x 640.
Implementation Details of Nano-Banana. We use the official PREVIEW API generate edits from source image pre and an estimated target mask M The mask M is obtained by taking the minimum enclosing square of the source mask M and translating it to the target position using normalized coordiacoo (NDC) shifts (re,ry) (cf. Sec. 2.1 We provide the API with Iero Mist and the following structured prompt: "Move the {object) to the (top/bottom). {left/rihtt} side of the image at target location fra fr,,} } so that its center lies inside the white square mask. Rotate the object by {yaw'g: yaw*Fo) degree and (pitch #rtch degree, while preserving the background and all other objects unchanged."
16

---

Published as a conference paper at ICLR 2026
A.5 DETAILS OF THE GEOEDITBENCH
To evaluate the performance of our object manipulation framework, we manually curated the GeoEd- itBench dataset. Specifically, collected 20 common objects and, for each, randomly combined with two or three other objects, then captured images with the target object moved and rotated across different locations and viewpoints. This process was repeated 3 times per object, yield- ing 300 source images for the benchmark. Additionally, we captured 4 multi-view images (front, back, left, and right) of each isolated object and reconstructed high-quality 3D meshes using the Hunyuan3D API. We manually filtered out meshes that did not accurately conform to the object images and annotated object masks using Grounded-SAM-2. Camera poses were estimated via dif- ferentiable rendering, and to establish reliable benchmark, we retained only images where the rendered silhouette achieved an IoU 2 0.95 with the annotated mask. After filtering, the dataset comprises 346 high-quality image pairs. To promote reproducibility and community value, will open-source GeoEditBench.
A.6 FAILURE CASE ANALYSIS
SUPER BIEH
SUPEO HON
Coca
Coaa
Figure 7: Limitations of Ctrl&Shift. Left: source, Right: edited result. Top: Loss of high- frequency details in objects with complex textures. Bottom: Artifacts when handling transparent objects due to texture-baking errors in the 3D reconstruction pipeline.
Figure iillustrates current limitations of Ctrl&Shift. the first row, we observe a loss of fidelity when editing objects with highly complex, fine-grained textures. This issue stems from the synthetic data generation pipeline; specifically, the image-to-mesh reconstruction step often fails to capture high-frequency geometric and texture details. Consequently, the model is trained on synthetic pairs that lack this granularity, limiting its ability to preserve intricate patterns during inference. In the second row, the model struggles with transparent objects. This limitation also originates from the underlying 3D reconstruction method, which cannot explicitly model transparency or refraction. In- stead, erroneously "bakes" background information onto the object's surface texture. This error propagates to the editing model, causing the generated output to appear blurry or opaque, as the model cannot correctly synthesize the dynamic background changes required for transparent mate- rials in new pose.
17

---

Published as a conference paper at ICLR 2026
A.7 MULTI-OBJECT EDITING
Figure 8: Multi-object editing results achieved through sequential editing of distinct objects. From left to right: source image, first edit, second edit.
As shown in Figure we present the results of multi-object editing using our method, which is achieved through sequential editing of distinct objects. Our framework does not currently support simultaneous multi-object editing, and we believe this represents promising direction for future work.
A.8 DETAILED DATA PREPROCESSING PIPELINE AND STATISTICS
Ganrs Dijact Priceet
2009 Otjert Trages
J03N chjeet Hodia
Praaee Sestttiringg Moniay ontinill
Mishoowd
murr der
Mare tos
1004 Helon
Vanes
Objert (nager
2012
BDU Charct
XX0 Visie
Oiject Fasting
S0V Dlym!
XDI Charct
Sooge
Teage
Figure 9: Detailed data preprocessing pipeline and statistics.
Our data pipeline designed to overcome the scarcity of high-quality paired data for object ma- nipulation. It operates in two distinct phases: Stage 1, which utilizes large-scale synthetic data to establish fundamental geometric consistency, and Stage which leverages curated real-world video data to enforce photorealism and correct texture handling.
18

---

Published as a conference paper at ICLR 2026
A.8.1 LARGE-SCALE SYNTHETIC DATA GENERATION
The primary objective of this stage is create a massive volume of object pairs with perfect geo- metric ground truth to train the model's manipulation capabilities. We begin by collecting a diverse set of 20k general object prompts, which are processed by Hunyuan T2I to generate 100k synthetic object images ensuring a wide variety of categories and appearances. These 2D images are lifted to 3D using Hunyuan 3D-2. To guarantee geometric quality, we apply mesh filtering step that iden- tifies and discards meshes exhibiting excessive outliers or incomplete surfaces, resulting in a clean set of 100k verified object meshes. Finally, we render these meshes from randomized novel camera poses; by pairing the original views with these novel views, we generate large-scale dataset of million image pairs.
A.8.2 REAL-WWOLLD DATA SYNTHESIS
To bridge the domain gap between synthetic renderings and real-world imagery, we curate a high- fidelity dataset starting from raw collection of 400k videos sourced from Pexels.
Data Proprecessing and Filtering. We first employ prompt-base semantie filtering to exclude subjects unsuitable for rigid manipulation, including background structures (e.g., buildings, roads), deformable entities (e.g., people), and amorphous textures (e.g., waterfalls); this reduces the cor- pus to 180k videos. Subsequently, we apply mask-based geometric filtering to ensure high-quality segmentation, pruning instances that exhibit disconnected masks, frame-edge truncation, or extreme size ratios. This rigorous filtering process yields a set of 80k candidate object images.
3D Validation and Alignment. The candidate images are processed through Hunyuan 3D-2 to generate corresponding 3D meshes. Since single-view reconstruction from real images is prone to misalignment, we enforce a strict consistency check. We perform camera pose estimation to reproject the generated mesh back onto the 2D image plane and calculate the Intersection over Union (IoU) between the projected mesh mask and the original object mask. Only meshes satisfying an IoU 0.9 are retained, resulting high high-quality subset of 50k verified object meshes.
A.9 ABLATION STUDY ON TARGET MASK STRATEGY
Table 4: Ablation Study on Target Mask Strategy
Training
Inference
PSNR t
DINO 1
CLIP t
DreamSim |
Pose MAPE
Obj IoU t
Estimated
Estimated
28.71
85.23
90.44
0.112
17.70%
0.83
Estimated
GT
28.95
85.51
90.48
0,110
17.35%
0.84
GT
GT
29.58
86.42
90.89
0.104
16.20%
0.86
GT
Estimated
26.83
80.67
84.72
0.185
25.85%
0.71
To demonstrate the robustness of our mask estimation strategy, we present an ablation study in Ta- ble The results reveal several key insights: First, when training with estimated masks, providing ground-truth masks at inference yields only marginal improvement (Row vs. Row 1), as the model has learned to rely primarily the pose descriptor rather than precise mask boundaries for geo- metric manipulation. That means our current deisgn is not sensitive to the target mask estimation. Second, training and inference with ground-truth masks achieves the best performance (Row 3), as accurate masks provide richer geometric information that simplifies the learning task. However, this approach is impractical for real-world deployment where ground-truth masks unavailable without explicit 3D reconstruction. Most importantly, Row 4 demonstrates that models trained on ground-truth masks suffer significant performance degradation when only estimated masks avail- able at inference, confirming that such models develop a strong dependency on precise mask infor- mation. In contrast, our approach (Row 1) maintains robust performance under realistic conditions, validating our design choice to train with estimated masks for practical applicability.
A. ABLATION STUDY ON STAGE II TRAINING STRATEGY
Freezing the backbone during Stage II prevents the model from overfitting to the narrower Stage II distribution and catastrophically forgetting the broad geometric understanding acquired in Stage
19

---

Published as a conference paper at ICLR 2026
I. To validate this design choice, we conducted ablation study comparing approach (frozen main branch with trainable control branch) against fully fine-tuning both branches in Stage II, as shown in Table 5 The results demonstrate that while full fine-tuning achieves marginally better photometric quality (PSNR, DreamSim), significantly degrades geometric accuracy (Pose MAPE: 17.70% 21.45% and object boundary preservation (Ohj IoU: 0.83 vs. 0.79). This confirms our hypothesis that unrestricted fine-tuning on the smaller Stage II dataset causes the model to lose its robust geometric manipulation capabilities. Our frozen-backbore approach strikes better bal- ance, maintaining strong geometric control while still learning effective photometric harmonization through the trainable control branch.
Table 5: Ablation Study on Stage Training Strategy (Zero-shot Evaluation on GeoFditBench)
Stage II Strategy
PSNR t
DINO
CLIP f
DreamSim
Pose MAPE
Obj IoU
Freeze main, train control
28.71
85.23
90.44
0.112
17.70% 21.45%
0.83
Fine-tune main & control
29.12
84.85
90.38
0.108
0.79
20

---

Published as a conference paper at ICLR 2026
A.11
MORE RESULTS
Source Image
Edited Result
Source Image
Edited Result
Figure 10: Additional Results on Object Manipulation.
21

---

Published as a conference paper at ICLR 2026
Source Image
Edited Result
Source Image
Edited Result
Figure 11: Additional Results Object Removal and Reference Image Inpainting with Camera Control.
22


Reviews:
[{"review_id": "SXGIlhyymN", "reviewer": "Reviewer_kSVy", "summary": "This paper introduces Ctrl&Shift, a novel end-to-end diffusion-based framework for geometry-aware object manipulation in images and videos. Unlike prior methods that either rely on explicit 3D reconstruction (e.g., NeRFs, 3DGS) or lack fine-grained geometric control (e.g., prompt- or mask-based diffusion editors), Ctrl&Shift achieves precise object relocation and rotation while preserving background integrity—without requiring explicit 3D modeling at inference time.\nThe core idea is to decompose object manipulation into two subtasks: (1) object removal, and (2) reference-guided inpainting under explicit camera pose control. These are unified within a single diffusion model trained via a multi-task, multi-stage strategy. The model conditions on: source frames, reference object image, source/target masks, and a relative camera pose descriptor (8D: axis-angle rotation, translation, and NDC screen shifts). This pose is encoded via Fourier features and injected through cross-attention.\nTo enhance generalization, the authors introduce a scalable pipeline for constructing real-world datasets with estimated relative camera poses. Extensive evaluations on fidelity, consistency, and controllability demonstrate superior performance over state-of-the-art methods in tasks like object relocation, rotation, removal, and inpainting.", "questions": "1.\tHave you explored extending the framework to handle multiple objects simultaneously or interactive edits (e.g., user feedback loops during inference)?\n2.\tThe relative pose descriptor f is an 8-dimensional vector. How does the model generalize to large, out-of-distribution relative camera movements (e.g., moving from a front view to a top-down view) that may be sparsely represented in the training data?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tInnovative decomposition of object manipulation into removal and pose-controlled inpainting within a unified diffusion model, enabling fine-grained geometric control (e.g., precise relocation and rotation) without relying on explicit 3D representations like NeRF or Gaussians, which improves scalability and avoids per-scene optimization.\n2.\tThe multi-task training approach effectively disentangles conditioning signals (background, object identity, camera pose), leading to interpretable and controllable edits that generalize to real-world content.\n3.\tThe experimental evaluation is comprehensive, including quantitative results on two benchmarks, a detailed user study, and extensive ablation studies that validate the contribution of each proposed component (multi-stage training, auxiliary tasks). The creation of the high-quality GeoEditBench dataset is itself a significant contribution.", "weaknesses": "Heavy Reliance on Data Synthesis Pipeline: The method's performance is heavily dependent on the quality of the synthesized training data generated by the multi-step pipeline (mesh reconstruction → pose estimation → harmonization). Errors or artifacts introduced at any of these stages could propagate into the final model, potentially limiting its performance on objects or scenes that are challenging for these upstream components (e.g., transparent objects, complex textures, artistic images) and also for extreme viewpoint shifts or complex occlusions. The paper would benefit from a deeper analysis of failure cases stemming from the data pipeline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ho44KUUeGD", "reviewer": "Reviewer_akfP", "summary": "This paper addresses geometry-aware object manipulation in images and videos—specifically, relocating and reorienting objects while maintaining photorealism, background preservation, and geometric consistency under viewpoint changes. The proposed solution, Ctrl&Shift, is a diffusion framework that decomposes the task into object removal and reference-guided inpainting under explicit camera control. This approach avoids the need for explicit 3D representations (like meshes or NeRFs) at inference time.\n\nTo enable training on real-world data with geometric supervision, the authors developed a pipeline involving 3D mesh reconstruction, pose estimation via differentiable rendering, and a learned harmonization model (Object Pasting Model) to synthesize target views. The authors introduce a new benchmark, GeoEditBench, and outperform recent methods in both visual fidelity and geometric accuracy.", "questions": "1. Data Pipeline Statistics and Bias: What is the retention rate of the initial data corpus (e.g., Pexels videos) after the IoU ≥ 0.90 filtering step? Could you comment on whether this filtering introduces a bias towards simpler objects or viewpoints, and how the model performs on categories where 3D reconstruction typically struggles?\n\n2. Sensitivity to Target Mask: How sensitive is the model's performance to the accuracy of the estimated target mask $\\hat{M}^{tgt}$ during inference? Could you provide an oracle experiment showing the performance if the ground-truth target mask were available at inference?\n\n3. Harmonization and Shadows: How does the framework ensure realistic shadow casting and lighting for the manipulated object? Are there examples where the object is moved between significantly different lighting environments (e.g., from shadow to direct light)?\n\n4. Stage II Training Strategy: In Stage II, the main branch is frozen (L297). What is the motivation for this? Does this prevent the model from learning complex interactions (e.g., realistic shadow casting) that might require updates to the main diffusion pathway? Was fine-tuning the full model attempted?\n\n5. Large Viewpoint Changes: How does the model handle large viewpoint changes where significant parts of the object, unseen in the source image, must be synthesized? Is the model relying purely on learned priors for this object-inpainting task?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Empirical Results and High-Quality Output. The quantitative results are compelling. On the new GeoEditBench (Table 2), Ctrl&Shift shows substantial improvements in geometric accuracy (17.70% Pose MAPE vs. 24.36% for the next best) while also achieving the highest fidelity scores (PSNR, DreamSim). The qualitative comparisons (Figure 4) are impressive, clearly demonstrating the model's superiority in handling complex rotations and perspective shifts where competitors fail.\n\n- Significant Enabling Contributions (Data Pipeline and Benchmark). The pipeline for creating paired supervision from real-world data (Figure 3) is a major contribution, addressing the bottleneck of acquiring high-quality, pose-annotated data. Additionally, the introduction of GeoEditBench provides a valuable tool for the community to evaluate geometry-aware editing.", "weaknesses": "Several aspects require clarification or further investigation.\n\n1. Reliance on 3D Supervision during Training and Generalization Limits. The paper emphasizes avoiding explicit 3D representations at inference. However, the training data generation (Section 2.5) heavily relies on explicit 3D reconstruction (Hunyuan3D-2) and differentiable rendering. The model's generalization is therefore constrained by the capabilities of the underlying 3D reconstruction method. Furthermore, the rigorous filtering (IoU ≥ 0.90 for pose estimation, L336) likely biases the dataset towards objects that are easy to reconstruct and align. The paper needs a more thorough analysis of the data pipeline's success rate, the diversity of the resulting dataset, and how the filtering affects the model's ability to handle complex geometries, materials (e.g., transparent or reflective), or intricate occlusions.\n\n2. Simplistic Inference-Time Target Mask Estimation. During inference, the target mask $\\hat{M}^{tgt}$ is estimated using simple heuristics: squaring the source mask, scaling by distance ratio, and shifting (Section 2.1). This approach is very coarse and does not account for changes in the object's silhouette due to rotation or perspective distortion. For significant viewpoint changes, this estimate could be highly inaccurate. The paper lacks a sensitivity analysis on how the accuracy of $\\hat{M}^{tgt}$ affects the final output, and it is unclear how the model corrects for large inaccuracies in the estimated mask.\n\n3. Evaluation of Harmonization, Lighting, and Shadows. Photorealistic manipulation requires accurate handling of lighting and shadows. The framework relies on a separately trained Object Pasting Model (A.5) to generate harmonized training data. However, there is no explicit evaluation of this component, nor of the main model's ability to generate plausible lighting and, crucially, realistic cast shadows in the edited scene. The current metrics (PSNR, DINO) do not measure physical correctness. The reliance on this specialized harmonization model during data synthesis might bottleneck the main model's understanding of complex lighting interactions.\n\n4. Underdeveloped Video Manipulation Capabilities. The abstract and introduction prominently feature video manipulation. However, the quantitative evaluations are image-based. The qualitative video results (Appendix A.8) exhibit noticeable flickering and lower visual quality, which the authors acknowledge. The claims regarding high-quality geometry-aware video manipulation are not fully substantiated, and the limitations should be discussed more prominently in the main paper.\n\n5. Lack of Failure Mode Analysis. Understanding when the model fails (e.g., specific object types, extreme pose changes, complex lighting) would provide a more complete picture of the method's limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3duE5oAbOF", "reviewer": "Reviewer_19Wi", "summary": "This paper introduces Ctrl&Shift, a novel diffusion framework for high-quality, geometry-aware object manipulation in images and videos. The core challenge it addresses is the fusion of precise controllability from geometry-based methods with the realism and generalization of diffusion-based approaches.\n\nThe paper's main contribution is to cleverly decompose the complex manipulation task into two sub-tasks: 1) object removal and 2) reference-guided inpainting under explicit camera pose control. The authors design an 8-dimensional relative camera pose descriptor, f, which is injected as a control signal into the diffusion model. This enables end-to-end geometric control during inference without requiring explicit 3D modeling.\n\nTo achieve this, the paper proposes a multi-task, multi-stage training strategy, supplemented by a sophisticated data construction pipeline to generate paired, geometry-supervised, real-world data. Experimental results on the authors' new GeoEditBench and the ObjectMover-A benchmark demonstrate state-of-the-art performance, outperforming existing methods in fidelity, identity preservation, and geometric control accuracy (e.g., pose MAPE and IoU).", "questions": "As weakness.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Conceptual Innovation: The paper's primary strength lies in its core idea. It represents a conceptual shift: instead of relying on expensive or unstable 3D reconstruction (like NeRF or Mesh) at inference time, it injects precise geometric control (a relative pose vector) as a condition into the 2D diffusion process. This is a very clever decoupling that elegantly combines the advantages of both domains.\n\nSystematic Framework Design: The Ctrl&Shift architecture is designed with systematic and sound principles. The multi-task training (main task, removal, inpainting) is clearly motivated and helps the model disentangle the functions of different control signals (identity, location, pose), which is strongly validated by the ablation study (Table 3). The multi-stage training strategy (Stage 1 for priors, Stage 2 for real-world backgrounds) is logical. It first learns the core geometric knowledge on controllable synthetic data before generalizing to complex real-world scenes, effectively balancing geometric understanding and realism.\n\n3.Strong Empirical Results and Evaluation:The method significantly outperforms existing SOTA approaches in both quantitative (Tables 1, 2) and qualitative (Fig. 4) comparisons.The evaluation metrics are comprehensive. They include not only traditional fidelity (PSNR) and identity preservation (DINO, CLIP) metrics but also creatively introduce geometric control accuracy metrics (Pose MAPE, Obj IoU), which are crucial for evaluating \"controllable\" generation tasks.The construction of the GeoEditBench benchmark is also a valuable contribution to the community, providing a standardized evaluation platform for this specific challenge.", "weaknesses": "I agree with the authors that this is excellent and inspiring work. To make the paper more complete and rigorous, I strongly recommend the authors add a 'Limitations and Future Work' discussion to the final version (e.g., in the conclusion or appendix).\n\nI would like the authors to specifically address the following points:\n\nFrom Technical Controllability to Practical Usability:\nThe authors should discuss the challenge of mapping intuitive user interactions (e.g., 2D mouse drags, rotational gestures) to the proposed 8D relative pose descriptor f. A powerful technology is of limited value without an intuitive interface. \n\nBeyond Geometry: The Challenge of Physical Realism:\nThe authors should acknowledge that the current framework focuses primarily on geometric consistency, while the modeling of physical interactions (especially lighting, shadows, and reflections) is limited. Generating physically correct new shadows and reflections based on the new location's lighting conditions is a major challenge for this method (and all manipulation methods) and a valuable direction for future research.\n\nGeneralization Boundaries Induced by the Data Pipeline:\nThe authors should discuss potential failure cases arising from the dependency on the Image2Mesh pipeline. The method will likely struggle to generalize to: Non-rigid objects (e.g., cloth, hair, pets) Transparent or highly reflective objects (e.g., glass, metal)\nTopologically complex objects (e.g., trees, smoke) Scenes with complex occlusion (e.g., moving an object behind another object in the scene; the current 'remove + inpaint' framework may not correctly handle this new depth relationship).\n\nAnd I expect the open-source of the benchmark.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HOLYtwVRq9", "reviewer": "Reviewer_sMZf", "summary": "This paper presents CTRL&SHIFT, a 3D-aware image editing approach. Given an image, the target location (specified by binary mask), and a relative camera pose transformation, the method is able to translate and rotate the object to the desired pose. The core model architecture is similar to VACE, with Control Blocks (Context Blocks) injecting the control signals. To train the model, authors design an automatic data construction pipeline, which produces paired images and ground-truth transformation. The training is done in two stages: first learning object-centric novel view synthesis, then learning blending objects to real-world background scenes. Quantative results on two benchmarks show clear improvement of CTRL&SHIFT over previous baselines.", "questions": "1. For camera conditioning, have you tried common methods in camera-controlled generation, such as Plucker embedding?\n2. I'm also a bit confused by the model architecture. In the dataset construction pipeline, it seems that each data sample contains only 2 images (source and target). However, I believe Wan's video VAE works for `1+4*n` frames. How do you get other frames as model input here?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "- While image editing is a well-studied task, 3D-aware editing is underexplored. This work brings recent advances in general editing to the field, and proposes a large-scale dataset to further improve results.\n- The background preservation when editing foreground objects seems to be pretty good. The visual effects like shadows also look realistic.\n- The new GeoEditBench can be a useful benchmark to the community.", "weaknesses": "1. The paper presentation needs a lot of improvement. The main paper spends too much space talking about details of the data format (Sec. 2.1, 2.2, 2.3), yet the pipeline figure is in the Appendix. I'd suggest introducing the model architecture first, and then define each input & output and their shapes;\n2. Similarly, the main paper only has one paragraph (in Sec.1) discussing related works. This makes it hard to position the paper in the literature. Please move the Related Work part from Appendix to the main paper.\n3. Methodology-wise, while authors claim that architectural design is a contribution of the paper, I don't see why this is novel compared to ObjectMover and VACE. ObjectMover first shows that one can leverage video priors for more consistent object editing. VACE proposes the Context Block control mechanism, which is similar to the Control Blcok in this work. The Reference Images, Source / Target Masks, are conceptually similar to the Context Tokens in VACE. The only difference seems to be the injection of camera pose, yet it has been extensively studied in novel view synthesis and camera-controlled video generation papers.\n4. I appreciate the effort to compare with 6 baselines. However, why not compare with methods that explicitly use 3D bounding boxes or 3D representations such as Diffusion Handles and Image Sculpting? An even simpler baseline can be simply conditioning the diffusion model on the input frames, the reference image, and the 3D object bounding box in the target frame (projected with MLP similar to how you handle camera poses).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)