ChatCam: Empowering Camera Control through Conversational AI
Authors: Xinhang Liu
Yu-Wing Tai
Chi-Keung Tang
Venue: NeurIPS 2024 poster (2024)
Keywords: camera operation
LLM
Primary area: machine_vision

Abstract: Cinematographers adeptly capture the essence of the world, crafting compelling visual narratives through intricate camera movements. Witnessing the strides made by large language models in perceiving and interacting with the 3D world, this study explores their capability to control cameras with human language guidance. We introduce ChatCam, a system that navigates camera movements through conversations with users, mimicking a professional cinematographer's workflow. To achieve this, we propose CineGPT, a GPT-based autoregressive model for text-conditioned camera trajectory generation. We also develop an Anchor Determinator to ensure precise camera trajectory placement. ChatCam understands user requests and employs our proposed tools to generate trajectories, which can be used to render high-quality video footage on radiance field representations. Our experiments, including comparisons to state-of-the-art approaches and user studies, demonstrate our approach's ability to interpret and execute complex instructions for camera operation, showing promising applications in real-world production settings. Project page: https://xinhangliu.com/chatcam.

ChatCam: Empowering Camera Control through Conversational AI
Xinhang Liu' *HKUST
Yu-Wing Tai
Chi-Keung Tang'
Dartmouth College
Abstract
Cinematographers adeptly capture the essence of the world, crafting compelling visual narratives through intricate camera movements. Witnessing the strides made by large language models in perceiving and interacting with the 3D world, this study explores their capability to control cameras with human language guidance. We introduce CharCam, system that navigates camera movements through conversa- tions with users, mimicking a professional cinematographer's 's workflow. To achieve this, we propose CineGPT, GPT-based autoregressive model for text-conditioned camera trajectory generation. We also develop an Anchor Determinator to ensure precise camera trajectory placement. ChatCam understands employs our proposed tools to generate trajectories, which can be used to render high-quality video footage on radiance field representations. Our experiments, including comparisons to state-of-the-art approaches and user studies, demonstrate our approach's 's ability interpret and execute complex instructions for camera operation, showing promising applications in real-world production settings, We will release the codebase upon paper acceptance.
requests and
Introduction
Cinematographers skillfully capture the essence of the 3D world by maneuvering their cameras, creating array of compelling visual narratives [8] Achieving aesthetically pleasing results requires not only deep understanding of scene elements and their interplay but also meticulous execution of techniques.
Recent progress of large language models (LLMs) II has marked a significant milestone AI development, demonstrating their capability understand and act within the 3D world 22 30 87). Witnessing this evolution, our work explores the feasibility of empowering camera control through conversational AI, thus enhancing the video production process across diverse domains such as documentary filmmaking, live event broadcasting, and virtual reality experiences.
Although the community has devoted considerable effort to controlling the trajectories of objects and cameras in video generation approaches for practical usage 4
82 75 28), or predicting similar sequences through autoregressive decoding processes 35 64], generating camera trajectories has yet to be explored. This task involves multiple elements such as language, images, 3D assets, and, beyond mere accuracy, necessitates visually pleasing rendered videos as the ultimate goal.
We propose CharCam, system that allows users control camera operations through natural language interaction. As illustrated in Figure [ leveraging an LLM agent to orchestrate camera operations, our method assists users in generating desired camera trajectories, which can be used to render videos on radiance field representations such as NeRF 52 or 3DGS (36
At the of our approach, introduce CineGPT, GPT-based autoregressive model that integrates language understanding with camera trajectory generation. We train this model using paired text-trajectory dataset equip with the ability for text-conditioned trajectory generation. We also
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

3D Scene
First capture the Opera House with the Harbour Bridge the background.
Generated Camera Trajectory & Rendered Video
Ascend top- down aerial shot, then pan the camera the opposite side.
Let's capture video for breathtaking Opera House!
Next, zoom in while decreasing the focal length, encompa ssing city skyline. Stand by, and action!
User
Suret you like it?
Alright, we can operate the
ChatCam
this...
Figure 1: Empowering camera control through conversational AI. Our proposed CharCam assists users in generating desired camera trajectories through natural language interactions. The generated trajectories can be used to render videos on radiance field representations such as NeRF [52 or 3DGS 336]
propose an Anchor Determinator, a module that identifies relevant objects within the 3D scene to serve anchors, ensuring correct trajectory placement based user specifications. Our LLM agent parses compositional natural language queries into semantic concepts. With these parsed sub-queries as inputs, the agent then calls proposed CineGPT and Anchor Determinator. It composes the final trajectory with the outputs from these tools, which can ultimately be used to render a video that fulfills the user's request.
With comprehensive evaluations and comparisons to other state-of-the-art methods, our method exhibits pronounced ability to interpret and execute complex instructions for camera operation. Our user studies further demonstrate its promising application prospects in actual production settings. In summary, this paper's s contributions are as follows:
We introduce ChatCam, a system that, for the first time, enables users to operate cameras through natural language interactions. It simplifies sophisticated camera movements and reduces technical hurdles for creators.
We develop CineGPT for text-conditioned camera trajectory generation and an Anchor Determinator for precise camera trajectory placement. Our LLM agent understands users' requests and leverages our proposed tools to complete the task.
Extensive experiments demonstrate the effectiveness of our method, showing how Al can effectively collaborate with humans on complex tasks involving multiple elements such as language, images, 3D assets, and camera trajectories.
2 Related Work
70 enabled by Multimodal Language Models. Large-scale language models (LLMs) 9 19
17
extensive datasets and model size, have demonstrated surprising emerging abilities. The emergence of multimodal models 43
33 captivating they process text alongside other modalities such as images [25]. audio 27. and videos [78). Some unified models can perceive inputs and generate outputs in various combinations of text, images, videos, and audio [5 68 77 86). LLMs hold the potential to act as agents [76] 80 622] allowing them to be driven by goals, reason about their objectives, devise plans, utilize tools, and interact with and gather feedback from the environment. Our proposed method involves multiple modalities including language. images, 3D fields, and camera trajectories, and utilizes LLMs as agents assist in operating cameras.
42
Radiance Field Representations. Utilizing continuous 3D fields modeled by MLPs and volumetric rendering, Neural Radiance Fields (NcRFs) [52) achieved breakthrough for novel view synthesis.

---

From outside the window, enter the bedroom and explore its layout following an S-shaped path.
Observation:
The user is requesting camera trajectory, starting from location outside the window and then covering the bedroom's layout following an S-shaped path.
need to find a starting point outside the window, then move through the window into the and capture the room using an S-shaped path. Additionally, I should smooth panning speed to better showcase the room's layout.
Reasoning:
Plan:
1. Call Anchor Determinator with "capture the bedroom from outside the window".
2. Call CineGPT with "pan straight forward, smooth panning speed".
3. Call CineGPT with "an S-shaped path, smooth panning speed".
5. Ask the user for further clarification needed.
4. Connect output trajectories from steps 2 and ensuring they pass through the anchor point determined in step
Render video using the generated camera trajectory.
Figure 2: Overview of the ChatCam pipeline. Given camera operation instruction, ChatCam reasons the user's request and devises a plan to generate a trajectory using our proposed CineGPT and Anchor Determinator. The agent then utilizes the outputs from these tools compose the complete trajectory and render a video.
Subsequent research has emerged to improve NeRFs and broaden their applications 69
including 22
enhancing rendering quality 7 .7], modeling dynamic scenes [84 [55 58
2 J.
improving computational efficiency 83
23
and facilitating 3D scene editing 45
46 40)
Replacing the deep MI.Ps with a feature voxel grid network has demonstrated enhancement in More recently, 3D Gaussian Splatting (36] has further advanced visual quality and rendering efficiency. Compared to traditional 3D representations, radiance field representations offer superior photorealistic rendering quality, therefore, this study focuses on camera manipulation upon mainstream radiance field representations such as NeRF or 3DGS.
both training and inference speed 13
66
24
54]
3D Scene Understanding. Early methods for 3D semantic understanding focused on the closed-set segmentation of point clouds voxels. NeRF's capability to integrate information from multiple viewpoints has spurred application 3D semantic segmentation [88]
primarily
20 46 533 65
26
60
31
48
49
21
Among these, 44 37 12] combine image embeddings from
effective 2D image feature extractors 4ll M 59
39 to achieve language-guides object localization, 22 proposes semantic anisotropic Gaussians simultaneously estimate pass. Another line of research integrates 3D with language models for tasks such 3D question answering [3], localization [4] 57 8I].
segmentation, and editing.
geometry, appearance, and semantics in single
and captioning [16]. Additionally, (29 30 87 propose 3D foundation models to handle various
perception, reasoning, and action tasks in 3D environments. However, the Al-assisted operation of
cameras within 3D scenes remains an unexplored area.
Trajectory Control and Prediction. Controlling the trajectories of objects and cameras is crucial to advance current video generation approaches for practical usage. TC4D 4 incorporates trajectory control for 4D scene generation with multiple dynamic objects. Prrect-a-Vides [82). MotionCtrl (75).

---

Inpet Images
Procnpt
Trajectory-text
Quantizarion
s close image of LEGO Techntc bailduzer with kitchen backsrrop.
Translation
Rotation
Gently push the camera jorward while leeping it ralling."
Focal
Velocity
Anchor Selector
Trajectory Tokenizer
Anchor Refinement
Tokenizerr
Text Tokens
Trajectory Tokens
Transformer Decoder
Output Tokens
Rendered Anchor Image
3D Reconstruction
CineGPT
Anehor Determination
Figure 3: (a) CineGPT. We quantize camera trajectories to sequences tokens and adopt GPT- based architecture to generate the tokens autoregressively. Learning trajectory and language jointly, CineGPT is capable of text-conditioned trajectory generation. (b) Anchor Determination. Given prompt describing the image rendered from anchor point, the anchor selector chooses the best matching input image. An anchor refinement procedure further fine-tunes the anchor position.
and CameraCtrl 28 manage camera pose during video generation; however, they are either limited to basic types or necessitate fine-tuning of the video diffusion model. Moreover, these approaches require user-provided trajectories, whereas we, for the first time, generate camera trajectories conditioned on text.
3 Method
Figure 2 provides an overview of our method's pipeline. ChatCam analyzes the user's camera operation instruction and devises plan generate trajectory using proposed CineGPT and Anchor Determinator. Finally, an Al agent utilizes the outputs from these tools to compose the complete trajectory.
3.1
Text-Conditioned Trajectory Generation
To enable eot-conditioned trajectory generation, we collect text-trajectory dataset and introduce CineGPT, GPT-based autoregressive model integrating language and camera trajectories. Illustrated Figure B (a), our method quantizes camera trajectories into sequence of trajectory tokens using trajectory tokenizer. Subsequently, multi-modal transformer decoder is employed to convert input tokens into output tokens. Upon training, our model adeptly generates token sequences based on user-provided text prompts. These sequences are then de-quantized to reconstruct the camera trajectory.
Camera Trajectory Parameterization. For each single frame, our camera parameters include rotation R E 3x3 translation t E R and intrinsic parameters K R3x3 We further convert the rotation matrix R into the $ x S space [89 to facilitate computational efficiency and simplify the optimization process. The total M-frame camera trajectory formulated as:
C1:as M {RR,, K.)
To additionally model the velocity of camera movement, we introduce a global parameter t represent- ing the total duration. Consequently, the instantaneous velocity of each frame be approximated by the relative translation and rotation to the previnous frame over unit time.
Text- Trajectory Dataset. Given the scarcity of readily available data on camera operations, manually constructed approximately 1000 camera trajectories using Blender [18]. These trajectories encompass a diverse range of movements, including various combinations of translations, rotations,

---

focal lengths, and velocities. Each trajectory is accompanied by a human language description detailing the corresponding movements. This dataset spans various scenarios, capturing both simple pan-tilt-7oom motions and more complex trajectories mimicking real-world scenarios.
Trajectory Tokenizer. We leverage trajectory tokenizer based the Vector Quantized Variational Autoencoders (VQ-VAE) architecture [72] to represent camera trajectories as discrete tokens. Our trajectory tokenizer consists of an encoder & and decoder D. Given an M M-frame camera trajectory C1:M = { the encoder & encodes it into L trajectory tokens I {zi]/=1 where L = M/l and I is the temporal downsampling rate. The decoder then decodes 21:L back into the trajectory CI:M {a Specifically, the encoder & first encodes Trame-wise camera parameters C1:M into latent vector &1.L {(CL:M), by performing ID convolutions along the time dimension. We then transform .1. into collection of codebook entries through discrete quantization. The i== consists of K latent embedding vectors, each with dimension d. The quantization process Q.) replaces each row vector with its nearest codebook entry, as follows:
learnable codebook Z {z: JKK
2; Q(2:) = arg min || Z4E2
zk 11
where || |l2 denotes the Euclidean distance. After quantization, the decoder projects æ1:7. back to the trajectory space as the reconstructed trajectory êI:M = D(zi.t)) In addition to the reconstruction loss, we adopt embedding loss and commitment loss similar to those proposed in [85] to train our trajectory tokenizer. With a trained trajectory tokenizer, a camera trajectory C1:ad can be mapped to sequence of trajectory tokens 21:L, facilitating the joint representation of camera trajectory and natural language for text-conditioned trajectory generation.
Cross-Modal Transformer. We utilize a cooss- transformer decoder to generate output tokens from input tokens, which may consist of text tokens, trajectory tokens, or a combination of both. These output tokens are subsequently converted into the target space. To train our decoder-only transformer, we denote our source tokens as X, = {ar1) and target tokens as X; = {-t] M We feed source tokens into it to predict the probability distribution of the next potential token at each step (x,)))) = II, (2) ( x.). The objective function is formulated as:
No Σ
LIM =
log logpo(zila x22)
By optimizing this objective, we aim to equip CineGPT with the ability to capture intricate patterns and relationships within the data distribution. We then fine-tune CineGPT supervised trajectory- language translation leveraging our paired text-trajectory dataset, where the input for this stage can either be camera trajectory text description, while the target is the opposite modality. During inference, CineGPT can generate camera trajectories solely from textual descriptions as inputs.
3.2 Object-Centric Trajectory Placement with Anchors
While CineGPT enables lext-conditioned trajectory generation, its generation process solely fo- cuses on determining the camera's movements, without contextual connection to specific scenes. Consequently, CineGPT alone cannot effectively handle user prompts that involve object-centric descriptions, such as directives like "directly above the Sydney Opera House". In this light, we bridge trajectory generation with each underlying scene with "anchors" serving reference points within the scene to achieve more accurate placement of trajectories, as illustrated in Figure (b).
Our anchor determination procedure takes natural language descriptions of an image as input. This procedure identifies a set of camera parameters that can render an image that best matches the given description. Current 3D visual grounding approaches [57] 81) typically entail learning 3D feature field 140 37 and localizing objects within the scene, which often results in high computational costs. In contrast, our anchor determinator adopts a different strategy. Initially, it selects the input image that best matches the given text description as an initial anchor. Subsequently, an anchor refinement process is employed to iteratively improve upon this initial anchor, ultimately yielding the final anchor. This approach offers a more efficient alternative to traditional methods, reducing computational overhead while still achieving accurate scene anchoring.
Initial Anchor Selector. Since our method leverages radiance field representations to render videos, naturally have access to the input images for training the 3D scene representations. We utilize
5

---

an initial anchor selector based on CLIP [59] to choose the image from these input images that best matches the text prompt. To be specific, for i-th input image 4; we extract their CLIP image features and convert the text prompt T into CLIP text feature. Next, we compute the cosine similarity between the CLIP text feature vector and each of the CLIP image feature vectors. We select the best matching image with the highest cosine similarity score as the initial anchor. This can be formulated as:
Simage (T)) Sexx (T)
fncchor = arg max im (ill fex (T)
where Simuge()) (-) and (+) represent the image and text feature extractor, respectively.
Anchor Refinement. Using the camera parameters Canchor associated with the selected image as initialization, we further minimize the following objective to obtain the final anchor camera parameters:
min Lanchor (c) = Simage (R(c)) Sien(T) (T)
immgg (ll))llll (T)Il
where R(-) is the rendering function and e: is initialized with Canchor. The optimization of is performed using gradient descent, with the update rule given by:
C++1 = Ce - 1/ cfamcaur (ct),
where 7) is the learning rate. The optimization typically achieves convergence within 100 to 1000 steps. This refinement process ensures that the camera parameters are adjusted to better match the text prompts, handling cases where the initial input images do not align well with the prompts.
3.3 Trajectory Generation through User-Friendly Interaction
With our proposed CineGPT and anchor determination, a large language model acts as agent to interpret the user's requests, generates a plan to use various tools, and composes a final camera trajectory. We adopt GPT-4 [1] to interpret users' natural language inputs and subsequently produce trajectory prompts. Specifically, we use a carefully designed prompt to instruct the LLM agent to reason about the user's requirements and devise plan consisting of the following steps: 1) Break down the complex text query into sub-tasks that CineGPT and the Anchor Determinator can effectively handle. 2) Use these tools to generate atomic trajectories and determine anchor points. 3) Compose the final trajectory by concatenating atomic trajectories and ensuring they pass through the anchors.
Observing, Reasoning, and Planning. Research indicates that LLMs can be prompted to decompose complex goals into sub-tasks, essentially thinking step-by-step [76). As illustrated in Figure 2 we begin by instructing the agent to describe its observations, providing a summary the current situation. The agent then uses this summary to reason and develop mental scratchpad for high- level planning. Finally, it outlines specific steps to achieve the overarching goal of generating the user-required camera trajectory.
We inform our agent of the expected input and output format, i.c.. the APIs, of our proposed CineGPT and Anchor Determinator, and instruct the agent to interact with them following the given format. its outlined specific steps to generate the user-required camera trajectory, it first calls CineGPT and Anchor Determinator to obtain atomic trajectories and anchor points, respectively. Note that both tools can called multiple times, and multiple atomic trajectories can later be concatenated into final trajectories that pass through all anchor points correctly.
Utilization of Proposed Tools.
Final Trajectory Composition. Here we explain how combine atomic trajectories from CineGPT with anchor points to form the final trajectory. The agent first decides the role of the anchors in the ultimate trajectory, either as a starting point an ending point some atomic trajectory. Then affine transformations are applied to the respective atomic trajectories to ensure that their starting or ending points align with the anchor points. For the remaining atomic trajectories not controlled by anchor points, affine transformations are applied to make the endpoint of the previous trajectory align with the starting point of the subsequent trajectory.
4 Experiments
We assess the performance of our proposed ChatCam for human Linguage-guides camera operation across a series of challenging scenarios. Through ablation studies, provide empirical evidence of

---

Zoom in from directly above the Sydney Opera House and roll the camera.
Do a dolly zoom focusing on the chandelier.
Pan the camera from left to right along the piano, then turn to look at the bicycle on the right
Figure 4: Qualitative results on indoor and outdoor scenes. Visualizations of our generated trajectories from input text descriptions and the frames the final rendered video. Our method is capable of understanding and executing instructions and providing correct translations, rotations, and camera focal lengths. Additionally, our method can comprehend more specialized terms such as "dolly zoom".
the effectiveness of its fundamental components. We kindly refer the reader to our supplementary material for additional experimental results, including rendered videos.
4.1 Experimental Setup
Implementation Details. We implement our approach using PyTorch (56) and conduct all the training and inference on a single NVIDIA RTX 4090 GPU with 24 GB RAM. The trajectory tokenizer has codebook with K 256 latent embedding vectors, each with dimension d = 256. The temporal downsampling rate of the trajectory encoder is 4. Our cross-modal transformer decoder consists of 24 layers, with attention mechanisms employing inner dimensionality of 64.
7

---

Sweep across the boy in black, the keyboard, and the boy white, then zoom out to frame the boys and the white guitar together.
Starting from the man playing cards, pan the camera along the corridor, then back up to the end and look to the right.
ONE
Figure 5: Qualitative results on human-centric scenes. Visualizations of our generated trajectories from input text descriptions and the frames in the final rendered video. Our method performs effectively in scenes with multiple humans.
The remaining sub-layers and embeddings have a dimensionality of 256. We train CineGPT using the Adam optimizer (38) with initial learning rate of 0.0001. It takes approximately 30 hours to converge. Our anchor determination utilizes CLIP (59] with a ViT-B/3 VTT-B/32 Transformer architecture. The learning rate of anchor refinement is 0.002. By default, use GPT-4 []] as our LLM agent, and its prompt will be released with our codebase. We render final videos using 3DGS 36 as the 3D) representation.
Tested Scenes. We tested our method on scenes from a series of datasets suitable for 3D recon- struction with radiance field representations, including: (i) mip-NeRF 360 (6), a real dataset with indoor and outdoor scenes. (ii) OMMO [50], a real dataset with large-scale outdoor scenes. (iii) Hypersim [61], synthetic dataset for indoor scenes. (iv) MannequinChallengg [44], a real dataset for numan-centric scenes. If camera poses associated with images were not provided, we used COLMAP [63] for camera pose estimation. For each scene, we reconstructed using all available images without train-test splitting.
Baselines. As the first method to enable human language-guided camera operation, there is no established direct baseline for comparison. Therefore, we adopt 3D understanding approaches based radiance field representations to let the LLM agent attempt to select series of images corresponding to the input text from input images and interpolate their camera poses to construct camera trajectories. These methods include LERF (37, utilizing CLIP embeddings, and SA3D [2]. utilizing SAM embeddings.
Evaluation Metrics.
To evaluate the accuracy of the generated trajectories. we manually construct ground truth trajectories and compute the mean squared errors (MSEs) of translations and rotations relative to them. Additionally, we conduct user study to evaluate the rendered videos using generated camera trajectories, where users are asked to select the video with the best visual quality and best alignment with the input text.
8

---

Facing the piano, pull the camera back, then glance over at the TV on the left, and back to the piano.
SA3D
LERF
Ours
Figure 6: Qualitative comparisons. Our approach avoids moving the camera to unreasonable positions such as inside objects, obtaining videos with better visual effects, and aligning best with input texts.
Table 1: Quantitative comparisons and evaluations. Our full model performs better than baselines and variants in terms trajectory accuracy, visual quality, and alignment with input text.
Method
LLM Agent
Anchor Determination
Translation MSE (1)
Rotation MSE (1)
Visual Quality (?)
Alignment (?)
SA3D
GPT-4
19.5
6.3
5.7
3.8
LERP
GPT4
17.7
4.9
9.4
28.3
ChalCam
LLaMA-2
3.6
ChaiCam (Ours)
GPT-3.5
/
7.3
3.5
ChaiCam (Ours)
GPT4
X
16.2
8.5
ChulCam (Ours)
GPT4
2.9
84.9
67.9
4.2
Results
As shown in Figure our method demonstrates the ability to understand and execute camera operation instructions on range of complex indoor and outdoor scenes, giving appropriate translation, rotation, and focal length. Our method also understands more technical terms such as dolly zoom, which creates a special visual effect by zooming the camera out while adjusting the focus. In Figure we further showcase the qualitative results of our method in human-centric scenes. Our method can correctly handle user instructions about specific people and create correct and vivid visual effects.
Comparisons. In Figure we qualitatively compare method with LLM agents utilizing SA3D LERF to locate target objects. The baselines do simple interpolation of keyframes because they have no knowledge about camera trajectories and tend to move the camera to unreasonable spots (such as entering an object). Therefore, the video rendered by baselines contains artifacts and is not correctly consistent with the input text. However, our method achieves better visual quality and alignment with input texts. Quantitative comparisons in Table further prove that our method has better performance and is preferred by users.
Ablation Study. We present our ablation study in Table We evaluate the performance of our method using different LLMS as agents. Our approach achieved the best accuracy using GPT-4 [1 as the agent, better than GPT-3 [9] and LLaMA-2 [70]. Without our proposed anchor determination, our method cannot correctly place trajectories within 3D scenes, thereby being less accurate than our full model.
Conclusion
This paper presents ChatCam, a system designed for camera operation through natural language interactions. By introducing CineGPT, we bridge the gap between human language guidance and camera control, achieving ext-coonditonedd trajectory generation. Our proposed anchor determination procedure further ensures precise camera trajectory placement. Our LLM agent comprehends users' requests and effectively utilizes our proposed tools to compose the final trajectory. Through extensive experiments, we demonstrate the effectiveness of ChatCam, showcasing its ability to collaborate with humans on complex tasks involving language, images, 3D assets, and camera trajectories. ChatCam has the potential to simplify camera movements and reduce technical barriers for creators.

---

Acknowledgements
This work was supported in part by Dartmouth College A&S Startup fund and by the Research Grant Council of the Hong Kong SAR under Theme-based Research Scheme, grant no. T22-606/23R.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tlge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303300000 08774, 2023.
[2] Benjamin Attal, Jia-Bin Huang, Christian Richardt, Michael Zollhoefer, Johannes Kopf, Matthew O' O'Toole, and Changil Kim. HyperReel: High-fidelity 6-DoF video with ray-conditioned sampling, IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[3] Daichi Azuma, Taiki Miyanishi. Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 19129-19139, 2022.
[4] Sherwin Bahmani, Xian Liu, Yifan Wang, Ivan Skorokhodov, Victor Rong, Ziwei Liu, Xihui Liu, Jeong Joon Park, Sergey Tulyakov, Gordon Wetzstein, et al. Tc4d: srnjectory-conditionn text-to-4d generation. arXiv preprint arXiv:2403. 2224403.77922 /7920, 2024.
[5] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In /EEE/CVF International Conference on Computer Vision (ICCV), pages 5855 5864, 2021.
[6] Jonathan T Barron. Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf' 360: Unbounded anti-aliased neural radiance fields. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5470-5479, 2022.
[7] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Zip-nerf: Anti- aliased grid-based neural radiance fields. In /EEE/CVF International Conference on Computer Vision (ICCV), pages 19697-19705, 2023.
[8] Blain Brown. Cinematography: theory and practice: image making for cinematographers and directors. Routledge, 2016.
[9] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS), 33:1877-1901, 2020.
[10] Ang Cao and Justin Johnson. Hexplane: fast representation for dynamic scenes. /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 130-141, 2023.
[11] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers, 2021.
[12] Jaazhong Cen, Zanwei Zhou, Jiemin Fang, Chen Yang, Wei Shen, Lingxi Xie, Dongsheng Jiang, Xiaopeng Zhang, and Qi Tian. Segment anything in 3d with nerfs, 2023.
[13] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision (ECCV), pages 333-350. Springer, 2022.
[14] Dave Zhenyu Chen, Angel Chang, and Matthias NieGner. Scanrefer: 3d object localization in rgh-d scans using natural language. European Conference on Computer Vision (ECCV), pages 202-221. Springer, 2020.
[15] Xiaokang Chen, Kwan- Yee Lin, Chen Qian, Gang Zeng, and Hongsheng Li. 3d sketch-aware semantic scene completion via semi-supervised structure prior. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
[16] Zhenyu Chen, Ali Gholami, Matthias Nießner, and Angel X Chang. ScanZeap: Context- aware dense captioning in rgb-d scans. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pages 3193 3203, 2021.
10

---

[17] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma. Gaurav Mishra, Adam Roberts. Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Sealing language modeling with pathways. Journal of Machine Learning Rexearch, 24(240):1-113, 2023.
[18] Blender Online Community. Blender 3D modelling and rendering package. Blender Foundation. Stichting Blender Foundation, Amsterdam, 2018
[19] Jacob Devlin, Ming- Wei Chang, Kenton Lec, and Kristina Toutanova. BERT: Pre Pre-training of deep bidirectional transformers for language understanding- In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-4188 Minneapolis, Minnesota, June 2019, Association for Computational Linguistics.
[20] Zhiwen Fan, Peihao Wang, Xinyu Gong, Yifan Jiang, Dejia Xu, and Zhangyang Wang. Nerf-sos: Any- view self-superviser object segmentation from complex real-world scenes. International Conference on Learning Representations (ICLR), pages arXiv 2209, 2023.
[21] Zhiwen Fan, Jian Zhang, Wenyan Cong, Peihao Wang, Renjic Li, Kairun Wen, Shijie Zhou, Achuta Kadambi, Zhangyang Wang, Danfei Xu, et al. Large spatial model: End-to-end unposed images to semantic 3d. Advances Neural Information Processing Systems (NeurIPS), 2024.
[22] Sara Fridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In IEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 12479-12488 2023.
[23] Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjao Kanazawa. Plenoxels: Radiance fields without neural networks. In /EEE/CVF International Conference on Computer Vision (ICCV), pages 5501-5510, 2022.
[24] Sara Fridovich-Kan Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5501-5510, 2022.
[25] Rohit Girdhar, Alaacldin El-Nouby, Zhuang I Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 15180-15190, 2023.
[26] Rahul Goel, Dhawal Sirikonda, Saurabh Saini, and PJJ. Narayanan. Interactive Segmentation of Radiance Fields. In JEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR). 2023.
[27] Andrey Guzhov, Federico Rauc, Jörn Hoes, and Andreas Dengel. Audioclip: Extending clip to image, text and audio. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 976 980. IEEE, 2022.
[28] Hao Hc, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation. arXiv preprint arXiv:2404.02201 2024.
[29] Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. arXiv, 2023.
[30] Yining Hong, Zishuo Zheng, Peihao Chen, Yian Wang, Junyan Li, and Chuang Gan. Multiply: A multisensory object- object-centric embodied large language model in 3d world, 2024.
[31] Benran Hu, Junkai Huang. Yichen Liu, Yu-Wing Tai, and Chi-Keung Tang. Nerf-rpn: A general framework for object detection in nerfs. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[32] Qiangui Huang, Weiyue Wang, and Ulrich Neumann. Recurrent slice networks for 3d segmentation on point clouds. JEFF/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2018
[33] Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, et al. Language is not all you need: Aligning perception with language models. Advances in Neural Information Processing Systems (NeurlPS), 36, 2024.
[34] Wonbong Jang and Lourdes Agapito. Codenerf: Disentangled neural radiance fields for object categories. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 12949-12958 2021.
[35] Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as foreign language. Advances in Neural Information Processing Systems (NeurIPS), 36, 2024.
11

---

[36] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics (TOG), 42(4), 2023.
[37] Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. In IEEF/CVF International Conference on Computer Vision (ICCV), 2023.
[38] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
[39] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wen Lo, et al. Segment anything. IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[40] Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmano. Decomposing nerf for editing via feature field distillation. Advances in Neural Information Processing Systems (NeurIPS), 35:23311 1223330, 2022.
[41] Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl. Language-driven semantic segmentation. In International Conference on Learning Representations (ICT.R), 2022.
[42] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2:ip-2: Bootstrapping language image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning (ICML), pages 19730-19742. PMLR, 2023.
[43] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-iman pre-training for unified disionllanguag understanding and generation. In International Conference Machine Learning (ICML), pages 12888 12900. PMLR, 2022.
[44] Zhengqi Li, Tali Dekel, Forrester Cole, Richard Tucker, Noah Snavely, Ce Liu, and William T Freeman. Learning the depths of moving people by watching frozen people. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4521-4530 2019.
[45] Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun- Yan Zhu, and Bryan Russell. Editing conditional radiance fields. In IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[46] Xinhang Liu, Jiaben Chen, Huai Yu, Yu-Wing Tai, and Chi-Keung Tang. Unsupervised multi-view object segmentation using radiance field propagation. Advances in Neural Information Processing Systems (NeurlPS), 35:17730-17743, 2022.
[47] Xinhang Liu, Shiu-hong Kao, Jiaben Chen, nerf reconstruction using pseudo-observations from diffusion models. arXiv preprint 151111 2023.
Yu- Yu-Wing Tai, and Chi- Shi-Keung Tang. Deceptive-nerf, Enhancing
[48] Yichen Liu, Benran Hu, Junkai Huang, Yu-Wing Tai, and Chi-Keung Tang. Instance neural radiance field. In IEEE/CVF International Conference Computer Vision (ICCV), 2023.
[49] Yichen Liu, Benran Hu, Chi-Keung Tang, and Yu- lu-Wing Tai. Sanerf-hq: Segment anything for nerf in high quality, IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[50] Chongshan Lu, Fukun Yin, Xin Chen, Wen Liu, Tao Chen, Gang Yu, and Jiayuan Fan. large-scale outdoor modall dataset and benchmark for novel view synthesis and implicit scene reconstruction. In IEEE/CVF International Conference Computer Vision (ICCV), pages 7557-7566 2023.
[51] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io:
A unified model for vision, language, and multi-modal tasks. In International Conference on Learning Representations (ICLR), 2022.
[52] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision (ECCV), 2020.
[53] Ashkan Mirzaei, Tristan Aumentado- Armstrong, Konstantinos G Derpanis, Jonathan Kelly, Marcus Brubaker, Igor Gilitschenski, and Alex Levinshtein. Spin-nerf: Multiview segmentation and perceptual in- painting with neural radiance fields. In /EEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 20669-20679 2023.
[54] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding, ACM Transactions Graphics (TOG), 41(4):1-15, 2022.
[55] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan Goldman, Steven M Seitz, and Ricardo Martin-Brualla Nerfies: Deformable neural radiance fields. In JEEE/CVF International Conference on Computer Vision (ICCV), pages 5865-5874, 2021.
12

---

[56] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
[57] Songyou Peng. Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Mare Pollefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 815 824, 2023.
[58] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10318-10327, 2021.
[59] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of Machine Learning Research (PMLR), volume 139, pages 8748 8763, 2021.
[60] Zhongzheng Ren, Aseem Agarwala, Bryan Russell, Alexander G. Schwing, and Oliver Wang. Neural volumetric object selection. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
[61] Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M Susskind. Hypersim: A photorcalistic synthetic dataset for holistic indoor scene understanding. In IEEE/CVF International Conference Computer Vision (ICCV), pages 10912-10922, 2021.
[62] Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36, 2024.
[63] Johannes L Schonberger and Jan-Michael Frahm. Structure-from-mmotee revisited. In IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 4104-4113, 2016.
[64] Ari Seff, Brian Cera, Dian Chen, Mason Ng. Aurick Zhou, Nigamaa Nayakanti, Khaled Refaat, Rami Al-Rfou, and Benjamin Sapp. Motionlm: Multi-agen motion forecasting as language modeling. In IEEE/CVF International Conference Computer Vision (ICCV), pages 8579 8590, 2023.
[65] Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Buld, Norman Müller, Matthias Nießner, Angela Dai, and Peter Kontschieder. Panoptic lifting for 3d scene understanding with neural fields. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9043-9052, June 2023.
[66] Cheng Sun, Min Sun, and Hwann- Tzonn Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5459-5469, 2022.
[67] Jiaxiang Tang, Xiaokang Chen, Jingbo Wang. and Gang Zeng. Point scene understanding via disentangled instance mesh reconstruction. European Conference on Computer Vision (ECCV), 2022.
[68] Zineng Tang. Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal. Any-to-any generation via composable diffusion. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
[69] Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul Srinivasan, Edgar Tretschk, Wang Yifan, Christoph Lassner, Vincent Sitzmann, Ricardo - Stephen Lombardi, et al. Advances in neural rendering. In Computer Graphics Forum, volume 41, pages 703-735. Wiley Online Library, 2022.
[70] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09238 2023.
[71] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of dynamic scene from monocular video. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 12959 12970, 2021.
[72] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.
13

---

[73] Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Clip-nerf: Text-and-iman driven manipulation of neural radiance fields. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 3835-3844, 2022.
[74] Liao Wang, Jiakai Zhang, Xinhang Liu, Fugiang Zhao, Yanshun Zhang. Yingliang Zhang, Minye Wu, Jingyi Yu, and Lan Xu. Fourier plenoctrees for dynamic radiance field rendering real-time. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pages 13524-13534, 2022.
[75] Zhouxia Wang. Ziyang Yuan, Xintao Wang, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. arXiv preprint arXiv:2312.03641 2023.
[76] Jason Wei. Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting clicits reasoning in large language models. Advances in neural information processing systems, 35:24824 35:24824-24837, 24837, 2022.
[77] Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal Ilm. CoRR, abs/2309.05519 2023.
[78] Hu Xu, Gargi Ghosh, Po- Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero- zero-shot video-text understanding. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6787-6800, 2021.
[79] Bo Yang, Jianan Wang, Ronald Clark, Qingyong Hu, Sen Wang, Andrew Markham, and Niki Trigoni. Learning object bounding boxes for 3d instance segmentation on point clouds, 2019.
[80] Hui Yang. Sifu Yue, and Yunzhong He. Auto-gpt gpt for online decision making: Benchmarks and additional opinions. arXiv preprint 2.Xiv:2300600222,, 2023.
[81] Jianing Yang, Xuweiyi Chen, Shengyi Qian, Nikhil Madaan, Madhavan Iyengar, David F. Fouhey, and Joyce Chai. Lim-grounder: open-vocabulary 3d visual grounding with large language model as an agent, 2023.
[82] Shiyuan Yang, Liang Hou, Haibin Huang, Chongyang Ma, Pengfei Wan, Di Zhang, Xiaodong Chen, and Jing Liao. Direct-a-vidco: Customized video generation with uscr-directed camera movement and object motion. arXiv preprint 2rXiv:2402.03162, 2024.
[83] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. In IEEF/CVF International Conference on Computer Vision (ICCV), pages 5752-5761, 2021.
[84] Jiakai Zhang, Xinhang Liu, Xinyi Ye, Fuqiang Zhao, Yanshun Zhang. Minye Wu, Yingliang Zhang, Lan Xu, and Jingyi Yu. Editable free-viewpoint video using layered neural representation. ACM Transactions on Graphics (TOG), 40(4):1-18, 2021.
[85] Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, and Xi Shen. 12m-gpt: Generating human motion from textual descriptions with discrete representations. arXiv preprint arXiv:2301.0660 2023.
[86] Juntao Zhang, Yuchuai Liu, Yu- fu-Wing Tai, and ChinKeung Tang. C3net: Compound conditioned controlnet for multimodal content generation. In JEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[87] Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3d-vla: A 3d tisson-langugge-action generative world model. International Conference on Machine Learning (ICML), 2024.
[88] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew Davison. In-place scene labelling and understanding with implicit scene representation. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 15838-15847, 2021.
[89] Yi Zhou, Connelly Barnes, Lu Jingwan, Yang Jimei, and Li Hao. On the continuity of rotation representa- tions in neural networks. /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
June 2019.
14

---

Supplementary Materials for ChatCam: Empowering Camera Control through Conversational AI
This supplementary document presents additional qualitative results and discusses the limitations and societal impacts of our proposed approach.
A
Video
For better visualization our reconstruction results, we create set of video visualizations. We highly recommend to watch http:l/mennar_yiddo..oo.. for more results.
Move the camera over the arm of the man with his arm raised, then pan to the guy and lady taking a selfie together.
Leap from the front of bulldozer the back.
Quickly move back and forth in front of the plants and bikes.
Figure Additional qualitative results. (1)
15

---

Can you try sitting on one of the little stools?
Start from capturing everybody, then move closer to the man wearing the red shirt.
Facing the piano, pull the camera back, then glance over at the TV on the left, and back to the piano.
Figure B: Additional qualitative results. (2)
B Additional Results
We present additional qualitative results in Figure A and Figure B
C Limitations
As the first Al-assisted system for language-guided camera operation, our method relies on LLMs as agents, and therefore its efficiency depends on L.LMs. With the rapid development of the community, this limitation may be alleviated.
16

---

Our current results are limited to static scenes due to the limited availability of high-quality 4D Dynamic NeRF/3DGS data. Extending our approach a dynamic scene would be straightforward by introducing timestamp in the Anchor Determinator. We leave this as one of our future work.
D Societal Impacts
Our approach has great potential to help creators in industries such as television, movies, games, etc. reduce their burden by simplifying the learning costs of utilizing 3D assets like radiance field reconstructions. This allows content creators to focus on their creations. We must also admit that existing 3D assets become content in them, and our method may contribute to the creation of harmful content. We encourage the community to play wisely with ChatCam.
and more abundant, it is inevitable that there will be harmful
17

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction state the contributions made in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in appendix.
18

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We fully describe our proposed pipeline and core building components.
Guidelines:
The answer NA means that the paper does not include experiments.
19

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
We will release the data and code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ 7/nips. /ttppp.cc/pblic/guideeee http:/ooeeosmiiiiooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
20

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We include implementation details for our experiments, to a level of detail that is necessary appreciate the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We describe in detail how we obtain quantitative metrics in our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-simma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We indicate the type and number of GPUs.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
21

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httppp///aaarps...ppoo..cccccced
Answer: [Yes]
Justification:
We conform with the NeurIPS Code of Ethies.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss both potential positive societal impacts and negative societal impacts of the work performed in appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
22

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We credit all such assets by appropriate citations and statements.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- ce-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [Yes]
23

---

Justification:
Our paper includes details about its user study.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not pose such risks.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
24


Reviews:
[{"review_id": "DnWfQa7Zzn", "reviewer": "Reviewer_kHgE", "summary": "The paper proposes a method to generate camera trajectories from user prompts. The idea is to pass on the prompt to an LLM, find a starting anchor location (searched and refined using initial images used to construct the radiance field) and then use CineGPT (a cross-model transformer) trained for the next token prediction on quantized camera trajectories. Compared with some minimal baselines, the proposed approach attains favourable results in terms of MSE errors (on translation and rotation) and user-reviewed visual quality and alignment metrics.", "questions": "- It appears a path prediction network might do well in this case. What is eventually needed is to select a set of anchor locations on the path and then interpolate between them. It is not clear which interpolation algorithm was used for the baselines in the paper, which I believe is a crucial detail. Also, please bring more insights on which aspect of the baselines actually fails, is it the keypoint/keylocation selection or the interpolation.\n\n- I would argue for replacing CineGPT with a rule-based optimizer or an interpolation algorithm. Since it does not observe visual modality, why is it not possible to do it through rule-based system (quantizing key kinds of camera movements).\n\n- It is not clear to me, how to select the extent of camera movement. For example, in Figure 2, after determining the initial anchor (outside the window), CineGPT is called to pan straight forward. What aspect of the algorithm decides how long this pan should go? What stops it from bumping into walls, if the visual modality is not seen or explicit start-end points are not given?\n\n- It is not clear how the dataset was constructed. Was it done in scenes with objects/people present in the CG? Who collected the dataset, and what instructions were given to the collector? How the collection was planned? What kind of camera movements were included? All these details are extremely crucial. \n\n- There are open-source blender movies (e.g. https://spring-benchmark.org/) which are utilized in several benchmarks. Would it be a good idea to exploit these in the dataset construction? \n\n- I would argue for an ordered keypoint prediction network and then spline interpolation between them. It would be a well designed meaningful baseline in this case. \n\n- Not sure if clip can efficiently perform the task of location selection, would like to hear more from the users on the same\n\n- Not fully clear, why would the method generalize to out-of-domain objects/places (for example, an opera house in this case. was it included in the train set?). That reduces the confidence on the final presented output. \n\n- It would be useful to share the used prompt into the author's response, which would help the reviewers to try it out themselves and observe its limitations\n\n- Discussion on some works on camera trajectories would be useful. Professional camera paths are often composed of constant, linear and parabolic segments [A][B]? The static trajectories also play a key role in this? Did the authors think about the importance of static trajectories and did they include it in the dataset?\n\n- The final trajectory composition step appears to be non trivial to me and not fully clear at this stage. It looks like a hack right now.\n\n\n[A] Grundmann et al. Auto-Directed Video Stabilization with Robust L1 Optimal Camera Paths. CVPR 2011\n[B] Gandhi et al. Multi-Clip Video Editing from a Single Viewpoint. CVMP 2014", "limitations": "The authors did not discuss the limitations properly. I do not see an explicit limitations section.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and well presented. I was able to understand the parts \"explained in the paper\" quickly. The figures are well made, and the videos in the supplementary section were easy to consume. \n\n- The idea is well-motivated and I felt the work is in a good direction.\n\n- I liked the idea of next token prediction on quantized camera trajectories and it appears novel to me.", "weaknesses": "- Baselines could be better; I feel the comparisons were extremely weak\n\n- Many details in the paper are missing, making it difficult to comprehend the approach fully. The major misses are the details on the dataset construction, the proper explanation of baselines, and the LLM prompts. I give more detailed questions on the same in the next section.\n\n- related work can be strengthened as well. They can cut a bit on the literature of radiance fields and 3D scene understanding and probably  add more on trajectory learning/optimization", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and insights. We appreciate your comments and would like to address your concerns in the following responses.\n\n**Baselines & Comparisons**\n- **Baseline Interpolation Algorithm.** We use cubic spline interpolation for translations and spherical linear interpolation (SLERP) for rotations, fixing camera intrinsics since baselines don’t account for them.\n\n- **Baseline Failures.** Baselines fail due to their limited understanding of text prompts, missing concepts like orientation (\"turn left\") or shape (\"S-shaped\"). Simple keypoint interpolation often produces incorrect results.\n\n- **Path Prediction Network.** We surveyed path prediction networks but found no related work. Specific references from the reviewer would be appreciated for comparison.\n\n- **Ordered Keypoint Prediction Network + Spline Interpolation** closely aligns with the approach used by ChatCam and our selected baselines, predicting keypoints through the Anchor Determinator or 3D latent embeddings, followed by cubic spline interpolation.\n\n- **Rule-based System** We built a rule-based system based on the suggestion. generating sub-trajectories from text via LLM, using set rules (1-to-1 mappings from text to trajectory) instead of ChatGPT. It predicts key locations using LERF.  Quantitative results are in the table (as suggested by reviewer **dvT5**, reporting metrics computed against multiple GTs for better fairness). **Qualitative results are in Figure C of the attached PDF.** The rule-based system performs worse than our baselines, as it cannot handle complex text descriptions like “go through the tunnel.”\n| Method          | Translation MDGT (↓)  | Rotation MDGT (↓)  |\n|-----------------|-----------------------|--------------------|\n| Rule-based System | 23.2                | 5.5                |\n| Ours             | 4.7                 | 2.1                |\n\n- **Weak Comparisons.** We argue that “as the first method to enable human language-guided camera operation, there is no established direct baseline for comparison.” To our knowledge:\n    1) No method achieves text-to-trajectory translation like CineGPT.\n    2) No multimodal-LLM system allows user-guided trajectory generation and optimization through conversation.\n\n    Therefore, we respectfully point out that stating “the comparisons were extremely weak” is somewhat unfair, as such interactions are not achievable by any baselines.\n\n\n**Extent of Camera Movement.** To keep the pipeline simple, we do not process scale specially. The trajectory scale largely depends on CineGPT's training data when the user doesn’t provide end anchor clues. This explains pan length decisions. Without clear visual modality or start-end points, bumping into walls is possible. Users can guide ChatCam to adjust trajectory scale by providing more anchor points through dialogue, as shown in the **conversation example provided in our general response**.\n\n**Dataset.** Our text-trajectory dataset contains trajectories and text descriptions, not attached to specific objects or scenes. Anchor determination bridges trajectories and specific scenes by placing trajectories in scenes through anchor points.\n\n- **“Why would the method generalize to out-of-domain objects/places?”** CineGPT and its training data are not tied to objects or places (no \"domain\" concept). Scenes like the opera house are recognized by the Anchor Determinator and its underlying CLIP model, allowing our method to work on them.\n\n- **Collection Details.** Data was collected by the authors. We enumerated textual descriptions of camera trajectories and built them in Blender. Movements included basic translations, rotations, focal length changes, and combinations (simultaneous or sequential). We included trajectories from professional cinematography literature. All the trajectories cover similar volumes with aligned translations and rotations.\n\n- **Blender Movies.** Since our dataset is not attached to any objects or places, suggested movies cannot be directly used in the dataset construction. However, they might be a meaningful application scenario for our future study with dynamic scenes.\n\n**CLIP-based Anchor Determination.** We use CLIP to select the image that best matches the text description of the anchor point, as it effectively aligns visual and textual information. CLIP can be replaced by other multi-modal models like BLIP. Our anchor refinement ensures key location accuracy. We verify CLIP-based anchor determination accuracy in the table below. We manually specified the optimal anchor points as ground truth in 10 cases and calculated the MSE between the Anchor Determination results and the ground truth.\n\n| Approach              | Anchor Error (↓) |\n|-----------------------|-------------------|\n| CLIP w/ refinement    | 3.7            |\n| BLIP w/ refinement    | 3.9          |\n| CLIP w/o refinement   | 8.5             |\n\nThese results show that the choice of multi-modal model has minimal impact on accuracy, with both models performing well. However, anchor refinement greatly enhances anchor determination accuracy.\n\n**LLM Prompt.** We post our designed prompt in the general official comment, containing detailed instructions, response template, and examples for LLM. We encourage reviewers to test it with LLMs to better understand our approach. We will also make it public upon acceptance.\n\n**Static Trajectories.** Our dataset does not contain static trajectories, and thus CineGPT fails to produce correct results when prompted with \"hold still\" or something close. This issue can be addressed by adding static trajectories to the dataset. We will discuss works on camera trajectories in the related works as suggested.\n\n**Trajectory Composition.** We elaborate on this step in the official comment below.\n\n**Limitations.** We have discussed the limitations of our method in the appendix and will add a discussion of supporting static trajectories as suggested.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0McxzQCDhJ", "reviewer": "Reviewer_dvT5", "summary": "The paper proposes a method for generating camera trajectories for rendering a 3D scene, conditioned on natural language. This problem statement is novel and original, very useful, and the paper demonstrates convincing results. The method uses two components: a language-conditioned camera trajectory generator, and a language-conditioned camera anchor generator. An LLM takes a high-level language query as input and creates a plan based on these two components.", "questions": "What are the limitations of the approach? Does the LLM fail to produce reasonable plans at times? When does that happen? Is the scale of the trajectories always fully determined? Can you comment more on the failure cases?", "limitations": "Limitations are not mentioned", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The problem statement is novel and seems to be very useful with high potential impact. \nThe method is creative and novel. \nResults are excellent. They demonstrate wide range of trajectories on complex scenes with several interesting elements.", "weaknesses": "The paper fails to demonstrate the planning output of the LLM. Is Fig. 2 an actual output of the method, or just an illustration? I would expect to see many more results of the actual plan used to generate the trajectories. It is very difficult to understand the workings of the method without these. \n\nThe method seems far from reproducible. Sec 3.3 talks about several important components of the method, but only at a very high-level that would not enable anyone to reproduce the results. The details of the finetuning proposed in L142-146 are also unclear. \n\nThe quantitative evaluations rely on ground truth trajectories. Shouldn't there be many possible GT trajectories consistent with a language query? How would this diversity impact the evaluations? The same holds true for output of the system. Can the method generate diverse outputs? If so, the paper should include such results, and ideally, should also update the metrics to reflect this. \n\nIn the output in Fig. 2, there does not seem to be an end anchor point. How would the method know the extent or volume of space the S-shaped path should cover? Are there many cases where the volume of the space is undetermined based on the plan? \n\nComparisons to baselines should also be included as videos, especially as the paper talks about their trajectories.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and valuable feedback. While we are encouraged by your appreciation of our efforts, the issues you point out are constructive. Below, we address them:\n\n**Conversation Example & Planning Output.** Figure 2 shows an actual conversation with the LLM (with visualizations and translation from JSON to human language by the authors). As suggested, in the **new example in the general response**, we show dialogue where users and ChatCam gradually correct and improve the output trajectory. At the same time, we also post our designed LLM prompt in the general response to help better understand such conversations.\n\n**Reproducibility.** To increase the reproducibility of trajectory generation through user-friendly interaction in Section 3.3, we post our designed **LLM prompt** in the general response and will make it public upon acceptance.\n\nL142-L146 describes the training of CineGPT. To provide more details:\ni) In the first stage, we perform unsupervised pre-training in an autoregressive manner, where it conditions on previous tokens in a sequence to predict the next token [1, 2]. This process involves maximizing the likelihood of the token sequences (Equation 3), allowing the model to capture relationships between tokens and generate coherent outputs.\nii) In the second stage, we use the paired text-trajectory dataset to supervise CineGPT for text-to-trajectory and trajectory-to-text translation, to obtain the model weights we finally use.\n\nWe are willing to answer any further technical questions to increase the reproducibility of our work.\n\n**Multiple Outputs/Ground Truths.** To strengthen our evaluation by considering multiple possible ground truth trajectories, we added a set of experiments where we built multiple GT trajectories. We calculated the distance between the predicted trajectory and the nearest GT trajectory using a new metric $\\min_i \\|\\| x_{\\text{pred}} - x_{\\text{gt}_i}\\|\\|_2$ that we call Minimum Distance to Ground Truth (MDGT), and compare it with baselines. From the results in the table, it is evident that considering multiple possible GTs, our method still achieves the best quantitative results.\n\nOur method can generate diverse results. Therefore, we report the Frechet Inception Distance (FID), which measures the distribution discrepancy between the ground truth and generated trajectories, and Diversity (DIV), which evaluates the diversity of generated motion by calculating the variance from the extracted features from the trajectory encoder. We do not report FID and DIV for our baselines because they can only generate deterministic trajectories using simple interpolation.\n\n| Method          | Translation MDGT (↓)  | Rotation MDGT (↓)  | FID (↓)   | DIV (↑)   |\n|-----------------|-----------------------|--------------------|-----------|-----------|\n| SA3D            | 15.7                  | 4.7                | –         | –         |\n| LERF            | 13.3                  | 4.3                | –         | –         |\n| Ours            | 4.7                   | 2.1                | 0.89      | 7.0       |\n\n**Extent of Camera Movement.** The extent of paths like S-shaped ones depends heavily on CineGPT’s training data. Users can control the scale of the trajectory by using textual cues describing the scale. Additionally, users can provide explicit cues to add more anchors to adjust the scale, as demonstrated in the conversation example provided in our general response.\n\n**Baseline Result Video.** We have highlighted the artifacts and inaccuracies caused by the baseline output in the form of images. Due to the limitations of external links, we are unable to include new videos at this stage but will update the baseline result video in our future version.\n\n**Limitations and Failure Cases.** We have discussed the limitations of our method in the appendix, including its efficiency depending on LLMs and the lack of exploration of dynamic scenes. We will incorporate the limitations pointed out by other reviewers in the updated version. The most common failure case of our method occurs when the generated trajectory bumps into or goes through objects like walls or doors. Additionally, if the text prompt input to CineGPT is too complex, especially containing rare descriptions of shapes, it may not generate the correct trajectory. However, we kindly point out that these failure cases can be corrected by chatting with the LLM agent.\n\n**References:**\n1. Radford et al., \"Improving Language Understanding by Generative Pre-Training\".\n2. Radford et al., \"Language Models are Unsupervised Multitask Learners\".", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zizemS5A8l", "reviewer": "Reviewer_Shd5", "summary": "This paper propose ChatCam, a pipeline that utilizes GPT to translate the natural language into professional camera trajectory, which enhances the video production process for common users.", "questions": "If the generated trajectory is not satisfactory or contains collision, how can user interact with the ChatCam? Whether the pipeline can modify the generated trajectory?", "limitations": "The collision in the 3D scenes is not explicitly and hardly constrained in this pipeline.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper divide the task into three steps: observation, reasoning and planning. The first two steps is completed by GPT and get the instruction of every subtasks. In the third step, the author proposes Anchor Determinator and CineGPT to generate the initialization point and camera trajectory.\n2. The qualitative and qualitative results and video demo proves the efficiency of this method.\n3. The paper is well written and easy to follow.", "weaknesses": "The design of this paper is clever and makes good use of the advantages of LLM. But I still has some concerns. If the authors can address my concerns, I would change my rate.\n1. In the planning part of Figure 2, how can you ensure there is no collision with 3D scenes in every sub-step, such as “an S-shaped path, smooth panning speed”. If the  structure of the room is complicated, it is easy to collide with the scenes.\n2. Is natural language really the best way for users to interact？Maybe user can import the 3D GS of scenes into Unreal Engine, where Luma AI provide this plugin (https://www.unrealengine.com/marketplace/en-US/product/luma-ai), and generate the camera trajectory by adding camera key points in the 3D space by UI interface, with position, rotation, camera intrinsics. The frames between key points can be obtained by using bilinear interpolation. You can preview whether collision has occurred in real time, and you can also adjust camera parameters by drag. (https://dev.epicgames.com/documentation/en-us/unreal-engine/creating-camera-cuts-using-sequencer-in-unreal-engine)\n3. Can you show more complicated camera trajectories, such as dolly. The “straight forward and roll”, “S-shaped path”, “from left to right”,  which just need coarse control, seems can be easily obtained by the item 2 I propose above (UE).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and valuable feedback. While we are encouraged by your appreciation for our ideas and results, the issues you point out are crucial.\n\n**Collision.** Collisions can indeed occur, especially when the scene is complex or the user does not provide enough anchor points. To keep the pipeline simple yet effective, we do not handle collisions specifically. In particularly complex scenes, such as indoor scenes with multiple rooms, users can provide more detailed text descriptions to guide ChatCam in finding anchor points to avoid collisions. Moreover, if ChatCam generates trajectories that include collisions, users can guide it to make corrections through further dialogue, as discussed below and demonstrated in the example we provide in the general response, where ChatCam first generates a trajectory that collides with the wall, so the user avoids it by providing more anchor points.\n\n**User Interaction & Modification.** The user and ChatCam can engage in multiple rounds of dialogue to iterate and improve the generated trajectories, similar to interactions between a director and a photographer. We verify this advantage in the **example provided in the general response**. In this example, ChatCam first generates a trajectory that collides with the wall, so the user avoids the collision by providing more anchor points. Furthermore, the user proposes to extend the trajectory with a dolly zoom and \n complete this improvement simply through dialogue.\n\n**Is Natural Language Really the Best Way for Users to Interact?** As discussed in the general response, ChatCam allows lay users without professional knowledge such as 3D rotation, translation, camera intrinsics, and keypoints to produce production-quality cinematography. ChatCam also benefits from a pre-trained LLM with multi-lingual understanding, enabling users to interact with ChatCam in languages other than English. Compared with professional software and plug-ins with UI interfaces, this greatly lowers the entry threshold for lay users.\n\nThrough ChatCam, users can engage in further conversation with ChatCam to interactively and iteratively refine the target camera trajectory, much like how a human director gives natural language suggestions to a human cinematographer in a movie post-production process.\n\nPutting ChatCam into perspective, its contributions are not limited to current camera applications. We believe our explorations in conversational AI’s capability for multimodal reasoning and planning have important implications for building embodied AI agents.\n\n**More Complicated Trajectories.** In in the general response, we present **an additional example of a complex camera trajectory**. This trajectory involves panning downstairs, passing through a tunnel, performing a U-turn, and executing a dolly zoom within a complex room. We believe such intricate operations are not easily achievable by lay users through professional software and plug-ins with UI interfaces, and demonstrate \"a wide range of trajectories on complex scenes with several interesting elements,\" as attested by reviewer **dvT5**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cNBq7JAamA", "reviewer": "Reviewer_oNWk", "summary": "This paper introduces ChatCam, a system that enables camera operation via natural language interactions. This system has two key components. CineGPT is proposed for text-conditioned trajectory generation and an Anchor Determinator for precise camera trajectory placement. Experimental results illustrate ChatCam’s effectiveness in text-conditioned trajectory generation and show the potential to simplify camera movements and lower technical barriers for creators.", "questions": "See limitations.\n\n1. Reference information inaccurate: [32,34], NeurIPS 2023 paper, not NeurIPS 2024\n\n2. For the same text prompt, ChatCam is expected to generate different trajectories for different scenes. A verification is interesting to illustrate the effectiveness of an Anchor Determinator.\n\n3. How long is the trajectory sequence?\n\n4. It’s not clear whether the approach works for dynamic scenes, but it seems the Anchor Determinator does not model the temporal relations.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tChatCam is a novel system to generate camera trajectory via natural language. \n2.\tThe paper is well-written and easy to follow.\n3.\tThe proposed GineGPT and Anchor Determinator are technically solid.", "weaknesses": "The system is only verified on several limited 3D static scenes, and the generated trajectory is relatively short. \n\nSome components are not fully verified. For example, what’s the effect of the text prompt “smooth panning speed” in Fig. 3? How does this text prompt affect the trajectory? \n\nSince the method was only evaluated on a small dataset, and the trajectory seems short, it's not clear whether the generations overfit to a small set of text prompts. \n\nSome experimental details are unclear.\n1.\tHow about the training dataset? L114 suggests there are 1000 trajectories to train the trajectory generation. I would suggest introducing the training dataset in detail in Sec 4.\n2.\tWhat’s the time cost of Anchor Determination in inference?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback as well as thoughtful suggestions and questions. Below, we address your points individually.\n\n**Limited Scenes & Overfitting & Trajectory Length.** The cases we test cover \"a wide range of trajectories on complex scenes (indoor/outdoor, object/human-centric) with several interesting elements\", as attested by reviewer **dvT5**. We do not observe significant overfitting, due to the comprehensive dataset and the leverage of CLIP. Furthermore, ChatCam allows users to correct suboptimal results through conversations, further mitigating possible errors.\n\nThe length of each trajectory in our dataset is about 60 frames, matching the length of a single inference of CineGPT. ChatCam has no limit on the length of the trajectory, and users can continue to add requirements to infer CineGPT multiple times and extend trajectories.\n\nPlease also refer to **the new example we provide in the general response**, where the trajectory involves panning downstairs, passing through a tunnel, performing a U-turn, and executing a dolly zoom within a large complex room.\n\n**Prompts about Speed.** Prompts such as \"smooth panning speed\" affect the speed of the trajectory generated by CineGPT. Specifically, for the same duration (frame number), trajectories can cover different distances. The table below illustrates the effect of such prompts.\n| Prompt               | Duration (frames) | Distance Covered |\n|----------------------|--------------------|------------------|\n| “Pan forward, slowly.”         | 60                 | 24.11       |\n| “Pan forward.”                 | 60                 | 64.5       |\n| “Pan forward, rapidly.”        | 60                 | 91.5       |\n\n\n\n\n**Validating Anchor Determinator.** In Figure D of the attached PDF, we qualitatively show how the same text prompt results in different trajectories in different scenes due to Anchor Determination.  Also, in the original submission, we conduct an ablation study on the Anchor Determinator, with quantitative results in Table 1.\n\n\n\n**Anchor Determinator Inference Time.** On a single NVIDIA RTX 4090 GPU with 24 GB RAM, a single inference of the Anchor Determinator takes approximately 10-15 seconds. The majority of this time is spent on anchor refinement rather than CLIP-based initial anchor selection.\n\n**Trajectory Dataset.** We will introduce the dataset in detail in Section 4.2 as suggested. We enumerate the textual descriptions of camera trajectories and then build a camera trajectory in Blender for each description. The camera movements included are basic translations, rotations, focal length changes, and combinations of these (simultaneous or sequential). We also include some camera trajectories mentioned in professional cinematography literature and build them in Blender. All camera trajectories cover similar volumes, with aligned translations and rotations in the first frame.\n\n**Dynamic Scenes.** CineGPT can be directly applied to dynamic scenes due to its independence from specific scenes. Extending our approach to dynamic scenes would be straightforward by introducing a timestamp in the Anchor Determinator so that the \"anchor\" contains temporal coordinates in addition to spatial coordinates. We will show results on dynamic scenes in an updated version.\n\n**Inaccurate References.** Thank you for pointing this out; we will correct them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We want to thank all the reviewers for their time and insightful feedback regarding our first attempt at empowering LLMs beyond 1D NLP to understand 3D spatial relationships, specifically operating 3D camera trajectories in this paper. We are encouraged by the positive reception of our motivation, recognized as a good direction (**kHgE**), useful with high potential impact (**dvT5**), and making good use of LLMs (**Shd5**). Our submission was considered easy to follow, with well-made figures and videos (**kHgE**, **Shd5**, **oNWk**). Our proposed method was deemed novel (**kHgE**), creative (**dvT5**), clever (**Shd5**), and technically solid (**oNWk**). The results were regarded as excellent, demonstrating a wide range of trajectories on complex scenes (**dvT5**), and proving the efficiency of this method (**Shd5**). On the other hand, we acknowledge and will first address common concerns, followed by detailed responses to individual reviewers.

**Our Purpose.** Our multimodal-LLM approach allows natural language "chat" to instruct the 3D "camera" on reasoning in a complex 3D world. Compared with conventional tools that operate with 3D rotation, translation, camera intrinsics, keypoints, etc., ChatCam enables lay users to produce production-quality cinematography without requiring such technical knowledge. Users can engage in further conversation with ChatCam to interactively and iteratively refine the target camera trajectory, much like how a human director gives natural language suggestions to a human cinematographer in a movie post-production process.

Moreover, beyond its camera applications, our multimodal-LLM approach showcases the potential of conversational AI in multimodal reasoning and planning. This capability is crucial for developing embodied AI agents. We believe our work will not only be influential in its direct applications but also significantly advance the frontiers of the field.

**LLM Prompt.** As suggested by reviewer **kHgE**, we post in the official comment below our designed prompt to instruct the LLM agent. In this prompt, we provide the LLM with detailed instructions and guidelines for tool usage to achieve the target. We also include a template and examples for the LLM's responses. We encourage reviewers to test it out with LLMs on their own to better understand our approach. We will also make it public upon acceptance.

**Complex Case Example with User Interaction.** We provide a new example to verify the advantages of our method. As suggested by reviewer **dvT5**, we post the complete conversation between the user and ChatCam in the comment below. We include the visualization of trajectories and rendered video frames in Figures A & B of the attached PDF. 
In this example, with a complex large-scale indoor scene, we asked ChatCam to generate a trajectory according to a detailed requirement. Initially, ChatCam's result goes directly through the wall. We then guide ChatCam to use new anchor points to avoid collision (concerned by **Shd5**) and successfully correct the trajectory. Finally, we extend this trajectory with a dolly zoom. 

This trajectory involves panning downstairs, passing through a tunnel, performing a U-turn, and executing a dolly zoom within a complex room. We believe such intricate operations are not easily achievable by lay users through professional software and plug-ins with UI interfaces. This example demonstrates how our method allows user modification and interaction, and generates long and complex trajectories in intricate scenes, addressing concerns raised by reviewers **Shd5** and **oNWk**.

Meta-review: This paper proposes a novel method for camera trajectory via the language model. Reviewers have questions and doubts mainly about the experiments including the baseline and data being used.  The author did a good job of addressing the concerns of the reviewer with detailed explanations and clarifications. Although one reviewer is still concerned about the experiment setting, the overall rating from the rest of the three reviewers is positive. Therefore, I suggest to accept this paper as a poster.
Decision: Accept (poster)