Ctrl123: Consistent Novel View Synthesis via Closed-Loop Transcription

## Abstract

Abstract Large image diffusion models have demonstrated zero-shot capability in novel view synthesis (NVS). However, existing diffusion-based NVS methods struggle to generate novel views that are accurately consistent with the corresponding ground truth poses and appearances, even on the training set. This consequently limits the performance of downstream tasks, such as image-to-multiview generation and 3D reconstruction. We realize that such inconsistency is largely due to the fact that it is difficult to enforce accurate pose and appearance alignment directly in the diffusion training, as mostly done by existing methods such as Zero123 [ liu2023zero ] . To remedy this problem, we propose Ctrl123, a closed-loop transcription-based NVS diffusion method that enforces alignment between the generated view and ground truth in a pose-sensitive feature space. Our extensive experiments demonstrate the effectiveness of Ctrl123 on the tasks of NVS and 3D reconstruction, achieving significant improvements in both multiview-consistency and pose-consistency over existing methods.

## 1 Introduction

Recent advancements in novel view synthesis (NVS) from a single image have generated significant enthusiasm within the 3D community [poole2022dreamfusion, wang2023prolificdreamer]. NVS is one of the fundamental tasks for any 3D content generation system. It is important for generating a full 3D model from a single image captured by users or generated by text-to-image models. More recently, Zero123 [liu2023zero] enhances the generalization capability
by leveraging powerful pre-trained image generation models such as Stable Diffusion (SD)  [latentdiffusion]. It fine-tunes Stable Diffusion on image renderings derived from a diverse 3D dataset [objaverse] to learn to generate a novel view conditioned on the reference view and a relative camera transformation. The zero-shot and open-world capabilities of Zero123 [liu2023zero] represent a significant advancement in the field of NVS. Following the pioneering work of Zero123 [liu2023zero], subsequent studies [shi2023zero123++, song2023consistency, liu2023syncdreamer, TOSS] further advanced the overall NVS performance. Zero123-XL [deitke2023objaverse] focuses on data quantity, scaling the training data to up to 10 times more; Zero123++ [shi2023zero123++] restricts novel view generation from arbitrary view to 6 predetermined views, which significantly reduces the complexity of the setting of the task.

Nevertheless, there still exists a fundamental problem in the current NVS models - it is difficult to ensure the generated views to be “consistent” with the desired poses and appearances (the ground truth), even on the training images, as demonstrated in Figure LABEL:fig:fail_case_of_zero123. Both Zero123-XL [deitke2023objaverse] and Zero123++ [shi2023zero123++] fail to generate novel views with 1) accurate poses closely aligned with the ground truth and 2) high-quality image details, which indicate that this inconsistency problem cannot be resolved by simply scaling up the training dataset or simplifying the task setting.

The inconsistency problem is rooted in the training strategy of diffusion models which separately trains the denoiser at different noise levels. More specifically, in each training step of diffusion models, a random timestep $t\in[0,T]$ is sampled with a pre-defined noising schedule $\bar{\alpha}_{t}\in(0,1)$ to corrupt a clean image $\bm{x}$ to a noised image $\bm{x}_{t}$ according to $\mathbf{x}_{t}=\sqrt{\bar{\alpha}_{t}}\mathbf{x}+\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon}$, where $\bm{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$.
Then, a U-Net based conditional denoiser $\phi(\bm{x}_{t},\bm{e};\omega)$ parameterized by $\omega$, takes $\bm{x}_{t}$ and the condition $\bm{e}$ as input and predicts the added noise $\hat{\mathbf{\epsilon}}$. The denoiser is trained with the score-matching loss $\mathop{\min}_{\omega}\|\phi(\bm{x}_{t},\bm{e};\omega)-\mathbf{\epsilon}\|_{2}
^{2}$, at each single noise level $t$, without any constraints on the whole denoising process (from Gaussian noise $\bm{x}_{T}$ to the clean image $\bm{x}_{0}$). This strategy significantly improves the diversity of the generated images but neglects the consistency required by NVS models. In this work, we explore ways to enforce consistency between the generated views and ground truth to enhance the capability of NVS models.

One straightforward way to enforce sample-wise consistency is to add a loss such as Mean Square Error (MSE) in the pixel space between the generated views and ground truth. However, through extensive experiments, a direct loss in pixel space often leads to training collapse due to the nature of Diffusion Models (see Appendix D for more details). Inspired by the recently proposed closed-loop transcription (CTRL) framework [dai2022ctrl, ma2022principles, tong2022incremental], in this work, we propose to align poses of the generated views and ground truth in a latent space using patch features which includes fine-grained information. We name our method Ctrl123, a CTRL-based novel view synthesis method that significantly alleviates the problem of pose inconsistency in NVS. Generally speaking, most current single-image NVS methods [liu2023zero, shi2023zero123++, weng2023consistent123] can be viewed as an open-loop auto-encoder (as shown in Figure LABEL:fig:diagram-a). In this work, we extend the open-loop framework to a closed-loop one by feeding the generated views into the encoder (as shown in Figure LABEL:fig:diagram-b). We then measure the difference between generated views and its corresponding ground truth in the latent feature space. To quantitatively measure the pose consistency of the generated views, we introduce metrics (Average Angle (AA) and Intersection over Union (IoU)) into NVS to measure fine-grained and coarse levels of pose-consistency. Through extensive experiments, we show that Ctrl123 significantly improves the NVS pose and appearance consistency which further leads to significantly better 3D reconstruction compared to the current SOTA methods.
The main contributions of this paper are:

- •
To improve pose and appearance consistency in NVS, we introduce Ctrl123, a novel closed-loop transcription-based novel view synthesis diffusion model that uses patch features to measure (and minimize) differences between generated views and the ground truth.
- •
An in-depth experiment on a sample set of 25 training objects demonstrates that Ctrl123 exhibits significant superiority to Zero123 [liu2023zero], generating novel views much more consistent with the ground truth. Quantitatively, Ctrl123 improves the quality of generated views by a 7 point increase in PSNR. Furthermore, Ctrl123 significantly improves the NVS consistency with a 35.1% increase in AA${}^{15^{\circ}}$ and 42.5% increase in IoU^0.7 (see Table 1).
- •
Encouraged by these findings, we further train Ctrl123 on a large-scale 3D dataset Objaverse [objaverse] and observe similar improvements. Ctrl123 improves AA and IoU on two evaluation datasets, with 2.5%, 4.9% improvement on AA${}^{15^{\circ}}$, and 14.8%, 9.5% improvement on IoU^0.7, respectively.

## 2 Related Works

### 2.1 Diffusion Models

Diffusion models [ddpm] demonstrate remarkable capability in image generation. Together with the development of Vision-Language Models such as CLIP [radford2021learning], it accelerates text-to-image [dalle2, imagen, GLIDE, latentdiffusion] models on large-scale web data [cc12m, laion400m, laion5b]. Subsequent works leverage the powerful pre-trained diffusion models for more controllable image generation, conditioning on more fine-grained prompts such as layout, sketch, and human pose [gligen, controlnet]. Recent studies have explored distilling knowledge from pre-trained diffusion models for text-to-3D generation through optimizing a differentiable 3D representation with image priors [poole2022dreamfusion, SJC, chen2023fantasia3d, lin2023magic3d, wang2023prolificdreamer, huang2023dreamtime]. However, this line of work suffers from optimization efficiency, resulting in blurriness and the Janus problem in the generated 3D models. This arises due to the lack of 3D-awareness in pre-trained text-to-image models.

### 2.2 Novel View Synthesis with Diffusion Models

Novel view synthesis (NVS) requires generating an object’s unobserved geometry and texture, which is a prerequisite capability for 3D generation. Generative models, such as diffusion models [latentdiffusion, ddpm] are well-suited for NVS due to their impressive generation capabilities. This is especially advantageous when dealing with sparse input views, to the extreme with a single image. Recently, Zero123 [liu2023zero] proposes to perform zero-shot open-set NVS by fine-tuning a pre-trained text-to-image diffusion model on multi-view renderings of diverse 3D data [objaverse]. However, this fine-tuning process is computationally expensive and data-demanding. It remains challenging to collect large amounts of high-quality 3D data. As a result, Zero123 [liu2023zero] often generates inconsistent poses compared to the ground truth. Subsequent efforts  [liu2023syncdreamer, weng2023consistent123, shi2023mvdream, long2023wonder3d] aim to resolve view inconsistency by facilitating information propagation across different views which naturally require generating multiple novel views concurrently. Typically, these methods construct training data by fixing an evenly spaced set of camera poses with fixed elevation, and stacking multiple views together to train attention modules that model multi-view dependencies. While these models excel in generating more multi-view consistent novel views, they face challenges in generalizing to arbitrary camera poses and remain 3D-data demanding.

## 3 Method

Our goal is to learn a novel view synthesis (NVS) model that alleviates the “inconsistency” problem by employing the idea of closed-loop transcription [dai2022ctrl, ma2022principles]. To this end, we formulate the current state-of-the-art (SOTA) methods for single-image NVS [liu2023zero, shi2023zero123++, weng2023consistent123] as an open-loop auto-encoder framework (Section 3.1 and Figure LABEL:fig:diagram-a). Then, we extend the open-loop to closed-loop to learn the decoder/generator with the help of the closed-loop transcription (CTRL) framework (Section 3.2 and Figure LABEL:fig:diagram-b). Finally, the choice of training strategy and denoise scheduler are discussed in Section 3.2.

### 3.1 Current NVS Methods and their Caveats

#### The Formulation of Current NVS Methods.

Given a single RGB image (reference view) of an object, the goal of single image NVS is to synthesize an image of the object from a different camera viewpoint (target view) given the relative camera transformation. Although various works improve the NVS task in different aspects, we could unify their formulation as an open-loop auto-encoder as follows.

Suppose the random variables $\bm{X}$, $\Delta\bm{RT}$, and $\bm{X}_{tg}$ denote the reference image, the relative camera extrinsic, and the target image, respectively. The dataset $\mathcal{D}$ comprising $n$ triplets represented as $\mathcal{D}=\left\{\left(\bm{X}^{i},\Delta\bm{RT}^{i},\bm{X}^{i}_{tg}\right)
\right\}_{i=1}^{n}$, where the given $n$ i.i.d. samples $\bm{X}^{1},\dots,\bm{X}^{n}\sim\bm{X}$, $\Delta\bm{RT}^{1},\dots,\Delta\bm{RT}^{n}\sim\Delta\bm{RT}$, and $\bm{X}_{tg}^{1},\dots,\bm{X}_{tg}^{n}\sim\bm{X}_{tg}$. Specifically, $\Delta\bm{RT}^{i}=(\Delta\bm{R}^{i},\Delta\bm{T}^{i})$, where $\Delta\bm{R}^{i}\in{\mathbb{R}}^{4\times 3}$ and $\Delta\bm{T}^{i}\in{\mathbb{R}}^{4}$ respectively represent the relative camera rotation and translation. The current single image NVS methods aim to learn a model $h$ that synthesizes new image $\hat{\bm{X}}_{tg}$ from the reference image $\bm{X}$ under the relative camera transformation $\Delta\bm{RT}$, i.e,

$$ $\displaystyle\hat{\bm{X}}_{tg}=h(\bm{X},\Delta\bm{RT}),$ (1) $$

where $\hat{\bm{X}}_{tg}$ is the synthesized target view image. The goal is for $\hat{\bm{X}}_{tg}$ to be perceptually similar to the true target view $\bm{X}_{tg}$.

The model $h$ in these methods [liu2023zero, shi2023zero123++, liu2023syncdreamer] is composited by an encoder $f:\bm{X}\mapsto\bm{Z}$, parameterized by $\theta$, and a decoder $g:(\bm{Z},\Delta\bm{RT})\mapsto\hat{\bm{X}}_{tg}$, parameterized by $\eta$, i.e., $h(\bm{X},\Delta\bm{RT})=g(f(\bm{X}),\Delta\bm{RT})$. Specifically, the encoder $f$ is commonly chosen to be the ViT [dosovitskiy2020image] model of a pre-trained CLIP [radford2021learning] and kept frozen during the NVS training. Let $\bm{Z}=f(\bm{X};\theta)\in{\mathbb{R}}^{d}$, where $\bm{Z}$ is the class feature (cls token) of ViT.

The decoder $g$ is commonly implemented as a process of multi-step conditional latent denoising [liu2023zero] with VAE encoder/decoder (as shown in the dashed box of Figure LABEL:fig:diagram-a). First, the process is conditioned on a transformation $\bm{e}=\psi(\bm{Z},\Delta\bm{RT})$ of the feature $\bm{Z}$ and relative camera extrinsic $\Delta\bm{RT}$, where $\psi$ is a linear layer. Then, for step $t\in\{0,\dots,T\}$, the noise $\hat{\mathbf{\epsilon}}$ is predicted, using an U-net $\phi$, based on the predicted target view image at time step $t$, and the condition $\bm{e}$, i.e., $\hat{\mathbf{\epsilon}}=\phi(\hat{\bm{x}}_{tg,t},\bm{e},t)$.
$\hat{\bm{x}}_{tg,t}$ is the latent target view feature $\bm{x}_{tg}$ plus $t$ steps noise where $\bm{x}_{tg}$ is latent feature of target view image $\bm{X}_{tg}$ by VAE encoder.

The multi-step denoising is performed through a denoiser $\mathcal{S}(\cdot)$, based on denoising diffusion implicit models (DDIM) [song2020denoising]. The predicted target view image at time step $t-1$ is generated by the denoiser, i.e., $\hat{\bm{x}}_{tg,t-1}=\mathcal{S}(\hat{\bm{x}}_{tg,t},\hat{\mathbf{\epsilon}},t)$ (^1^11$\mathcal{S}(\hat{\bm{x}}_{tg,t},\hat{\mathbf{\epsilon}},t)=\frac{1}{\sqrt{\bar {\alpha}_{t}}}(\hat{\bm{x}}_{tg,t}-\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t }}}\hat{\mathbf{\epsilon}})$ where $\bar{\alpha}_{t}=\prod_{i=1}^{t}\alpha_{i}$ and $\alpha_{t}=1-\beta_{t}$, $\beta_{t}$ is a pre-defined variance of $t$-th step.). For the initial step $t=t_{\infty}$, the $\hat{\bm{x}}_{tg,t_{\infty}}$ is randomly sampled from isotropic Gaussian. Then, after $t_{\infty}$ denoising steps, we get $\hat{\bm{x}}_{tg,0}=\hat{\bm{x}}_{tg}$. Finally, the denoised latent feature $\hat{\bm{x}}_{tg}$ was lifted to generate target view image $\hat{\bm{X}}_{tg}$ through the VAE decoder.

We denote random variable $\hat{\bm{X}}_{tg}\doteq g(\bm{Z},\Delta\bm{RT})$ as the decoded from Gaussian noise $\hat{\bm{x}}_{tg,t_{\infty}}$ through $t_{\infty}$ denoising steps according to $\mathcal{S}$ conditioned on $\bm{Z}$ and $\bm{\Delta RT}$. Current single-image NVS methods can be summarized as the following open-loop framework,

$$ $\bm{X}\xrightarrow{f(\bm{X};\theta)\hskip 5.69054pt}\bm{Z}\xrightarrow{\hskip 5 .69054ptg(\bm{Z},\Delta\bm{RT};\eta)}\hat{\bm{X}}_{tg}.$ (2) $$

Such diffusion-based NVS methods typically adopt a standard diffusion training strategy by supervising single-step denoising results with a score-matching loss, i.e.,

$$ $\mathop{\min}_{\eta}\mathbb{E}\|\phi(\hat{\bm{x}}_{tg,t},\psi(\bm{Z},\Delta\bm {RT}),t)-\mathbf{\epsilon}\|_{2}^{2},$ (3) $$

where the expectation is taken over the encoded feature $\bm{Z}$, timestep $t$, relative camera extrinsic $\Delta\bm{RT}$, and randomly sampled Gaussian noise $\mathbf{\epsilon}$.

#### The Caveats of Current NVS Methods.

While the showcased results in most single-image NVS works appear impressive, we make a critical observation previously overlooked: the generated novel views often lack consistency with the ground truth, even when evaluated on the training data, as exemplified in Figure LABEL:fig:fail_case_of_zero123. This inconsistency problem is rooted in the training strategy of diffusion models which separately trains the denoiser at different noise levels using solely score matching loss without any constraints on the whole denoising process (from Gaussian noise $\hat{\bm{x}}_{tg,t_{\infty}}$ to the clean image $\hat{\bm{X}}_{tg}$). This strategy significantly improves the diversity of the generated images but neglects the consistency required in NVS models. In this work, we explore ways to enforce consistency between the generated views and ground truth to enhance the capability of NVS models.

### 3.2 Ctrl123: Consistency via a Closed-Loop Framework

A straightforward method to enforce sample-wise consistency is to add a loss such as Mean Square Error (MSE) in the pixel space between the generated view $\hat{\bm{X}}_{tg}$ and ground truth $\bm{X}_{tg}$. However, through extensive experiments, a direct loss in pixel space often suffers training difficulties leading to training collapse (See Appendix D and Table
9 for more details).

The recently proposed closed-loop transcription (CTRL) framework [dai2022ctrl] is naturally suited to solve this problem. The difference between $\bm{X}$ and $\hat{\bm{X}}$ can be measured through the distance between their corresponding features $\bm{Z}=f(\bm{X};\theta)$ and $\hat{\bm{Z}}=f(\hat{\bm{X}};\theta)$ mapped through the same encoder, i.e.,

$$ $\bm{X}\xrightarrow{\hskip 5.69054ptf(\bm{X};\theta)\hskip 5.69054pt}\bm{Z} \xrightarrow{\hskip 5.69054ptg(\bm{Z};\eta)\hskip 5.69054pt}\hat{\bm{X}} \xrightarrow{\hskip 5.69054ptf(\hat{\bm{X}};\theta)\hskip 5.69054pt}\ \hat{\bm {Z}}.$ (4) $$

Inspired by CTRL, we apply this idea to the current NVS model that measures the difference between $\bm{X}_{tg}$ and the generated $\hat{\bm{X}}_{tg}$ in the CLIP feature space. In other words, we apply it to framework 2 to get the following new framework:

$$ $\bm{X}\xrightarrow{\hskip 5.69054ptf(\bm{X};\theta)\hskip 5.69054pt}\bm{Z} \xrightarrow{\hskip 5.69054ptg(\bm{Z},\Delta\bm{RT};\eta)\hskip 5.69054pt}\hat {\bm{X}}_{tg}\xrightarrow{\hskip 5.69054ptf(\hat{\bm{X}}_{tg};\theta)\hskip 5. 69054pt}\ \hat{\bm{Z}}_{tg}.$ (5) $$

Different from the direct CTRL formulation equation 4, we can not directly calculate the loss between $\bm{Z}$ and $\hat{\bm{Z}}_{tg}$ since they are the features of different views. Hence, we add the feature $\bm{Z}_{tg}$ of the ground truth target view $\bm{X}_{tg}$ as the following,

$$ $\displaystyle\bm{X}\xrightarrow{f(\bm{X};\theta)\hskip 5.69054pt}\bm{Z} \xrightarrow{\hskip 5.69054ptg(\bm{Z},\Delta\bm{RT};\eta)}\hat{\bm{X}}_{tg}$ $\displaystyle\xrightarrow{f(\hat{\bm{X}}_{tg};\theta)\hskip 5.69054pt}\hat{\bm {Z}}_{tg},$ (6) $\displaystyle\bm{X}_{tg}$ $\displaystyle\xrightarrow{f(\bm{X}_{tg};\theta)\hskip 5.69054pt}{\bm{Z}}_{tg}.$ $$

Now we can measure the $\bm{Z}_{tg}$ and $\hat{\bm{Z}}_{tg}$ through Mean Squared Error,

$$ $\mathop{\min}_{\eta}\|\bm{Z}_{tg}-\hat{\bm{Z}}_{tg}\|_{2}^{2}.$ (7) $$

Building upon the previously established definition of $\bm{Z}$, we identify it as the class feature within the outputs of the Vision Transformer (ViT) model. Revisiting the ViT model, its outputs are bifurcated into class features and patch features. The class features capture high-level information, whereas the patch features are indicative of low-level information. Our methodology prioritizes the analysis of patch features due to their richer and more detailed informational content, which significantly aligns with ground truth data and enhances consistency. This preference is empirically validated in our ablation study, which demonstrates superior performance of patch features over class features (refer to Appendix D and Table 10 for detailed results).

To accommodate this distinction, we introduce a revised notation for the encoder outputs, denoted as $\left[\bm{Z}_{c},\bm{Z}_{p}\right]=f(\mathbf{X},\theta)$ and $\left[\bm{Z}_{tg,c},\bm{Z}_{tg,p}\right]=f(\mathbf{X}_{tg},\theta)$, where $\bm{Z}_{c}$ and $\bm{Z}_{tg,c}$ represent the high-level class features, and $\bm{Z}_{p}$ and $\bm{Z}_{tg,p}$ correspond to the low-level patch features. Re-writing the framework 6 with the re-defined notation as the following,

$$ $\displaystyle\bm{X}\xrightarrow{f(\bm{X};\theta)\hskip 2.84526pt}\bm{Z}_{c} \xrightarrow{\hskip 2.84526ptg(\bm{Z}_{c},\Delta\bm{RT};\eta)}\hat{\bm{X}}_{tg}$ $\displaystyle\xrightarrow{f(\hat{\bm{X}}_{tg};\theta)\hskip 5.69054pt}\hat{\bm {Z}}_{tg,p}$ (8) $\displaystyle\bm{X}_{tg}$ $\displaystyle\xrightarrow{f(\bm{X}_{tg};\theta)\hskip 5.69054pt}{\bm{Z}}_{tg,p},$ $$

where $\bm{Z}_{c}$ is the class feature of reference views $\bm{X}$ and ${\bm{Z}}_{tg,p}$, $\hat{\bm{Z}}_{tg,p}$ are the patch features of target view ground truth $\bm{X}_{tg}$ and generated target view $\hat{\bm{X}}_{tg}$, and we can measure the difference between $\bm{Z}_{tg,p}$ and $\hat{\bm{Z}}_{tg,p}$ through Mean Squared Error,

$$ $\mathop{\min}_{\eta}\|\bm{Z}_{tg,p}-\hat{\bm{Z}}_{tg,p}\|_{2}^{2}.$ (9) $$

#### Alternative Training Strategy

Instead of directly applying 9 as a regularization term over the score-matching loss, we propose an alternative training strategy which is ablated in Table 7. We perform fine-tuning on pre-trained Zero123 [liu2023zero] (denoted as Zero123 in the rest part of our paper).
One round of training involves 1) $m$ iterations of closed-loop training via CTRL framework as depicted in 8 with supervision loss according to 9. We call this process CL training; 2) $n$ iterations of standard diffusion model fine-tuning with score-matching loss. We call this process SM training for short. Our experiment results show that more rounds of such training could continuously improve NVS performance. Such an alternative training strategy is more efficient and practical as the hyper-parameters of CL training and SM training are vastly different. It is time-consuming and energy-inefficient to grid search a sweet spot of hyper-parameters for both CL and SM training. In section 4.5 we demonstrate the effectiveness of this alternative training strategy.

## 4 Experiments

We evaluate Ctrl123 on the tasks of single-view NVS and 3D reconstruction. 3D reconstruction is a more challenging task that requires strong multiview consistency. In Section 4.1, we describe our data collating and the implementation details. We introduce the metrics used to measure pose consistency between the generated novel views and the ground truth in Section 4.2, including our introduced metric Angle Accuracy (AA) and Intersection over Union (IoU). We provide in Section 4.3 results on both small-scale and large-scale experiments, trained on a subset of the curated dataset and the full dataset, respectively. Results on 3D reconstruction are provided in Section 4.4. Ablations on the design choices are presented in Section 4.5.

**Table 1: Ablation study on the number of rounds for alternative training strategy when trained on a subset of the curated dataset, measured with 4 metrics: KID($\downarrow$), PSNR($\uparrow$), AA${}^{x^{\circ}}$($\uparrow$), and IoU^x($\uparrow$). While Zero123-small shows an unnoticed problem when trained on the subset dataset - Zero123-small generates pose-inconsistent novel views, Ctrl123-small produces significantly more consistent results which continue to improve with more rounds of alternative training. Please refer to Appendix A.1 for more visualization results.**
| Method vs. | Training | Image Quality | Alignment |  |
| --- | --- | --- | --- | --- |
| Metrics | steps | (MegaPose/DIS) |  |  |
| Zero123-small | 20,000 | KID:0.0254 | AA${}^{15^{\circ}}$:22.62% |  |
| PSNR:19.3839 | IoU^0.7 :30.93% |  |  |  |
| Ctrl123-small | 10,000 | KID:0.0087 | AA${}^{15^{\circ}}$:32.02%  (+9.40%) |  |
| (1 round) | PSNR:23.6080 | IoU^0.7 :55.01%  (+24.08%) |  |  |
| Ctrl123-small | 20,000 | KID:0.0067 | AA${}^{15^{\circ}}$:57.78%  (+35.16%) |  |
| (2 rounds) | PSNR:26.5348 | IoU^0.7 :73.44%  (+42.51%) |  |  |

### 4.1 Settings

#### Dataset and Baseline.

We use Zero123 as our baseline, and utilize the public 3D dataset - Objaverse [objaverse] which contains around 800K diverse 3D models created by artists. For training efficiency and quality, we apply the data curation similar to [li2023instant3d], eliminating 3D assets with sub-optimal geometry, low-quality texture, or small sizes. This curation process results in a refined training dataset with approximately 100K 3D models.
All other configurations follow the settings in the literature of baseline (e.g. the number of view sampling, sampling strategy, and camera model, among others).

#### Implementation.

We train Ctrl123 on 100K curated Objaverse 3D models for 2 rounds of alternative training. We initialize Ctrl123 with pre-trained weights of Zero123 and train on 8 A100 GPUs with 80GB memory. For each alternative training round, we conduct 500 steps of closed-loop (CL) training with a total batch size of 320 and a learning rate of $10^{-5}$, followed by $t_{\infty}=1000$ steps of score-matching (SM) training with a total batch size of 1536 and a learning rate of $10^{-4}$, each taking around 2 days. We use gradient accumulation to increase the training batch size, 160 for CL training and 2 for SM training respectively. During model training, we use the Adam optimizer [Kingma2014AdamAM] with $\beta_{1}=0.9$ and $\beta_{2}=0.99$. In all the experiments, we train our model with the 16-bit floating point (fp16) format for efficiency.

**Table 2: Quantitative comparison of Ctrl123 and four baselines, measured with 4 different metrics: PSNR ($\uparrow$), KID ($\downarrow$), AA${}^{x^{\circ}}$($\uparrow$), and IoU^x($\uparrow$), and evaluated on 3 datasets: GSO, RTMV, and 25 randomly sampled objects from the training set.**
| Method | GSO | RTMV | 25 Training Objects |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PSNR | KID | AA${}^{15^{\circ}}$ | IoU^0.7 | PSNR | KID | AA${}^{15^{\circ}}$ | IoU^0.7 | PSNR | KID | AA${}^{15^{\circ}}$ | IoU^0.7 |  |
| Zero123 | 17.2049 | 0.0055 | 14.41% | 47.32% | 8.9123 | 0.0340 | 7.04% | 2.91% | 15.9452 | 0.0043 | 14.28% | 31.86% |
| SyncDreamer [liu2023syncdreamer] | 17.6317 | 0.0077 | 15.94% | 51.65% | / | / | / | / | / | / | / | / |
| Zero123++ [shi2023zero123++] | 18.2247 | 0.0140 | 15.63% | 32.45% | / | / | / | / | / | / | / | / |
| Consistent123 [weng2023consistent123] | 13.8337 | 0.0456 | 3.17% | 29.85% | / | / | / | / | / | / | / | / |
| Ctrl123 | 18.4712 | 0.0045 | 16.96% | 62.11% | 10.8938 | 0.0294 | 11.98% | 12.41% | 18.9398 | 0.0052 | 19.79% | 45.97% |

Figure: Figure 3: Qualitative comparison of NVS generalization capability on GSO (Left) and RTMV (Right) after training on large-scale dataset (100K). More cases can be found in Appendix A.2.
Refer to caption: x2.png

### 4.2 Angle Accuracy (AA) and Intersection over Union (IoU)

The uniqueness of NVS makes the evaluation a multifaceted problem. We cannot only evaluate the model’s performance via traditional image generation benchmarks such as KID, FID, PSNR, and SSIM. As discussed in Sections 1 and 3, it is equally important to evaluate the model’s ability to generate pose “consistent" views. In doing so, we introduce two metrics Angle Accuracy (AA) and Intersection over Union (IoU) that measure pose consistency on both fine-grained and coarse level.

#### Angle Accuracy (AA).

AA is first introduced in [zhou2019neurvps]. For each camera pose predicted in the generated novel view, we calculate the angle difference between the ground truth and the predicted pose. The development of the 6D pose estimation task enables the prediction of 6D pose from a single image with its GT CAD model. In this work, we use MegaPose [labbe2023megapose] which is a single image pose estimator, to estimate the camera pose of the generated view. We then identify the percentage of predictions as AA${}^{x^{\circ}}$ where the angle difference falls within a predefined threshold $x^{\circ}$. To varify the credibility of AA, we calculate multiple AA${}^{x^{\circ}}$ when $x$ is set as 5,10,15,20, which all reflects better pose alignment of Ctrl123. Please refer to Appendix B to see more illustrations for the credibility of our introduced AA. After careful comparison and analysis, we choose AA${}^{15^{\circ}}$ as the final metric to report in the comparison tables. The numerical analysis for the chosen AA${}^{15^{\circ}}$ can be found in Appendix B.

#### Intersection over Union (IoU).

We also employ a segmentation metric IoU [garcia2017review] that captures more coarse aspect of pose consistency. We use DIS [qin2022highly] to predict the mask of the generated view and the ground truth view to measure the IoU. The calculation of IoU is defined by dividing the overlap between the predicted and ground truth mask by the union of them.

For all experiments in this paper, we report KID($\downarrow$), PSNR($\uparrow$), AA${}^{x^{\circ}}$($\uparrow$) and IoU^x($\uparrow$) to compare the high-level quality, low-level details, and pose consistency of the generated novel views among all methods. ($\uparrow$) denotes the higher the better, verse vice for ($\downarrow$)). The PSNR metric is an overall metric of texture and silhouette and it is not linearly related to silhouette. We also introduce IoU metric which evaluates silhouette directly and reflects coarse pose alignment.

Figure: Figure 4: NVS/3D examples using Ctrl123 on images from training set (100K).
Refer to caption: x3.png

### 4.3 Comparison with SOTA

#### Improvement with Multi-rounds Optimization.

First, we evaluate the performance of multi-rounds training strategy of Ctrl123 on a small dataset. The dataset consists of 25 objects from Objaverse, each with 12 randomly sampled views. We first train a baseline (Zero123-small in Table 1), which is initialized with pre-trained weights of Stable Diffusion, and then fine-tuned on the 25 objects by score matching loss for 20,000 iterations with batch size of 96. We train Ctrl123-small following the alternative training strategy as detailed section 3.2. For fair comparison, it is initialized with the checkpoint of Zero123-small at 10,000 iterations. We train for 500 closed-loop iterations and 4,500 score-matching iterations as one round. Therefore, after two rounds of training, Ctrl123-small will accumulate the same number of training iterations as the baseline (20,000).

An interesting problem happens: even if Zero123-small has already converged, its pose alignment is worse than that of Zero123 (based on the comparison of metrics: AA${}^{x^{\circ}}$ and IoU^x), as shown in Appendix D and Table 8. Therefore, The training strategy of Zero123 will cause pose inconsistency, which is more obvious when training on a small dataset. As shown in Table 1, Ctrl123-small solves this problem and significantly improves over Zero123-small both qualitatively and quantitatively (more quantitative results in Appendix D and Figure 6). After 2 rounds of training, Ctrl123-small achieves better pose-alignment.
The PSNR increases by 7.2 points which is almost 2$\times$ better than baseline (^2^22Based on the definition of PSNR, 3 points equals 1$\times$ visual quality improvement [wang2004image]). Ctrl123 also generates more pose-consistent views. On the fine-grained level, AA increases from 22.62% to 57.78%; One a more coarse level, IoU increases from 30.94% to 73.44%.

#### Improvements extend to larger datasets.

Following Zero123 [liu2023zero], we evaluate on GSO [downs2022google] and RTMV [tremblay2022rtmv] for NVS quality in Table 2. We evaluate 20/110 randomly selected objects/scenes for 17/50 sampled views from GSO/RTMV respectively.
We compare with Zero123 on GSO and RTMV , and compare with SyncDreamer [liu2023syncdreamer], Zero123++ [shi2023zero123++] and Consistent123 [weng2023consistent123] on GSO.(^3^33We also evaluate their (Zero123++, SyncDreamer, and Consistent123) performance on RTMV [tremblay2022rtmv] under similar settings and put the results in Appendix A.2 and Table 3. Note that RTMV [tremblay2022rtmv] only offers images of randomly sampled views, which is not fair for these work when evaluating.) Note that we only compare with works under similar settings. Therefore, Wonder3D [long2023wonder3d], which takes RGB images and normal images as training data, is not in our comparing list.
As shown in Table 2, Ctrl123 outperforms baselines in all four metrics (PSNR, KID, AA${}^{x^{\circ}}$, and IoU^x) on evaluation datasets. In addition, Ctrl123 fits better on the subset of training dataset (randomly selected 25 training objects). In particular, Ctrl123 obtains much better pose-consistency (see AA${}^{x^{\circ}}$, and IoU^x). We visualize the differences in Figure 3 for qualitative comparison between Ctrl123 and Zero123. Ctrl123 generates higher quality and more pose-consistent views comparing to the baseline.

Note that the code base we use to train and inference is threestudio [threestudio2023], which select a more efficient but lower quality inference method. Therefore, metrics of Zero123 reported in Table 2 are lower than that reported in the literature of Zero123 (see more detailed comparison of different code base in appendix D and Table 11). The conclusion that Ctrl123 shows better performance than Zero123 is credible because we evaluate Ctrl123 and Zero123 with the same code base. In addition, we reproduce the results of SyncDreamer [liu2023syncdreamer], Zero123++ [shi2023zero123++] and Consistent123 [weng2023consistent123] on the same setting as ours and report them in Table 2.

### 4.4 3D Reconstruction

We also evaluate 3D reconstruction of Ctrl123 on Objaverse instances, as well as randomly selected web images. We use the SDS [poole2022dreamfusion] loss implemented by threestudio [threestudio2023] for 3D reconstruction. We then evaluate qualitatively on the exported mesh from reconstructed 3D (refer to Appendix C for implementation details and Figure 9 shows the comparison of the generated 3D mesh). Ctrl123 exhibits better reconstruction quality with smooth surfaces and detailed geometry.

Figure 5 and 5 show the NVS and 3D reconstruction results on both the training set and images in the wild. Ctrl123 performs higher-fidelity and finer detail 3D reconstruction, and multi-view consistent pose consistent novel view synthesis. Figure 9 shows the NVS/3D mesh comparison with other NVS methods. In comparison, Ctrl123 achieves the best reconstruction quality with smooth surfaces and detailed geometry.

### 4.5 Ablation Study

We conduct ablation studies to verify the importance of multi-step distillation and alternative training strategies (Section 3.2). We sample a subset of Objaverse, 25 objects with 12 randomly sampled views for the ablation studies.

#### Different Training Strategy.

In Section 3.2, the “Simultaneous” implementation of the training strategy involves adding Equation 9 as a regularization term to the score-matching loss of Zero123, with the aim of optimizing both simultaneously. The results in Appendix D and Table 7 shows the “Alternative” strategy is better.

#### Denoise Scheduler.

The number of denoising steps $n$ used in the function $g$ to generate $\hat{\bm{X}}_{tg}$ is crucial for balancing the quality of $\hat{\bm{X}}_{tg}$ against memory consumption. To investigate this, we conducted experiments with different numbers of denoising steps (1, 10, 30, 50) to evaluate their impact on image quality and pose alignment (see more details in Appendix D and Table 6). Ultimately, we select 50 denoising steps during CL training.

#### Different Random Seeds.

For the initial step $t=t_{\infty}$, the $\hat{\bm{x}}_{tg,t_{\infty}}$ is randomly sampled from isotropic Gaussian. Therefore, different seeds lead to different $\hat{\bm{x}}_{tg,t_{\infty}}$. We explore whether different random seeds influence the good pose-consistency performance of Ctrl123, as shown in Appendix D and Table 12. Results show that the influence caused by different random seeds is negligible.

## 5 Conclusion

In this paper, we introduce Ctrl123, a closed-loop transcription-based NVS method that significantly alleviates the problem of pose and appearance inconsistency between the generated view and ground truth in NVS. To quantitatively measure the performance of consistency improved by Ctrl123, we introduce metrics AA and IoU. Through extensive experiments, we show that the closed-loop Ctrl123 significantly improves pose-consistency (and appearance consistency) for NVS, and leads to significantly better 3D reconstruction compared to the current SOTA methods. Note that in [ma2022principles] the closed-loop framework is proposed as a general framework for ensuring consistency. The work of [dai2022ctrl] has shown that the closed-loop formulation is also effective in ensuring consistency for image classes, besides poses. Hence we believe such a closed-loop framework is necessary to ensure consistency in content generation for other attributes, such as relative pose between objects, as well as their shapes and textures etc. We will leave such generalization to future investigation.

## 6 Broader Impacts

Our work applies a general closed-loop framework to generative models for producing 2D images and 3D content with precise desired attributes. However, our work may raise ethical concerns regarding the potential generation of misinformation. It is important to note that we do not introduce new real-world applications, so the potential harms appear to be minimal.

## 7 Limitations

We find an unnoticed problem of current NVS methods and our work solved this problem. Our method requires more computational resources. Nevertheless, our work is an important step to introduce closed-loop framework into diffusion models to enhance generality abilities.

## Appendix A More Qualitative Results

### A.1 Comparison on consistent multi-view generation.

In Figure 6, we showcase more cases comparing the quality and pose consistency between the generated novel views and ground truth. It is evident that Ctrl123-small generates much more pose and appearance consistent views for instances in the training set.

Figure: Figure 6: More qualitative comparison of the quality and pose on the generated views with the ground truth views trained on the subset dataset (25 objects). Comparisons between Zero123-small, Ctrl123-small, and GT.
Refer to caption: x5.png

### A.2 More results generated by Ctrl123 after training on 100K objects.

**Table 3: Quantitative comparison of Ctrl123 and four baselines, measured with 4 different metrics: PSNR ($\uparrow$), KID ($\downarrow$), AA${}^{x^{\circ}}$($\uparrow$), and IoU^x($\uparrow$), and evaluated on 3 datasets: GSO, RTMV, and 25 randomly sampled objects from the training set.**
| Method | GSO | RTMV | 25 Training Objects |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PSNR | KID | AA${}^{15^{\circ}}$ | IoU^0.7 | PSNR | KID | AA${}^{15^{\circ}}$ | IoU^0.7 | PSNR | KID | AA${}^{15^{\circ}}$ | IoU^0.7 |  |
| Zero123 | 17.2049 | 0.0055 | 14.41% | 47.32% | 8.9123 | 0.0340 | 7.04% | 2.91% | 15.9452 | 0.0043 | 14.28% | 31.86% |
| SyncDreamer [liu2023syncdreamer] | 17.6317 | 0.0077 | 15.94% | 51.65% | 9.6344 | 0.0702 | 2.06% | 1.48% | / | / | / | / |
| Zero123++ [shi2023zero123++] | 18.2247 | 0.0140 | 15.63% | 32.45% | / | / | / | / | / | / | / | / |
| Consistent123 [weng2023consistent123] | 13.8337 | 0.0456 | 3.17% | 29.85% | 7.1517 | 0.0857 | 1.35% | 1.06% | / | / | / | / |
| Ctrl123 | 18.4712 | 0.0045 | 16.96% | 62.11% | 10.8938 | 0.0294 | 11.98% | 12.41% | 18.9398 | 0.0052 | 19.79% | 45.97% |

In this subsection, we present more results of Ctrl123 after large-scale training on Objectverse [objaverse]. In Figure 7, we observe that Ctrl123 generates high-quality novel views on images from the training dataset; In Figure 8, Ctrl123 also generates high-quality novel views for data outside the training dataset. We also evaluate performance of Zero123++, SyncDreamer, and Consistent123 on RTMV [tremblay2022rtmv] under our settings in Table 3. Note that RTMV [tremblay2022rtmv] only offers images of randomly sampled views, which is not fair for these work when evaluating.

Figure: Figure 7: Examples of new views generated by Ctrl123 for images in the training dataset (100K Objaverse [objaverse]).
Refer to caption: x6.png

**Table 4: Average angular difference results through Megapose for nearly identical picture pairs.**
| angular difference(through MegaPose) | view-pair1 | view-pair2 | view-pair3 | average |
| --- | --- | --- | --- | --- |
| 3D_Dollhouse_Sink | $3.5833^{\circ}$ | $2.4288^{\circ}$ | $1.0024^{\circ}$ | $5.9619^{\circ}$ |
| JUNGLE_HEIGHT | $2.0103^{\circ}$ | $6.6258^{\circ}$ | $5.2131^{\circ}$ | $5.0780^{\circ}$ |
| Lenovo_Yoga_2_11 | $3.8219^{\circ}$ | $6.0171^{\circ}$ | $2.9925^{\circ}$ | $6.9773^{\circ}$ |
| Shark | $5.2212^{\circ}$ | $3.9884^{\circ}$ | $6.0070^{\circ}$ | $5.2961^{\circ}$ |
| Sonny_School_Bus | $6.9927^{\circ}$ | $4.5645^{\circ}$ | $15.1978^{\circ}$ | $7.4337^{\circ}$ |

**Table 5: AA values for different thresholds of multi-rounds Ctrl123 when trained on a subset of the curated datase.**
| Method | AA${}^{5^{\circ}}$ | AA${}^{10^{\circ}}$ | AA${}^{15^{\circ}}$ | AA${}^{20^{\circ}}$ | AA(Average) |
| --- | --- | --- | --- | --- | --- |
| Zero123-small | 6.29% | 12.55% | 22.62% | 32.20% | 18.42% |
| Ctrl123-small | 14.23% | 21.25% | 32.02% | 41.89% | 27.35% |
| (1 round) |  |  |  |  |  |
| Ctrl123-small | 30.41% | 39.34% | 57.78% | 69.51% | 49.26% |
| (2 round) |  |  |  |  |  |

Figure: Figure 8: Examples of using Ctrl123 to generate new views for images in the GSO dataset[downs2022google].
Refer to caption: x7.png

## Appendix B The credibility of our introduced AA metric

### B.1 Precise single image pose estimation by MegaPose

To evaluate the accuracy of pose estimation through Megapose [labbe2023megapose], we randomly select 5 objects from GSO dataset [downs2022google] and randomly sampled 12 view pairs, ensuring that the angular difference between each pair is less than 1° but not equal to 0 (to avoid identical view pairs). Subsequently, we utilized MegaPose [labbe2023megapose] to estimate the average angular difference for each object. For brevity, Table 4 only displays results for three cases and the average results for 12 cases. In Table 4, Only one pair (view-pair3 of Sonny_School_Bus) results in an error exceeding 15°, while the average error for each object is less than 7.5° (half of 15°). These findings prove the credibility of MegaPose [labbe2023megapose] and our setting for the predefined threshold 15° of our introduced AA metric.

### B.2 Our setting for the predefined threshold of our proposed AA metric

One may concern that AA of Ctrl123 is only better than that of Zero123 under a certain threshold. To eliminate this type of concern, we select 4 thresholds (5, 10, 15, and 20) and computed the corresponding AA values. Table 5 shows results for the experiment in Paragraph Improvement with Multi-rounds Optimization of Section 4.3. Results in Table 5 proves the credibility of AA when the threshold is set as $15^{\circ}$.

**Table 6: Ablation study on the number of denoise steps for $\hat{\bm{X}}_{tg}$ generation.**
| Method vs. | Image Quality | Alignment(MegaPose/DIS) |
| --- | --- | --- |
| Metrics |  |  |
| Zero123 | KID :0.0254 | AA${}^{15^{\circ}}$:22.62% |
| PSNR :19.3839 | IoU^0.7:30.93% |  |
| Ctrl123 | KID:0.0194 | AA${}^{15^{\circ}}$:23.56% |
| (1 denoise steps) | PSNR :20.1593 | IoU^0.7:32.18% |
| Ctrl123 | KID:0.0102 | AA${}^{15^{\circ}}$:25.71% |
| (10 denoise steps) | PSNR:22.5739 | IoU^0.7:38.97% |
| Ctrl123 | KID:0.0095 | AA${}^{15^{\circ}}$:29.47% |
| (30 denoise steps) | PSNR:23.1165 | IoU^0.7:49.32% |
| Ctrl123 | KID: 0.0087 | AA${}^{15^{\circ}}$:32.02% |
| (50 denoise steps) | PSNR:23.6080 | IoU^0.7: 55.01% |

## Appendix C Implementation Details for Image-to-3D Generation

Figure: Figure 9: Qualitative comparison of 3D reconstruction from single view images with different methods (Ctrl123 vs Zero123).
Refer to caption: x8.png

In Section 4.4, we demonstrate the enhancement of 3D reconstruction quality by substituting Zero123 with Ctrl123. All results are obtained using the SDS [poole2022dreamfusion] loss implemented by threestudio [threestudio2023]. In threestudio, we input one image and train a NeRF[Mildenhall2020NeRFRS] with 800 training steps. For each training step, we set the noise scale from 0.4 to 0.85. We employ the Adam optimizer[Kingma2014AdamAM] with $lr=0.01$, $\beta_{1}=0.9$ and $\beta_{2}=0.99$. After training, in order to get more detailed models, we use marching cube method [1987Marching] to export NeRF to mesh.

## Appendix D More Ablation Study

**Table 7: Ablation study on the optimization strategies.**
| Strategy | Image Quality | Alignment(MegaPose/DIS) |
| --- | --- | --- |
| Simultaneous | KID:0.0218 | AA${}^{15^{\circ}}$:25.21% |
| PSNR:17.9643 | IoU^0.7:35.41% |  |
| Alternative | KID:0.0087 | AA${}^{15^{\circ}}$:32.02% |
| PSNR:23.6080 | IoU^0.7: 55.01% |  |

#### Ablation study on denoising steps when CL training.

It is slow to generate a sample with DDPM [ddpm] by following the Markov chain of the reverse diffusion process, as $t_{\infty}$ can as many as a few thousand steps. One simple way to accelerate the process is to run a strided sampling schedule (DDIM) [song2020denoising] with every $\lceil t_{\infty}/n\rceil$ steps to reduce the process from $t_{\infty}$ to $n$ steps. The results, presented in Table 6, indicate that the pose alignment improves as the number of denoising steps increases while image quality improvement stops at the 30 steps. We choose denoising step 50 since the pose alignment is the property that we care the most.

#### Simultaneous vs Alternative training strategy.

In Table 7, the row “Alternative” is the results of the row “Ctrl123-small(1 round)” in Table 1. We implement “Simultaneous” by changing the loss of the experiment “Ctrl123-small(1 round)” from the alternative version to the simultaneous version. All other settings like learning rate, optimizer etc are the same. The results in Table 7 shows the “Alternative” strategy is better. Note that we also try adjusting related hyper-parameters to increase the performance of the simultaneous version training. However, all results show bad performance.

**Table 8: Ablation study on the number of rounds for alternative training strategy when trained on a subset of the curated dataset, measured with 4 metrics: KID($\downarrow$), PSNR($\uparrow$), AA${}^{x^{\circ}}$($\uparrow$), and IoU^x($\uparrow$).**
| Method vs. | Training | Image Quality | Alignment |  |
| --- | --- | --- | --- | --- |
| Metrics | steps | (MegaPose/DIS) |  |  |
| Zero123-small | 20,000 | KID:0.0254 | AA${}^{15^{\circ}}$:22.62% |  |
| PSNR:19.3839 | IoU^0.7 :30.93% |  |  |  |
| Zero123 | 10,000 | KID: 0.0041 | AA${}^{15^{\circ}}$:26.40% |  |
| PSNR:19.3163 | IoU^0.7 :55.35% |  |  |  |

#### Zero123 vs Zero123-small.

As shown in Table 8, we evaluate the performance of Zero123 and Zero123-small on the same 25 objects. Based on that Zero123-small has already converged, Zero123-small shows worse pose-inconsistency (based on the comparison of metrics: AA${}^{x^{\circ}}$ and IoU^x).

Figure: Figure 10: Qualitative comparison on the difference type consistency loss (Pixel space vs Latent/feature space).
Refer to caption: extracted/5684613/Figs/ablation-pixel-vs-feat3.png

**Table 9: Ablation study on different types of consistency losses (pixel space vs latent/feature space).**
| Consistency | Image Quality | Alignment(MegaPose/DIS) |
| --- | --- | --- |
| Type |  |  |
| Pixels | KID:0.1490 | AA${}^{15^{\circ}}$:4.61% |
| PSNR:10.6905 | IoU^0.7:21.49% |  |
| Feature | KID:0.0301 | AA${}^{15^{\circ}}$:29.75% |
| PSNR:18.7103 | IoU^0.7:48.19% |  |

**Table 10: Comparsion of the closed-loop loss on class feature vs patch feature.**
| Feature | Image Quality | Alignment(MegaPose/DIS) |
| --- | --- | --- |
| Type |  |  |
| Class Features | KID:0.0677 | AA${}^{15^{\circ}}$:18.40% |
| PSNR:14.5749 | IoU^0.7:32.44% |  |
| Patch Features | KID:0.0301 | AA${}^{15^{\circ}}$:29.75% |
| PSNR:18.7103 | IoU^0.7:48.19% |  |

#### Pixel space vs Latent space.

To verify the effectiveness of the closed-loop framework, we conducted an ablation study that applies mean square error (MSE) directly between $\hat{\bm{X}}_{tg}$ and $\bm{X}_{tg}$. As shown in Table 9 and Figure 10, enforcing MSE directly in pixel space does not lead to the expected convergence and results in divergence. This highlights the challenges in achieving consistency in different representation spaces.

#### Closed-loop loss on class features vs patch features.

As discussed in section 3.2, the choice of closed-loop loss on the class feature $\bm{Z}_{tg,c}$ or the patch feature $\bm{Z}_{tg,p}$ is important for the method. Hence, we conduct the ablation study to experiment with the different choices of class features and patch features. As shown in Table 10, loss on the patch features could improve the image quality and alignment better than that on the class feature.

**Table 11: Comparsion of Zero123 performance with the different code base: threestudio [threestudio2023] vs zero123[liu2023zero]. The evaluation setting follows Table 2**
| Code Base | Zero123 | threestudio [threestudio2023] |
| --- | --- | --- |
| PSNR | 18.1865 | 17.1495 |

**Table 12: Comparsion of Ctrl123 performance with the different random seeds. The evaluation setting follows Table 2**
| random seed | 35 | 40 | 45 | 50 | 55 |
| --- | --- | --- | --- | --- | --- |
| PSNR | 18.4531 | 18.4767 | 18.4712 | 18.3951 | 18.4106 |

#### Different code base.

In our work, we use threestudio [threestudio2023] to conduct all experiments. Note that threestudio uses a more efficient but worse quality inference method, which leads to lower metrics. Table 11 presents the image quality of generated novel views of Zero123 with different code base.

#### Errors on different random seeds.

Note that different random seeds will cause the difference of $\hat{\bm{X}}_{t_{\infty}}$, which will lead to the variance of generated novel views. Table 12 illustrates PSNR of novel views generated by Ctrl123 with different random seeds. Results presented in Table 12 prove that random seed doesn’t influence our claim of Ctrl123.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: I don't have any questions
soundness: 3
strengths: The experiments show that the method improves the performance measures chosen, and that the proposed method generates images that depict more consistent geometry than do the images generated by Zero123.

The paper offers the following insights which are of interest:
- path features are more informative than class features in training new view synthesis models
- applying an MSE-based loss in the latent space of an autencoder leads to increasing vulnerability to training collapse
- Section 3.1 provides a good explanation of current NVS methods
summary: The paper proposes a method labelled Ctrl123 that is an extension of the Zero123 model for new view synthesis.  The motivation is to improve the accuracy with which the resulting generated images reflect the shape of a viewed object viewed from a particular angle, and to improve the generality of camera placement for the new image. Specifically the authors seek to encourage shape and texture consistency between multiple synthetic views of a single generated object, rather than have the model generate different shapes for each new view. They also aim to have the model render images that accurately reflect the impact of a given change in camera pose. The proposed model extends the Zero123 model by finetuning with altered loss functions that reflect their goals. They particularly introduce a closed-loop cost which compares rendered new views against ground truth.
weaknesses: The primary contribution of the paper is a method for finetuning Zero123. The fact that the proposed method applies only to Zero123 means it is of limited significance, or interest to the rest of the field. 

The change to Zero123 is cosmetic rather than fundamental, and is not transferable to other methods.

The presentation of the paper is good and bad.  It is well laid out, and effort has definitely been applied to survey the relevant literature. The English and the mathematics are very difficult to read, however, to the point where the Introduction is hard to parse.  This is particularly true of the literature review in the Introduction which lists many papers but leaves the reader more confused than illuminated. Effort has been made with the maths, but there is no consistency. 

Some indicative examples of the problems with the presentation:
- There are at least 3 problems with the first sentence of the body of the paper: "Recent advancements in novel view synthesis (NVS) have sparked considerable excitepment on 3D generation"
- "Although the modified task settings, like generating fixed-view multiple views"
- The first three mathematical quantities introduced are \bf{X}, \bf{\Delta R T}, and \bd{X}_{tg}. These are latter explained to be an image, a tuple of matrices, and an image, respectively, despite the fact that they have the same notation. Using a symbol to mean one thing, then the same symbol with a subscript to mean something different is challenging.
- Line 211 introduces a decoder 'g()' as being parameterised by \eta, despite there being no \eta shown.
- Equation 3 minimizes over \eta^* despite the fact that this variable is not a parameter to the expression to be minimized
- g() has variable numbers of parameters, and none of them are \eta^*

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Out of curiosity, what are the computational and memory costs of optimizing through the full denoising steps? Was any heavy engineering work required to make training efficient?
2. Have the authors considered using LPIPS instead of MSE in the CLIP feature space? Since GRM, LGM, GS-LRM both uses LPIPS for better visual quality. My feeling here is that the CLIP loss in this paper is similar to a LPIPS loss. 
3. The authors report that alternating between closed-loop and diffusion training produces the best results. Have you tested performance when using only closed-loop training (e.g., no diffusion training after 500 steps of closed-loop training)? Additionally, what is the performance drop if closed-loop training is removed and the saved computational resources are allocated to additional diffusion training, as in the setup for Table 2?
soundness: 3
strengths: 1. The author achieves state-of-the-art results on three common benchmarks! 
2. The author designs Angle Accuracy and Mask IoU to closely evaluate the 3D consistent, which is interesting. 
3. The authors conduct extensive ablations on the design space of the closed-loop reconstruction loss, investigating factors such as the number of denoising steps, the feature extractor (VAE vs. CLIP), and the effectiveness of simultaneous vs. alternating training.
summary: This paper addresses the problem of object-centric novel view synthesis from a single input image. It builds upon a pretrained Zero-123 model and applies rounds of mixed closed-loop training and standard diffusion training on the Objaverse dataset. In closed-loop training, the novel view rendering loss is optimized in the CLIP feature space over 50 denoising steps. The authors first demonstrate the effectiveness of closed-loop training through an overfitting experiment on 25 objects, showing significantly improved PSNR scores. They then compare their approach with recent state-of-the-art methods on GSO, RTMV, and OmniObjects3D, showing enhanced performance.
weaknesses: 1. The proposed methods is very computationally heavy, back-propogating from 50 steps of network evaluations. Can the authors provide more details about the speed and memory cost of the proposed closed-loop training?

2. The methods sections is not writting very clearly (e.g. section 3.2)! A few concise paragraphs presenting the loss function and a diagram illustrating backpropagation through 50 denoising steps would improve readability. Since I am not familar with the Closed-Loop paper, (cited in line 258), and it takes me a long time to understand section 3.2. 

3. I recommend the authors cite relevant works on alignment for diffusion models, such as "Directly Fine-Tuning Diffusion Models on Differentiable Rewards." The method in this paper is conceptually similar, with MSE in the CLIP feature space serving as a reward signal.

4. Figure-2 has formatting errors.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: Mentioned in the weaknesses section
soundness: 3
strengths: - Ctrl123 effectively improves multi-view consistency in novel view synthesis while maintaining the flexibility to generate arbitrary novel views. This addresses a key challenge in existing methods by balancing consistency with generation diversity. 
- Also, the training strategy of Ctrl123 demonstrates improvements in quantitative metrics, such as PSNR and SSIM, as well as rotation accuracy metrics (AA and IoU). These results show that Ctrl123 not only aligns better with ground truth images but also surpasses the performance of prior models like Zero123. 
- Finally, when trained on the large-scale 3D dataset Objaverse, Ctrl123 maintains its performance across different evaluation datasets, showcasing its scalability and robustness in handling diverse 3D data while improving view consistency and rotation accuracy.
summary: The paper proposes a novel multi-view diffusion model named Ctrl123, which significantly improves multi-view consistency in novel view synthesis (NVS) while retaining the flexibility of generating arbitrary novel views. The method addresses the core issue of excessive diversity in generative models that leads to inconsistencies, by leveraging a closed-loop transcription-based framework that enforces alignment in the CLIP patch feature space. The main contributions include the introduction of the Ctrl123 method, improved training alignment with ground truth, and performance gains shown through experiments on training objects and a large-scale 3D dataset, Objaverse.
weaknesses: - The biggest drawback of this paper lies in the difficulty of understanding how performance and results have improved specifically. The LPIPS metric (used in Zero123++) is not mentioned in the evaluation, and it might be worth considering if the authors could add this to their assessments. Could you include the LPIPS metric in the evaluation and comparison?

- The paper does not cite Zero123++ in the abstract, but it indirectly refers to Zero123++, by mentioning its fundamental limitation by mentioning 'restricted fixed view generation.' Does this mean the performance of Ctrl123 is better than Zero123++? However, there is no comparative analysis with Zero123++, so it is unclear what has improved in this paper. For the rebuttal process, could you present a comparative analysis with Zero123++? If there is a methodological reason for not including a comparative analysis with Zero123++, it would be helpful to explain why such a comparison was not included.

- It is stated that an in-depth analysis was conducted on a set of 25 objects. However, the current results seem to focus solely on simple, specific objects such as toys, dolls, or avatars for NVS. Can the method handle more complex objects for NVS? It seems that Zero123++ tackles more challenging problems compared to this paper. If good results are also produced for more complex sets, I suggest reflecting this in the rebuttal process.

- Ctrl123 claims to enforce alignment using CLIP, but how would it have been if the evaluation included a CLIP score? It would be beneficial to include the CLIP score in the evaluation metrics and discuss how the CLIP score relates to alignment and consistency improvements.

- Due to these points, it is difficult to objectively evaluate the results. While the claim that consistency is achieved can be acknowledged, it is challenging to give a high score due to the issue of objectively evaluating the results.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 1
questions: 1. I suggest that the author draw inspiration from other works on multi-view consistency to enhance the novelty of their approach. For example, [2] improves consistency by optimizing ray casting, while [3] achieves this by creating a smooth camera trajectory. More detailed innovations could be explored in the image generation process, such as adding geometric constraints, applying regularization, optimizing pixel rays, or incorporating more explicit 3D modeling to strengthen consistency, rather than simply adding a CLIP feature loss.
2. How does the author select the small dataset used to test alignment capability in Table 1? Is it representative? A similar question arises in the SOTA comparison in Section 4.3, where only 20 objects are used for quantitative analysis. Using more objects would strengthen the evidence for the method's generalization ability.
3. For 3D reconstruction, the author should include comparative experiments with other methods.

[2]	Seo S, Chang Y, Kwak N. Flipnerf: Flipped reflection rays for few-shot novel view synthesis[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 22883-22893.
[3]	Kwak J, Dong E, Jin Y, et al. Vivid-1-to-3: Novel view synthesis with video diffusion models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 6775-6785.
soundness: 1
strengths: The paper introduces a novel approach, Ctrl123, which uniquely combines closed-loop transcription with CLIP-based patch features to enhance multi-view consistency in novel view synthesis. By addressing the issue of excessive diversity in generative models, this method removes a significant limitation of prior approaches like Zero123, improving applicability to NVS tasks.
summary: This paper proposes Ctrl123, a multi-view diffusion model that improves view consistency in novel view synthesis. By enforcing alignment with ground truth images in the CLIP patch feature space, Ctrl123 reduces excessive diversity in generative models, achieving consistency across views.
weaknesses: 1. This article lacks true innovation; simply adding a CLIP feature loss between ground truth and the predicted image does not constitute a novel contribution, as CLIP loss has already been employed extensively in previous works. For example, [1] has used clip feature space alignment to supervise the reconstruction of the scene, which is exactly the same as your method. 
2. The authors spend too much space in Sec 3.1 to explain the previous work. You should try to simplify the content in Sec 3.1 and use more space to explain your own work. In addition, your formulas 2, 4, 5, 6, and 8 are too similar. You can consider simplifying the repeated content.

[1] Jain A, Tancik M, Abbeel P. Putting nerf on a diet: Semantically consistent few-shot view synthesis[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 5885-5894.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Clarity: The authors should really clarify that the function g() unrolls a large number of denoising steps. This is only clear at the very end of the paper in line 523 and is *crucial* to understand why the proposed approach makes sense. Without unrolling, there is little difference between the denoising objective used in DDPM and the proposed "closed loop" regression loss. Please consider adding this in the method description.

Likewise, the authors may want to elaborate more on why applying the regression loss after unrolling the denoiser tends to make the latter regress to the mean, killing its variance. They should also explain better how this connects to the choice of applying this loss not in VAE space or RGB space, but, further, in CLIP space, in order to "soften" variance killing and hence mode collapse.

The point of unrolling should probably be made quite evident in Figure 2 too, making it an obvious part of the diagram. This is a good place to show how one loss is applied after unrolling.

Novelty is relatively modest, but then again the paper introduces a simple tweak that does result in SoTA performance, which is nice. This is a well explored art and it is not trivial to outperform prior works with a simple change to a relatively old baseline. However, the fact that no videos are given in the supplementary material makes me wonder how well these claims hold up qualitatively. The images in the paper seem fine to me, but showing an animation can really make most inconsistencies jump out.

The authors should discuss a bit more the issue with unrolling in terms of memory impact, and do so in the main paper. 50 steps of unrolling mean that the method can multiply GPU memory fifty-fold, meaning that only ultra-tiny training batches are possible, I presume.

# Minor issues

* You should have a space (or punctuation) after citations
* Zero123 is not styled correctly; the paper is called Zero-1-to-3. You should try to match as closely as possible the styling used by the original authors. Also, it is a "to", not a "two", in the middle.
* Line 212: the "decoder" is usually stochastic, not a function. In general, I found the notation a little confusion as f() is a function, and g() is a distribution, but they are marked in the same way in the diagrams. There is no indication that g() is sampled from.
* Eqs (5) to (9): notation is essentially repeated twice, which seems unnecessary
* Fig (4): you could have included ground truth new viewpoints for reference
soundness: 3
strengths: * The method is a relatively simple tweak on Zero-1-to-3 which seems to improve NVS quality significantly. This is something that may be useful in practice.
* The use of MegaPose for assessing generated viewpoints is nice.
* The proposed tweak is simple conceptually, although it may take significant amounts of GPU memory to implement due to the unrolling.
summary: The paper consider the problem of generating novel views of an object conditioned on an input image and a camera viewpoint change. It starts from the Zero-1-to-3 formulation and adds to it an additional loss, whose purpose is to reduce the diversity of images generated by the model, with the hope to focusing on increasing multi-view consistency. This additional loss simply compares the generated and ground truth target images in the CLIP feature space after unrolling the denoising process.  The resulting new-view generator is compared to prior works using several sensible metrics and is shown to perform convincingly better.
weaknesses: * The method is a relatively simple tweak on Zero-1-to-3 which seems to improve NVS quality significantly. This is something that may be useful in practice.
* The use of MegaPose for assessing generated viewpoints is nice.
* The proposed tweak is simple conceptually (although it may take significant amounts of GPU memory to implement due to the unrolling)
* I could not find any supplementary material, and in particular no videos. This is odd for this paper, as videos are a great way for assessing qualitatively multi-view consistency.

Meta-review: The paper introduces Ctrl123, which aims at improving multi-view consistency in novel view synthesis by aligning generated views with ground truth in the CLIP patch feature space. It extends Zero123 by fine-tuning with modified loss functions, including a closed-loop transcription cost. This approach enhances multi-view consistency while supporting arbitrary camera placements.

The common strengths identified: (1) its innovative use of CLIP-based patch features and closed-loop transcription to enhance multi-view consistency; (2) its flexibility to generate arbitrary views while improving NVS quality; and (3) its extensive experimental validation and new metrics like Angle Accuracy and Mask IoU.

The major weaknesses include: (1) the method is a minor tweak to Zero123, lacking significant novelty or broader applicability; (2) high computational costs due to unrolling in training; (3) unclear presentation, with missing supplementary materials like videos to demonstrate multi-view consistency; and (4) no comparative analysis with Zero123++ or inclusion of key metrics like LPIPS.

The overall ratings are consistently below the broardline and the rebuttal is missing, thus, the ACs agreed on rejection.

Decision: Reject