TOSS: High-quality Text-guided Novel View Synthesis from a Single Image
Authors: Yukai Shi
Jianan Wang
He CAO
Boshi Tang
Xianbiao Qi
Tianyu Yang
Yukun Huang
Shilong Liu
Lei Zhang
Heung-Yeung Shum
Venue: ICLR 2024 poster (2024)
Keywords: novel view synthesis
3D object synthesis
diffusion model
Primary area: generative models

Abstract: In this paper, we present TOSS, which introduces text to the task of novel view synthesis (NVS) from just a single RGB image. 
While Zero123 has demonstrated impressive zero-shot open-set NVS capabilities, it treats NVS as a pure image-to-image translation problem. This approach suffers from the challengingly under-constrained nature of single-view NVS: the process lacks means of explicit user control and often result in implausible NVS generations.
To address this limitation, TOSS uses text as high-level semantic information to constrain the NVS solution space.
TOSS fine-tunes text-to-image Stable Diffusion pre-trained on large-scale text-image pairs and introduces modules specifically tailored to image and camera pose conditioning, as well as dedicated training for pose correctness and preservation of fine details. 
Comprehensive experiments are conducted with results showing that our proposed TOSS outperforms Zero123 with higher-quality NVS results and faster convergence. We further support these results with comprehensive ablations that underscore the effectiveness and potential of 
the introduced semantic guidance and architecture design.

Preprint
TOSS: HIGH-QUALITY TEXT-GUIDED NOVEL VIEW
SYNTHESIS FROM A SINGLE IMAGE
Yukai Shi
Jianan Wang He Ca02.3*
Boshi Tang 1.3
Xianbiao Qi
Tianyu Yang*
Yukun Huang²
Shilong Liu 1.3
Lei Zhang
Heung-Yeree Yeung Shum 1.3
1 Tsinghua University
2 Hong Kong University of Science and Technology International Digital Economy Academy (IDEA)
"o Aurferun sait flaverrco ar
pownger
(a) TOSS increases plausibility with text
(b) TOSS increases controllability with text
Sigure player Neilinn
(c) TOSS generates novel views with higher quality multiview--cossisennyy
Seepiratii
"ayoiiee say
holling socgeee
(d) TOSS generates novel views higher quality (random views)
Zero123
TOSS
TOSS significantly boosts novel view plausibility with additional textual guidance and grants users better controllaiility over concealed parts object (h). We demonstrate higher novel view generation quality and mulliview-comsistencyy with look-around views (c) and random views (d).
Figure 1:
ABSTRACT
In this paper, we present TOSS, which introduces text the task of novel view fromjust
synthesis (NVS) from just a single RGB image. While Zero has demonstrated impressive zero-shot open-set NVS capability, treats NVS as a pure image-to- image translation problem. This approach suffers from the challengingly under- constrained nature of single-view NVS: the process lacks means of explicit user control and often results in implausible NVS generations. To address this limi- tation, TOSS uses text as high-level semantic information to constrain the NVS solution space. TOSS fine-tunes texi-to-image Stable Diffusion pre-trained on large-scale text-image pairs and introduces modules specifically tailored to image and camera pose conditioning, as well as dedicated training for pose correctness and preservation of fine details. Comprehensive experiments are conducted with results showing that our proposed TOSS outperforms Zero123 with more plausi- ble, controllable and multiview-consee connssteetn NVS results. We further support these results with comprehensive ablations that underscore the effectiveness and poten- tial of the introduced semantic guidance and architecture design.
"Equal contribution.
Work done during internship IDEA.

---

Preprint
INTRODUCTION
Novel view synthesis (NVS) from a single RGB image is a severely under-constrained problem, which requires imagining object's unobserved geometry and textures. Despite of its ill-posed nature, single-view NVS aims to change the viewpoint of an object given just single image. There- fore has been regarded as one of the foundation tasks and prerequisite capability for 3D genera- tion (Qian et al. 2023 Liu et al. 2023a), especially when generalizing to any object captured by users or generated by text-to-image models (Rombach al., 2022).
Recent advances in modeling, especially score-based diffusion models (Ho et al. 2020: Sohl-
Dickstein et al.] 2013) have sparked interest in approaching novel view synthesis (NVS) as an -mage translation problem (Chan et al. [0233 Watson et al. 2022). Pioneering work of Zero123 (Liu et al., (223b) extends the generalization capability from a single category to the open world. In concrete, leverages image-to-image model fine-tuned from Stable Diffusion (Rom-l bach et al. 2022) and learns to control relative camera transformation during the generation process by further fine-tuning worth noting that Stable Diffusion is pre-trained on large-scale web data (Schuhmann et al. 2022), which empowers Zerol23
image renderings derived from 3D dataset (Deitke al., 2023). It is
(Liu et al. 2023b) with impressive zero-shot open-set NVS capability.
However, current NVS methods often exhibit shape and texture deformations that lead to implausible semantic meaning (Liu al. 2023c) of their results: e.g. the generated novel view of shark may have two tails as demonstrated in Fig. [
(a). The reason for the poor results is rooted in the ill-posed nature of single-view NVS. where there exist multiple solutions for the unobserved geometry and textures. Previous works constrain the solution space with the input view image and the camera pose. However, the NVS solution space constrained by the two conditions is still too large guarantee a plausible NVS result. Moreover, even when the generation is plausible, users still desire more means to control the NVS process, otherwise the generated result may be visually plausible but fail to fulfill their expectation as shown in Fig. (b).
To address the issue of implausible semantic meaning and add more controllability to NVS, we introduce TOSS, high-quality text-guided NVS model that explicitly leverages textual description as semantic guidance. Before we introduce more algorithm design, it is worth pointing out the benefits of TOSS: 1) Text can impose constraints on the unobservable parts of an object, granting users explicit control to narrow down the NVS solution space as desired; 2) Text can serve as high- level semantic information that complements the low-level details provided by images, which can act as a strong prior for generating plausible NVS results; 3) Users can further refine the focus and granularity of the text description, providing explicit and flexible means to constrain the NVS solution space according to generation requirements. By introducing text information, we observe a significantly improved quality of NVS results. This improvement is attributed to a more informative exploration of the optimization landscape with extra semantic guidance.
More specifically, as we aim to synthesize novel view images under the guidance of text semantics, naturally we capitalize on Stable Diffusion (Rombach et al. 2022) for its remarkable ability to gen- crate diverse images based on text descriptions. To adapt Stable Diffusion for the NVS task, we modify it by introducing novel dense cross attention module to condition the generative process on both the input image's 's feature maps produced by Stable Diffusion's VAE encoder, and the frequency encoding on the relative camera pose transformation. We fine-tune proposed model TOSS on the 3D dataset Objaverse (Deitke et al., 2023). To achieve higher-quality NVS results while main- taining the same training and inference computation cost, specializes two expert diffusion models to focus on novel view pose correctness and preservation of details. Moreover, our improvements are orthogonal to and can be readily combined with advance- ments for text-to-image generation, e.g. better diffusion models, textual inversion (Gal et al. 2022 etc., to further boost the performance.
training strategy that efficiently
The main contributions of this paper can be summarized as:
We introduce TOSS, zero-shot open-set novel view synthesis (NVS) diffusion model that plicitly uses text to narrow down the NVS solution space to yield more plausible, controllable and multiview-consiseee results, given just a single RGB image.
2

---

Preprint
We propose image and camera pose conditioning modules that are meticulously tailored to the text-guided NVS model, alongside training strategies that dedicate specialized denoising experts for novel view pose correctness and preservation of fine details.
Extensive experiments demonstrate that TOSS exhibits superiority to Zero 123 on the tasks of NVS and 3D reconstruction, in terms of plausibility, controllability and thermore, empirically showease the substantial potential inherent to approach, including improved quality through the utilization of higher-capability text-to-image models and techniques such as textual inversion.
Fur-
2 RELATED WORK
2D Diffusion Model. Diffusion models (Ho et al. 2020; Croitoru et al., 2023) have demonstrated re-
markable performance in generating 2D images. Their stable training objectives and enhanced distri- bution mode coverage have enabled diffusion models to surpass previous generative models (Good- fellow 2020p. Numerous studies have explored modeling conditional distributions (Song al.
2021 Dhariwal & Nichol 202I) with diffusion models, such as text-to-image generation (Ramesh
jt al.
2022, Nichol et al. 2021) y and image super-resolution (Saharia et al. 2022b] Ho et al. [0022).
Large effort has been devoted to training text-to-imag models on large-scale web datasets (Chang-
pinyo et al. 2021, Schuhmann et al.) 2021 2022), equipping such pre-trained models (Rombach et al. [022 Ramesh et al. 2022 Saharia et al. 2022a) with rich real-world knowledge. Subsequent
works (Li et al. 2023b; Zhang et al. 2023) consequently leverage powerful pre-traine pre-trained diffusion models for controllable generation, e.g. luyout-to-image pose-to-image and etc.
Novel View Synthesis from a Single Image. Novel view synthesis from a single RGB image re- quires imagining an object's unobserved geometry and texture, which is a precursor to 3D generation despite its ill-posed nature. Regression-basel methods enhance NVS quality with geometry priors, but often with blurry novel view results because of the mean-seeking nature (Yu al.. 2021 Le let al. 2020 Or-El et al., [2022] Park et al., 2019). Generation-based methods leverage generative capabilities produce high-fidelit novel view predictions. GAN-based approaches (Chan et al., 2021 Noguchi et al. 2022) can generate high-resolution novel view images, but suffer from unsta- ble training. Liu al. (2023b): Watson et al. (2022); [Chan et al. (2023) leverage diffusion models' stronger modelling capability and training stability for higher-quality NVS results. Nevertheless, their efficacy remains confined to specific categories. Recently, Zero1 (Liu et al., (023b) pio- neers in zero-shot open-set NVS utilizing pre-trained diffusion models and diverse 3D data (Deitke) ctt al. 2023 Zerol23 frames NVS as a neewpont-conditioon lem. It leverages a pre-trained image translation model and fine-tunes multi-view renderings of 3D dataset. However, the image- mmage-to-imane to-image translation model lacks high-level semantic guidance to constrain the NVS solution space illustrated in Fig. (a-b), often resulting in implausible results.
image-to-image translation prob-
3D Generation with Diffusion Models. Drawing inspiration from 2D generation, recent endeav- ours have been made to train generative models on 3D datasets conditioned on text images (Cheng et al. 2023) Liu et af. (023d Gupta et al., 2023 Zheng et al. 2023). However, they are usually con- fined to specific categories with unsatisfactory generation results. On the other end, optimization- based 3D generation (Poole et al. 20222 Lin et al. [2023 Deng et al. (023) can distill knowledge of pre-trained text-to-image diffusion models 3D model given any text. However, due to lim- ited camera control over 2D pre-trained models, they often suffer from blurriness and the Janus problems, with low generation success rates. Subsequent research has mitigated the problem with additional priors such as condition images (Melas- Kyriazi et al. 2023 Tang et al., (023). depth (Seo) et al. 2023), and human body skeletons (Huang et al. al.. 2023b). Zero 123 (Liu et al. 2023b) empowers pre-trrinne maaee-to-image diffusion model with NVS capabilities, which leads to more accurate control over camera poses of generated images. However, due to the limitation of NVS quality, the 3D generation results still remain sub-optimal.
Remark. TOSS lies in the intersection of the above three directions. It utilizes a 2D Stable Diffu- sion model to target at the zero-shot open-set single-view NVS task. The capability of generating semantically plausible muultvieew-consistee images enables TOSS to be readily integrated in various downstream applications, such as 3D generation and reconstruction.
3

---

Preprint
3 METHOD
We aim solving the issue of NVS models being under-constrained with text guidance. Here provide Fig. (a-b) to exemplify the problem and motivate our approach. We start with discussion some preliminary knowledge in section 3.1 To effectively incorporate text information into framework, we make two efforts. First, in section [.2 formulation and discuss the advantages of employing textual guidance. Subsequently, section 3.3 elaborates on our novel mechanisms to enable text-to-image diffusion models to condition on images and camera poses, that they can fit into our desired model formulation and be used for the NVS task. Fig. 2eg.2iin illlustraees the overall pipeline of our proposed method TOSS.
(.2 we describe our novel text-conditioned model
3.1 PRELIMINARY
Diffusion Models consist of two processes: the forward process of adding noise and the reverse process of denoising. In the forward process, a noisy image is obtained by adding random Gaussian noise ~ (0, , I) to a clean image x ~ p(x) according to the weights ã; determined by timestep E [0, 1000). while in the reverse process, trainable U-Net Eø predicts noise and denoises images at different timesteps: ming Et,,,c ||i (x,, ( After training, one can generate an image & ~ p(x) by starting from an isotropic Gaussian noise and gradually denoising it.
Single-view Novel View Synthesis. Given condition image x, of object, the single-view NVS task is to synthesize an image of the object from another camera viewpoint. Let / denote the camera transformation, i.e., rotation and translation, of the desired viewpoint relative the viewpoint of x The single-view NVS problem can be formulated as X = f(Xc, P), where we intend to learn function -(.-) for conducting NVS.
3.2 TOSS: TEXT-GUIDED NOVEL VIEW SYNTHESIS
T
Yeuyelow logo
Trm
Faeder
Cros Alicatiaa
Kacider
Fenaler
P
Canara Peef
Sec. 3.2
Cras
yellow lego bulllozer'
Output
Figure 2: The pipeline of TOSS (Left) and our conditioning mechanisms (Right).
Formally, the model formulation of TOSS differs from previous works by integrating text descrip- tions into the novel-view synthesis process. TOSS is modeled as:
= f(x, P.
(I)
where T denotes the textual description that aligns with x.- As shown in Fig, 2 information first encoded by frozen text encoder (CLIP) and then integrated into the novel view generation network through cross-attention mechanism.
(Right), the text
Compared to the original NVS, TOSS further constrains the solution space of NVS with extra text information, demonstrating two substantial advantages:
Increasing Plausibility. Text can serve as high-level semantic information that directs NVS gen- eration toward plausible solution that is consistent with the text description. As shown in Fig. (a), being aware that the input image is a shark will ensure that the synthesized novel view should not have tails.

---

Preprint
Increasing Controllability. When generating unobservable parts of object, text impose constraints on the solution space where multiple syntheses may be plausible. As shown in Fig. [ (b), users could explicitly specify the content (e.g. tap flowers) be synthesized within the concealed part of the input image.
Since the problem of novel view synthesis from a single-view image is severely under-constrained, formulate f by adapting text-to-image diffusion model condition both the input view image and the camera pose, in order to account for the non-deterministic nature of the synthesis process. Formally, our 6g takes the form as Eø (x,, L, Xx, P. T J. and is trained as follows:
min E Ex,,,,,,,. |||e 60 60 Xe,
T
For sampling, we employ classifier-free guidance for both condition image and text, and set two guidance scales (a, 3) to control their influence respectively. Since these two conditions are not independent (Liu et al.) 2022), we borrow inspiration from (Brooks et al.] 2023) and employ composed score function éo follows:
Éo(X,, Xc: P. T
= t, P,
+ alco(xx,, Xc: P. () co(x,, 0, P,9)]
(3)
+ B(eEo(x,, X., P. 6x(xe, L, Xc: P,O). 0)].
where we slightly abuse our notation and let @ represent the null symbol for both conditions.
Note that TOSS is highly flexible framework. Even in the absence of text descriptions, which means that TOSS degenerates to a classical NVS model f(x.. P). it consistently generates high- quality target images. We will demonstrate this ability in our ablation study.
3.3 ADAPTING A TEXT-TO-IMAGE MODEL FOR NVS
Recall that TOSS is formulated x - fxxe P. T Diffusion Rombach et al. 2022). whose formulation Practically, in TOSS capitalize on Stable fsD T due to its remarkable ability to generate diverse images according to text descriptions. While retaining Stable Diffusion's 's text- conditioning mechanism, we still need to enable to condition on the input view image x and the relative camera pose P for the task of NVS. Hence in the following subsections, will describe:
A dense cross attention module to condition fsD input view images. Introducing this mecha- nism fsp gives model g of the form g(xe,
Our mechanism for merging camera poses into the cross-attention module of g, to reach to our desired function, namely TOSS, fxx,, P. T that conditioned on condition images, camera poses, and text descriptions.
3.3.1 ENABLING SOXT-TO-IMAGE MODELS TO CONDITION ON IMAGE
Given input view image x., Zero (Luu al. 2023b) utilizes two image conditioning mecha- nisms as illustrated in Fig 3 (a) concatenating features of input image X. with x, along the channel dimension; (b) cross attention with x, CLIP embedding. However, both mechanisms possess in- herent limitations and are inadequate for the novel view synthesis task.
The underlying reasons for the inadequacy of previous conditioning mechanisms can be attributed to information misalignment and excessive information compression. Information misalignment: the feature maps produced by the VAE encoder of Stable Diffusion indicate not only the existence of certain local features but also their positions in the 2D image. Thus, when image features are con- catenated for NVS as shown in Fig. (a), the aforementioned property implicitly requires the feature maps of and x, to be aligned spatially. Namely, and x, must have alike object poses, which hardly met in real-world NVS scenarios. Therefore the concatenation method is inappropriate for learning features of novel poses. Excessive information compression: Zero123 utilizes features of x, as queries and CLIP embeddings of x, as keys and values in the cross-atie cooss-ttentionn mechanism. However, since CLIP contracts most of the local features and embeds the whole image into single
5

---

Preprint
Alload
Meed
Xt
X
Xt
Ideally, the relative camera pose should provide informative guidance on feature querying e.g- x, and x, share the same viewpoint, where the relative camera pose zero, the querying transform (b) Coos Attemto CLIP erihedling
Clazned Concrnmntion.
wi VAV Frbbodling
Figure 3: Comparing previous image conditioning mechanisms (a-b) and TOSS (c).
token, the cross attention module of Zero12 23 is actually incapable of attending to local features of the underlying object to be synthesized.
In contrast to previous approaches, we conduct cross attention between VAE embeddings of x, and X.: preservation of local details. Let fx,; fx. Rxx indicate the aforementioned VAE embeddings respectively, where h, w, and d denote the height, width and channels of the feature maps. We linearly project fo and fe into query, key. value features Qx Kx, Vx. Faxd where h w, and perform standard cross attention.
Compared to the concatenation method, in our setting any feature from x; could freely attend to any feature from x., unconstrained by their pose difference. Compared to the coarse attention with CLIP embeddings, as the features from x, and x, pass through the downsampling modules of our the information granularity gradually increases. Hence the cross attention module takes into consideration information from both global and local levels.
We summarize the three conditioning mechanisms below:
Channel Concatenation: output fx. + fxx (R"xd [ndd - 2d
Cross Attention: output = Cross-
Attn(Qxx: Cross-AAtt(Qu.. Qxi, KRaup V (R"xd, Ri Ridd) + Baxd
Ours: output Cross-Attu( Qx, Kx.; Vx.) (R"Xd paad pandd - gaxd
3.3.2 ENABLING TEXT-TO-IMAGE MODELS TO CONDITION RELATIVE CAMERA POSE
The relative camera pose is extremely important for modulating the correspondence between the input view and the target view for NVS. However, although being conditioned on texts, Stable Diffusion does not depend on camera poses, making it not directly applicable to the NVS task. Therefore, in this paper, we introduce simple yet effective pose conditioning mechanism that reflects such modulation effects.
tention should concentrate more keys from corresponding positions. Therefore
the camera pose to an extra key-value pair Kp, Vp glxd via learned projection matrices, ap- pend them to Kx, and Vx. respectively Kx.PP [ Ky] and V Vx,,P' [Vx.: Vp). where Kx_,P and Vx..P [(=+1)xd The pose-augmented key and value features then used in the cross attention module as shown Fig. The introduction of a new pose key-value pair alters the distribution of its attention coefficients, thereby implicitly modulating the attention strength between x, and x,- In contrast to previous works (Watson et al., 2022) (Liu et al. 2023b), we additionally perform frequency encoding on the camera pose so that the pose key-value pair preserves more high-frequener information Mildenhall et al. 2021).
More Accurate Pose via Expert Denoisers. Concurred with recent analyses on diffusion pro- cess (Huang al., 2023a; Choi al., 2022; Balaji et al.) 2022). we observe that during sampling, Co at large timesteps dictates novel view pose and further refines the NVS details at small timesteps. See illustrations in App. C.3 Drawing inspiration from ensemble expert denoisers (Balaji et al.) 2022), we train two expert denoisers that are specialized for denoising different timestep intervals, responsible for pose correctness and fine details respectively during the generative process. order to boost NVS quality while maintaining the same training and inference computation cost, begin by training a model shared among all noise levels. Then we initialize two expert models from this base model and specialize them high (timestep 1000-800) and low (timestep 800-0) noise levels.

---

Preprint
4 EXPERIMENTS
We evaluate TOSS on the tasks of single-view NVS and 3D reconstruction. Note that the 3D construction task more challenging, and demands higher multiview consistency. In Sec. 4.1 we describe data collating and model training protocols. The results single-view NVS and 3D reconstruction are presented in Sec. |422and and Sec. |.3 respectivey.. Ablations on the design choices provided in Sec. [4.4 Finally, we highlight in Sec. [4.5 that our improvements can be readily combined with upgrades in text-to-image models and techniques for even higher-quality NVS. All the visualizations provided in this section are produced without the two-stage expert denoisers.
4.1
EXPERIMENTAL SETTINGS
Captioning 3D Data. We employ an automated captioning procedure similar to that of Cap3D (Luo et al. 2023) but alter its final-stage caption fusion strategy slightly to caption Objaverse (Deitke et al.) 2023). the currently largest and most diverse, but captionless, 3D dataset. The detailed captioning procedure can be found in App. C.I We integrate Cap3D's 660K 3D-text pairs with 140K self- captioned 3D objects, resulting in a total of 800K Objaverse 3D-text pairs for training TOSS.
Implementation Details. We train TOSS captioned 3D data described above. For each 3D instance, we randomly sample 12 views for training. As shown in Fig. 2 we initialize TOSS with pre-trained Stable Diffusion v1.5 with both CLIP encoder and VAE encoder frozen. We train 120K steps with a total batch size of 2048 on 8 A100 GPUs which takes about 7 days. For training with expert denoisers, we first train 60K steps for all noise levels, then initialize two expert models from this base model and resume training for 12K and 48K steps respectively for high (timestep 1000- 800) and low (timestep 800-0) noise levels.
4.2 NOVEL VIEW SYNTHESIS FROM SINGLE IMAGE
2023b). we evaluate on GSO (Downs et al,, 2022) and RTMV (Tremblay et al., 2022) for NVS quality in Tab. [ each with 20 randomly selected objects/scenes rendered for 17 randomly sampled views, which are not included in the training dataset. We use four standard metrics for evaluation: PSNR, SSIM (Wang et al. 2004), LPIPS (Zhang et al., 2018) and KID. We keep the number of training images the same as the base- line (Zero123) for fair comparison.
Quantitative Results. Following Zero 13 Liu et al.
Method
Training images
Google Scanned Objects (GSO)
RIMV
PSNR (1) SSIM (1) LPIPS (L) KID (L) PSNR (t) SSIM (t) LPIPS (L) KID (1)
Image Variation
10.33
0.3094
0.3618
0.0543
Diet-NeRF
12.34
0.3290
0,4611
0,1211
Zerol-to-2
160M
17.75
0.8139
0.1369
0.0046
9.58
0.4180
0.3845
0.0267
TOSS (inference text)
160M
18.45
0.8401
0.1231
0.0046
10.50
0.5080
0.3497
0.0147
TOSS (inference w/ text)
160M
19.49
0.8580
0,1142
0.0036
10.75
0.5187
0.3360
0.0128
TOSS (w/ expert denoisers)
160M
19.70
0.8589
0.1131
0.0027
11.22
0.5823
0.3353
0.0132
Zerol -trol-to-3
250M
18.67
0.8322
0.1257
0.0023
10.28
0.4867
0.3592
0.0156
TOSS (inference w/o text)
250M
19.91
0.8649
0.1116
0.0034
11.39
0.5660
0.3213
0.0130
TOSS (inference w/ text)
20.09
0.8685
0.1114
0.0032
11.54
0.5734
0.3139
19
TOSS (w/ expert denoisers)
250M
20.16
0,8693
0.1109
0.0032
11.62
0.5754
0.3115
0.0104
Table 1: Quantitative comparison of single-view novel view synthesis GSO and KTMV.
Furthermore, we evaluate the 3D consistency scores following (Wat-) son et al., 2022). Specifically, we randomly sample 100 camera poses in the whole sphere, with 80 used for
SSIM
EPIPS (3)
PSNR 11. 38
SS)
LPIPS
Zeml22
0.880
0.2754
0,4350
0.642
TORS wi test)
21.54
0,2700
12.36
0.4696
0.6186
training and 20 for testing. Given a single image, different methods gen- erate novel views based on the 100 camera poses according to the input view image. Then we fit NeRF model on the generated novel-view images, and evaluate the NeRF model with PSNR, SSIM and LPIPS over the 20 testing images. As shown in Tab. 2 our model consistently improves multi-view consistency amongst different generated views.
Table 2: Comparing 3D consistency scores on GSO and RTMV.
7

---

Preprint
yellow sheil"
Auskpack Noddy ooher scalined
kay
sariety iacloling
Input
Zeroi23
(ours)
Inpot View
TOSS (ours)
Figure 4: Qualitative comparison of single-view NVS, GSO (Left) and RTMV (Right).
Qualitative Results. We show that TOSS consistently generates higher-quality novel view images compared to Zero123, evaluated across GSO (Fig. Left), RTMV (Fig. (Fig. Right) and tn-the-wild web images (Fig. (Fig.[ (c-d). Fig. (a.b) shows that TOSS produces more plausible and controllable gener- ations, respectively. Furthermore, it can be observed in Fig. (c) that TOSS achieves significantly higher multiview-ccnsiittene across multiple generations, using the same textual description.
4.3 3D RECONSTRUCTION
We evaluate TOSS for 3D reconstruction on GSO and RTMV instances, as well as randomly se- lected web images, with corresponding captions obtained via BLIP2 (Li 2023a). We use the SDS (Poole et al./ (2022) loss implemented by threestudio Guo et al. (023) for 3D reconstruction, evaluated both qualitatively, and quantitatively on exported mesh from reconstructed 3D NeRPs.
Quantitative Results.To To evaluate 3D reconstruction, we report the commonly used Chamefer Distances (CD) and Volume IoU between ground truth meshes and reconstructed meshes. Following previous work, we randomly select 20 instances each for GSO and RIMV. For each instance, we render an image with resolution 256 x 256 the input view. The numerical results in Tab. demonstrate that TOSS outperforms Zero123 in terms of reconstruction accuracy.
Method
GSO
KIMV
CD (1)
lol/ (1)
CD (4)
toU/ (*)
Magie123
0.1508
0.5220
Zerol 23
0,0757
0.5151
0.1546
0,2037
TUSSuuference wi text)
0,0655
115205
0.1236
0,2136
Table 3: Quantitative comparison of single-
view 3D reconstruction on GSO and RTMV.
Qualitative Results. Fig 3 shows that TOSS reconstructs figher-iielitty 3D models than baseline method Zero123, with finer details and better 3D meshes, given just a single image.
"Ad andee of backpook, bag show""
Rauriee brown apons"
"D0 parent ZOOM farsiure sot"
yullow anck with nccees
so fitcken with rind and
Saput
TOSS teans)
Zerv333
Eapat
10S8
Zenv123
T0S8 teani
Zevil3
Figure 5: Qualitative comparison of 3D reconstruction on GSO and RTMV.
4.4 ABLATION STUDY
We present in Tab, ablations the contributions of different design choices of TOSS. For more ablations please refer to App- D We ablate on a subset Objaverse, dubbed Objaverse vehicles, constituted of instances from the "vehicles" category including cars, ships, trains, spaceships and etc. The subset contains 3,000 instances with 12 randomly sampled views for training, and 20 instances with 16 randomly sampled views for testing. The subset constructed to be both efficient for
8

---

Preprint
training (comparing the full Objaverse) and contains enough diversity for evaluation (comparing to ShapeNet). For each ablation, we train on Objaverse vehicles on a total number of 450K images with batch size of 1024.
We start from Zero123 and progres- sively ablate the contributions of the following changes: 1) replace the orig- inal image conditioning mechanisms with our dense cross attention as de- tailed in 3.3.1 condition on text by replacing the CLIP image encoder with text encoder, initialized with pre-trained weights from Stable Diffusion; 3) con- dition on camera pose with dense cross attention instead of concatenation with CLIP embedding as detailed in 3.3.2 We observe that all introduced designs contribute positively to the NVS task, where dense cross attention and text conditioning attains the most significant improvements.
Modules
Contribution
Metries
PSNR (1)
SSIM (f)
PSNR (1)
SSIM (t)
Zero123
0.6123
+ Tokca-level attention
+2,70
+0.1762
13.43
0.7885
Text prompt
+1.41
+0.0390
16.84
0.8275
Camera pose token
+0.11
+0.0094
16.95
0.8369
Table Module ablations on Objaverse vehicles.
4.5 INTEGRATION WITH LIGHER-CAPABILITY
MANT-TO-IMAGE MODELS AND TECHNIQUES
Higher-capability Text-to-image Model. We replace the SD v1.4 model in TOSS with the higher- resolution (256 512) SD v1.5, and evaluate for NVS. As shown in Fig. 6 (Left), TOSS-512 not only generates finer details, but also aligns its generations better with the text descriptions.
"a model of redl and grece racing car"
"vol Aloster
Isput
TOSS 256 Resolation
TOSS 512 Resolation
Inpu: View
TOSS
Textual Invervion
knppor" "B.
Figure 6: Higher-quality NVS with TOSS. (Left) TOSS produces much higher quality NVS images with higher-capability generative models. (Right) TOSS further improves NVS quality by optimizing a S. object caption via textual inversion (Gal et al. 2022). Input texts are augmented with shared prefix "a photo of and fine details are highlighted with dashed rectangles.
Captioning Input View Image with Textual Inversion. Describing specific object precisely can be challenging. In such cases, we can leverage techniques from text-to-image concept generation, such as textual inversion (Gal al. 2022). To this end, we learn unique text embedding S. for the input view image and use it for subsequent NVS with TOSS. As depicted in [6) (Right), textual inversion further enhances the consistency with the input view (e.g., for the "cat statue" example, produces more plausible color stripes on the back and avoids adding extra illusionary tail).
LIMITATIONS AND CONCLUSION
Limitations and Future Works. Comparing to traditional NVS, TOSS optionally takes an ex- tra textual description, which can obtained automatically via captioning models. Developing captioning models more suited describing object details for NVS remains for future work. Sim- ilar to Z3,o123, we fine-tune Stable Diffusion on synthetic 3D datasets, which prone to training distribution shift, utilizing natural images and videos for training would alleviate this problem.
Conclusions. In this paper. have presented TOSS generate more plausible, controllable, and multiview-consistee novel view images from a single image. TOSS utilizes text as semantic guid- ance to further constrain the solution space of NVS. We naturally capitalize on Stable Diffusion and introduce novel dense cross attention module to condition the novel view generative process on in- put view image and relative camera pose. We also propose using expert denoisers to further enhance novel view pose correctness and fine details. Extensive experiments demonstrate that TOSS gener- ates novel view images that are significantly more plausible, controllable, and multiview-consister than Zero 123n NVS and 3D reconstruction tasks. Furthermore, showcase the substantial po- tential inherent to our approach by utilizing cigher-capability text-to-image models and techniques to further boost NVS quality.

---

Preprint
REFERENCES
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming-Yi Yu Liu. eDiff-I: Text-to-Image Diffusion Models with Ensemble of Expert Denoisers. arXiv preprint arXiv:2211.01324, 2022.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. InstructPix2Pix: Learning to Follow Im- age Editing Instructions. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 18392-18402, 2023.
Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient Geometry-awee aware 3D Generative Adversarial Networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16102-16112, 2021.
EricR Chan, Koki Nagano, MatthewA Chan, AlexanderW Bergman, JeongJoon Park, Axel Levy, Miika Aittala, ShaliniDe Mello, Tero Karras, and Gordon Wetzstein. Generative Novel View Synthesis with 3D-Aware Diffusion Models. arXiv preprint arXiv:2304.02602. 2023.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing Web- Scale Image- Text Pre-Traie Training To Recognize Long-Tail Visual Concepts. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3558- 3568, 2021.
Yen-Chi Cheng, Hsin-Ying Lee, Sergey Tulyakov, Alexander G Schwing, and Liang- Yan Gui. Sd- fusion: Multimodal 3d shape completion, reconstruction, and generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4456 0166-4465.40 2023.
Jooyoung Choi, Jungbeom Lee, Chachun Shin, Sungwon Kim, H Kim, and Yoon. Perception Pri- oritized Training of Diffusion Models. 2022 IEEE. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1 11462-11471, 2022.
Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. Diffusion Models in Vision: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A Universe of Anno- tated 3D Objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13142-13153, 2023.
Congyue Deng, Chiyu Jiang, Charles R Qi, Xinchen Yan, Yin Zhou, Leonidas Guibas, Dragomir Anguelov, et al. NeRDi: Single-View View NeRF Synthesis With Language-Guided Diffusion As General Image Priors. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 20637-20660 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion Models Beat GANs on Image Synthesis. Ad- vances in Neural Information Processing Systems, 34:8780-8794, 2021.
Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, ThomasB. McHugh, and Vincent Vanhoucke. Google Scanned Objects: A High-Quali Dataset of 3D Scanned Household Items. 2022 International Conference on Robotics and Automation, pp. 2553-2560, 2022.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion. arXiv preprint arXiv:2208 01618, 2022.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Networks. Communications of the ACM, 63(1 1):1119139144 2020.
Yuan-Chen Guo, Ying-Tiae Tian Liu, Ruizhi Shao, Christian Laforte, Vikram Voleti, Guan Luo, Chia- Hao Chen, Zi-Xin Zou, Chen Wang, Yan-Pei Cao, and Song- Hai Zhang. threestudio: A unified framework for 3d content generation. lhttps:// http://www........t.eeeetteeeeea http:/eeeeeeetoproeeeeeeeee threestudio 2023.
com/
10

---

Preprint
Anchit Gupta, Wenhan Xiong, Yixin Nie, lan Jones, and Barlas Oğuz. 3DGen: Triplane Latent Diffusion for Textured Mesh Generation. arXiv preprint arXiv:2303.05371, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. Advances in Neural Information Processing Systems, 0146640-65111 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Sali- mans. Cascaded Diffusion Models for High Fidelity Image Generation. The Journal of Machine Learning Research, 23(1):2249-288) 2022.
Yukun Huang, Jianan Wang. Yukai Shi, Xianbiao Qi, Zheng-Jun Zha, and Lei Zhang. Dream- Time: An Improved Optimization Strategy for Text-to-3D Content Creation. arXiv preprint arXiv:2306. 12422, 2023a.
Yukun Huang. Jianan Wang. Ailing Zeng, He Cao, Xianbiao Qi, Yukai Shi, Zueng-Jun Zha, and Lei Zhang. DreamWaltz: Make Scene with Complex 3D Animatable Avatars arXiv preprint arXiv:2305. 12529, 2023b.
Ho Le, Thomas Mensink, Partha Das, and Theo Gevers. Novel View Synthesis from Single Images via Point Cloud Transformation. Proceedings of the British Machine Vision Conference, 2020.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping Language- Image Pre-training with Frozen Image Encoders and Large Language Models. arXivv preprint arXiv:2301. 22212.991.297,, 2023a.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. GLIGEN: Open-Set Grounded Text-to-Image Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22511-22521, 2023b.
Henn- Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3D: High-Resolution Text-to-3D Content Creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 300-309, 2023.
Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Zexiang Xu, Hao Su, al. One-2-3-45: Any Single Image to 3D Mesh 45 Seconds without Per-Shape Optimization. arXiv preprint arXiv:2306. 16928, 2023a.
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional Visual Generation with Composable Diffusion Models. In European Conference on Computer Vision, pp. 423 439. Springer, 2022.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. --to---to-3 Zero-shot One Image to 3D Object. arXiv preprint arXiv:2303. 1132, 2023b
Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. SyneDreamer: Generating Multiview-consiitent Images from Single-view Image. arXiv preprint arXivv2 000.05533 2023c.
Zhen Liu, Yao Feng, Michael J. Black, Derek Nowrouzezahrai, Liam Paull, and Weiyang Liu. MeshDiffusion: Score-based Generative 3D Mesh Modeling. International Conference on Learning Representations, 2023d.
William E Lorensen and Harvey E Cline. Marching Cubes: A High Resolution 3D Surface Con- struction Algorithm. In Seminal Graphics: Pioneering Efforts that Shaped the Field, pp. 347-353. 1998.
Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In International Con- ference on Learning Representations, 2017.
Tiange Luo, Chris Rockwell, Honglak Lee, and Justin Johnson. Scalable 3D Captioning with Pre- trained Models. arXiv preprint arXiv:2306.07279 2023.

---

Preprint
Luke Melas Mariaziyriazi, C. Rupprecht, Iro Laina, and Andrea Vedaldi. RealFusion 360 Reconstruction of Any Object from a Single Image. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8446-8455, 2023.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. Communi- cations of the ACM, 65(1):99-106, 2021.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. ACM Transactions on Graphics (ToG), 41(4): 1-15, 2022.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. In International Conference on Machine Learning, 2021.
Atsuhiro Noguchi, Xiao Sun, Stephen Lin, and Tatsuya Harada. Unsupervised Learning of Efficient Geometry-Aware Neural Articulated Representations. In European Conference on Computer Vi- sion, pp. 597-614. Springer, 2022.
OpenAl. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774 2023.
Roy Or-EI, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, and Ira Kemelmacher- Shlizerman. StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13503-13513, 2022.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 165-174, 2019.
Ben Poole, Ajay Jain, Jonathan Barron, and Ben Mildenhall. DreamFusion: Text-to-3D using 2D Diffusion. arXiv preprint arXiv:2209.14988. 2022.
Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin- Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, al. Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors. arXiv preprint arXiv:2306. 17843, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning Transferable Visual Models From Natural Language Supervision. International Conference on Machine Learning. pp. 8748-8763 PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text- Conditional Image Generation with CLIP Latents. arXiv preprint arXiv:2204.06125, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- Resolution Image Synthesis with Latent Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Bureu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Pho- torealistic Text-to-Image Diffusion Models with Deep Language Understanding. arXiv preprint arXiv: 14487. 2022a.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image Super-Resolution via Iterative Refinement. IEEE Transactions on Pattern Anal- ysis and Machine Intelligence, 45(4):4713 4726, 2022b.
Christoph Schuhmann, Richard Vencu. Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION-400M: Open Dataset of CLIP-Filtered 400 Million Text Pairs. arXiv preprint arXiv:2111.021 2021.
12

---

Preprint
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. LAION-5B. An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.0844 2022.
Junyoung Seo, Wooseok Jang, Min-Seop Kwak, Jachoon Ko, Hyeonsu Kim, Junho Kim, Jin-Hwa Kim, Jiyoung Lee, and Seungryong Kim. Let 2D Diffusson Model Know 3D-Consistency for Robust Text-to-3D Generation. arXiv preprint arXiv:2303.07937, 2023.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep Unsuper- vised Learning using Nonequilibrium Thermodynamies. In International Conference on Machine Learning. pp. 2256-2265. PMLR, 2015.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-Based Generative Modeling through Stochastic Differential Equations. In Interna- tional Conference on Learning Representations, 2021.
Junshu Tang, Tengfei Wang, Bo Zhang. Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make- It-3D: High-Fidellt 3D Creation from A Single Image with Diffusion Prior. arXiv preprint arXiv:2303. 14184, 2023.
Jonathan Tremblay, Moustafa Meshry, Alex Evans, Jan Kautz, Alexander Keller, Sameh Khamis, Thomas Müller, Charles Loop, Nathan Morrical, Koki Nagano, et al. RTMV: A Ray-Traced Multi View Synthetic Dataset for Novel View Synthesis. arXiv preprint arXiv:2205.07058, 2022.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE transactions on image processing, 13(4): 600-612, 2004.
Daniel Watson, William Chan, Ricardo Martin-Brualla, Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi. Novel View Synthesis with Diffusion Models. arXiv preprint 2r11::2210.04628, 2022.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelNeRF: Neural Radiance Fields from One or Few Images. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 4576-4585, 2021.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding Conditional Control Text-to-Image Diffusion Models. In IEEE International Conference Computer Vision, 2023.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The Unreason- able Effectiveness of Deep Features as a Perceptual Metric. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 586-595, 2018.
Xin Zheng, Hao Pan, Peng-Shuai Wang, Xin Tong, Yang Liu, and Heung yeung Shum. Locally Attentional SDF Diffusion for Controllable 3D Shape Generation. ACM Transactions on Graphics (TOG), 42:1 13, 2023.
13

---

Preprint
A INTRODUCING TEXT TO CONSTRAIN THE NVS SOLUTION SPACE
Current NVS methods (Liu al. 2023b Chan et al. 2023 generate novel view images condi- tioned on input view images and camera poses, resulting in solution space highlighted in pink as shown in Fig. 7 TOSS further constrains the novel-view generation process with text as high- level semantics, which yields more compact solution space as highlighted in green Plausible image generation results conditioned each of the modalities (image, camera pose, text) pre- sented for reference. Notably, image-conditioned generation is widely known as image-to-image variation which focuses on producing images that roughly "look like" the input images (in this case red-colored standing characters in the lower left part). On the other end, text-conditioned image generation represented by Stable Diffusion (Rombach et al. 2022) emphasizes semantically correct generations of "Iron Man" in different poses.
Image linage
Stabic Diflisiian
Zare
TOSS (Ouns)
Text
Camera Pose
Image
PA
Input View
Target Novel View
Figure 7: An overview of utilizing different conditions for image generation. For each condition, three plausible solutions obcying the respective constraint provided for illustration. While Zero 123 considers the input view image and camera pose, TOSS further introduces text semantic constraint for generating plausible and controllable novel view synthesis (NVS) result.
B CONCISE PIPELINE FOR NVS
Other than the capability of utilizing text to enable more plausible and controllable NVS generation, TOSS also employs more concise training pipeline shown in Fig. 8 Zerol23 (Liu et al. 2023b requires fine-tuning Stable Diffusion on sarge-scale image collections to convert the original text-to- image model distribution shift. TOSS completely skips this problematic step, making our improvements fully orthogonal and be readily combined with upgrades in foundation text-to-image models, e.g. better architectures, textual inversion (Gal et al. 2022). etc., to boost the performance further.
image-to-image one, which is both resource demanding and subject to training
Stable Diffusion
Zero123
Texl-to-Image
Image-tu- Image
(Image, Camera Pose)-lo-
TOSS (ours)
Text-to- to--mage
(Text, Image, Camera Pose)-to-NY
Fine-tunes large-seale image collections (replace CLIP encoder with image encoder)
Fine-tunes on renderings of 3D instances
Figure 8: TOSS utilizes more concise training pipeline.
14

---

Preprint
C MORE EXPERIMENTAL SETTINGS
C.1 MORE DETAILS ON CAPTIONING DATA
We utilize an automated pipeline to caption 3D data. Specifically, we first employ BLIP2 (Li al. 2023a) to caption collection of 3D object renderings. Subsequently, we apply CLIP-based (Rad-) ford et al. 2021) ranking mechanism mitigate potential inconsistencies that may arise when gen- erating captions for different viewpoints of the same object. Lastly, we incorporate several regular expression rules to filter out inaccurate descriptions. Concurrently, we observe that Cap3D (Luo et al. 2023 follows similar procedure but utilizes GPT-4 (OpenAl 2023) at the final stage to fuse filtered captions across different views.
At inference time, users can either provide text information use image captioning models such as BLIP2 (Li et al. 2023a) for automatic captioning.
C.2 MORE TRAINING DETAILS
During model training we employ an AdamW optimizer (Loshchilov & Hutter 2017) with learning rate of 10-3 for the camera pose encoder and 10 for other modules. In all the experiments, we train our model with the 16-bit floating point (fp16) format for efficiency. To enable classifier-free guidance, we randomly mask 50% of the samples for text in each batch and 10% for condition images. For computing the consistency score, we directly train an instant-ngp (Müller et al.) 2022) for 30 epochs 2022) and hyper- parameeer settings implemented by threestudio (Guo et al. 2023). The instant- ngp model first renders novel-view images at resolution of 64 64, which are then up-sampled to 256 x 256 resolution and supervised the latent space. It takes about 3 minutes for training 400 epochs with batch size of 12. After model training, we apply the marching cubes algorithm (Lorensen & Cline
the generated novel views. For 3D generation we adopt the SDS loss (Poole al.,
1998) to export meshes from the trained instant-ngp.
C.3 EXPERT DENOISERS
We visualize the novel view image generation process in Fig. 9 For large timesteps f, the gradients provided by supervise the geometry while small timesteps induce gradients that focus more on fine details. The observation motivates us to train dual expert denoisers for novel view pose correctness and fine details respectively as described in the main paper.
Input
Timesteps
Target
1000
800
600
400
200
Eg high
Eg low
Figure 9: The generative process of NVS. ea large timesteps ensure the accuracy of novel view poses and small timesteps responsible for generating fine details.
15

---

Preprint
D MORE ABLATION STUDY
Guidance Settings Given that we employ classifier-free guidance (CFG) for both conditional im- ages and texts, the need of trade-off between their respective guidance scales naturally arises. Hence we have conducted ablations GSO and RTMV datasets, explore various combinations of guid- ance scales as illustrated in Fig. T0 It is worth emphasizing that the dual-guidance configuration suggest raising the text guidance scale, 3, when prioritizing the text controllability of TOSS. Conversely, a higher image guidance scale, a, is recommended when the provided condition images contain sufficient information on the objects.
offers flexibility to cater to different scenarios. In conclusion,
Text guidance
Text guidance
PSNR
5
PSNR
19.26
19.26
19.26
19.27
11.25
11.25
11,25
11.25
20.14
20.14
20.10
20.01
11.59
11.62
11.58
11.54
Image guidance
Image guidance
20.14
20.14
20.14
5
11.54
11.53
11.54
11.53
19.94
19.97
19.99
19.96
11.42
11.41
11.43
11.43
GSO
RTMV
Figure 10: Performance of different guidance settings GSO and RTMV.
Quality of Texts TOSS aims to utilize text as high-level semantics to constrain the NVS solution space. Consequently, the quality of the given text plays a significant role in influencing the quality of the generated results. Here we delve into the impact text descriptions at varying levels of granularity on the results. From Figure II] it is evident that when provided with a coarse text, there may exist slight deviations in the generated object details (for instance, a change in the color of the unicorn's horn). However, by further refining the granularity of the prompt description, these deviations can be avoided with more precise textual control.
Furthermore, we conduct manual re-annotation with more detailed descriptions on the GSO subset. This text refinement leads to an increase of 0.1 in the PSNR value compared to the initial assessment.
Input
Hhitt aniove"
smiconn rimble with Alose eust grewn
Imput
"Viphhar"
cute gray mified ripphanr nimeg"
pationes
Input
"alor clock"
Inpat
"roneer"
comb vising comboy for"
back
Figure 11: Refining text prompts enhances the quality of generated novel views.
Importance of Using Pre-trained Weights. During our experiments, we also explore training TOSS from scratch rather than initializing TOSS with pre-trained weights of Stable Diffusion. Our experiments are conducted on the Objaverse vehicles dataset as detailed in the main paper Sec. 4.4 with the same number of training steps. The comparative analysis of the novel view synthesis results
16

---

Preprint
presented in Fig. 12 It is evident that in the absence of text-to-image pre-training, the novel view generations exhibit significant distortions in shape and texture, validating the importance of utilizing pre-trained knowledge from generative models.
"Whe International station shows 5d rondaringg
"o model ir*
red land rover discovery mode!"
rod atrpione
fiying
Input
pretrain
pretrain
Target
Input
w/o pretrain
pretrain
Turget
Figure 12: Importance of initializing TOSS with pre-trained Stable Diffusion.
17

---

Preprint
E QUALITATIVE COMPARISON FOR CONTRIBUTIONS OF DIFFERENT MODULES
"a police car is shown 3d"
"a international space station
shown in this 3d
rendering"
Input
Zero 123
+Token-level attention
+Token-level attention &text prompt
GT
Figure 13: Qualitative comparison for contributions of different modules.
F MORE PLAUSIBILITY RESULTS
koola
chimpanzee space weaarnng helmet"
"a white stuffed animal with red tongue"
standing"
"yellow duck with a
"a white unicorn figurine"
Japtop with black back
Input
Zerol23
TOSS
Input
Zerol23
TOSS
Input
Zero123
TOSS
Figure 14: More results demonstrating the increased NVS plausibility with TOSS..
18

---

Preprint
G MORE CONTROLLABILITY ResUlTs
"a box with paintings in it"
"a box with cat in it"
"a box with candies in it"
"a box with lego toys in it"
Input
"a dragon stuffed toy with fire on the back"
"a dragon stuffed toy with ice on the back"
"a dragon stuffed toy with green leaf the back"
"a dragon stuffed toy with sword on the back"
Input
Input
bottle"
"a ball" "
cup"
hoop"
"minion with bug the back" "
"minion with rocket the back"
"minion with fox tail on the back"
"minion with black cape on the back" "
Input
Figure 15: More results demonstrating the increased NVS controllability with TOSS.
19

---

Preprint
H QUALITATIVE COMPARISON ON MULTI-VIEW CONSISTENCY
"minion
"a red fire extinguisher"
without
backpack"
Input
Zero123
TOSS
Input
Zerol23
TOSS
Figure 16: Qualitative comparison on multi-view consistency.
I
MORE NVS RESULTS
Input
Generated Multivien-comistent images
green
"drwns"
"mirrpplone"
"hotdogs"
To yelfow legoo
buildazer"
Figure 17: NVS examples using TOSS Synthetic NeRF datasct Mildenhall al. 2021
20

---

Preprint
Input
Random sampled novel views
Inpet
Random sampled nevel views
Youny
rohooo buy"
muttlerr steel
pressuee cooker
"ptrto
"nccual dragon vittoe
"Sokyy boar
"heo
"mug shud!
park"
Figure 18: Random sampled novel views using TOSS. Input images from the GSO dataset [Luu al. 2023c
and 72ro123 (Liu 2023b) released on GitHub.
21

---

Preprint
J
MORE 3D CONSTRUCTION RESULTS
Input
3D Reconstruction
"a red fire hydrant
"a purple and yellow dragon stuffed toy"
"a bottle of lysol all- purpose cleaner" "
"nintendo mario bros"
yellow toy taxi car"
"a rubber duckk wearing red hot"
"a gray stuffed elephant with burgundy bow"
"a cupcake with green frosting and mint leaves"
Figure 19: 3D reconstruction results on in-the-wild images.
22


Reviews:
[{"review_id": "cE8dqJpLCi", "reviewer": "Reviewer_gFkU", "summary": "This paper proposes to use text as high-level semantic information to constrain the NVS solution space. With this text, the proposed method can generate the multi-view consistency images. Meanwhile, it proposes dense cross-attention to align the reference and target image.", "questions": "see the weakness", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The whole idea is Intuitive and effective.  Using the text as semantic guidance to preserve multi-view consistency is reasonable since the ref image can not provide enough information. Meanwhile, since the proposed model uses the text prompt, it can finetune the SD directly, which preserves the ability of generation.\n2. The attention strategy is reasonable, which will help these two images align better.", "weaknesses": "1. My main concern is Whether the texts bring enough improvement.  As shown in Table 4, I find \"Token-level attention\"  improves the model the most, while the \"Text prompt\" is not the best.  Meanwhile, the author should show these cases visually: 1. Zero123 + Token-level attention 2. Zero123 + Text prompt.\n2. I'm curious about the effect of this model on the human portrait since the Zero123 performance is bad in human portraits (Dataset limitations). The author can show two cases visually: 1. Taylor Swift (you can select an image of Taylor Swift as input) 2. Young male (you can randomly select a young male image).\n\nIf the authors can address my question properly, I will be glad to improve my grade.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T7BVDAYc7g", "reviewer": "Reviewer_wyYi", "summary": "Novel view synthesis from a single RGB image is an under-constrained problem, and Zero123 solves this problem by training a view-conditioned latent diffusion model. However, Zero123 treats it as a pure image-to-image translation problem and thus suffers from pixel-level inconsistency problems. In this paper, TOSS adds text as high-level semantic information and, more importantly, proposes the cross-attention mechanism for better 3D consistency. Through exhaustive experiments, TOSS outperforms baseline Zero123 with more plausible and multiview-consistent NVS results, thus leading to substantial improvement in 3D reconstruction.", "questions": "1. During the cross-attention process, have you tried other orders or combinations of text, image, and camera pose?\n2. Expect more detailed elaborations on expert denoisers in the paper, in particular the motivation.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed dense cross-attention mechanism is quite reasonable, and the explanation, especially in Figure 3, is persuasive. In the experiment section, both quantitative and qualitative results of novel view synthesis demonstrate the effectiveness of this module. More importantly, it significantly improves the 3D consistency, reflected in better 3D reconstruction results.\n2. The additional text conditions increase the controllability of the TOSS model. Hence, text prompts lead to diverse NVS results as presented in the paper.\n3. This paper is well-written and presented with a clear architecture. For the contributions summarized in the introduction, the corresponding experiment support can almost be found in the experiments.", "weaknesses": "1. The authors claim that the text prompts increase the plausibility of novel view synthesis results. However, only the corresponding quantitative results (especially in the ablation study) can be found in the paper. Apart from the illustration in Figure 1, more qualitative results should be presented to support this claim.\n2. From the comparisons in Tables 1&2, TOSS only brings minor improvements over baseline method Zero123. Actually, prior work One-2-3-45 presents an interesting heatmap in Figure 4, showing the PSNR values are significantly affected by the relative azimuth and elevation angles. Hence, the authors should elaborate on how the results in Tables 1&2 were obtained to avoid unfair comparison.\n3. For the 3D reconstruction, the authors should also compare with relevant methods such as Magic123 and Consistent123.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "emimCS8fOR", "reviewer": "Reviewer_Givb", "summary": "This paper proposes a TOSS that is text to the task of novel view synthesis (NVS) from a single image. Compared to a previous method (Zero123), TOSS utilizes a text as high-level semantic information to constrain the NVS solution space. It is based on the text-to-image Stable Diffusion pre-trained on the large-scale dataset. The proposed method achieves plausible results and those are controllable. The effectiveness of the proposed method is validated on the dataset.", "questions": "- In Table 1, there are no experiments for \"inference w/o text\" and no experiments for \"w/ expert denoisers\" for 160M?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "[Novelty] \n\n- This paper utilizes diffusion prior and text embedding features with cross attention to perform NVS using a single image.\n\n- This improves reconstruction performance for unseen areas and allows for consistent image generation.\n\n[Quality] \n\n- Overall, the paper is easy to follow and well written.\n\n- This paper reports the impact of each module of the proposed method in terms of performance as appropriate.\n\n[Clarity] \n\n- The paper is clearly written.\n\n- Motivation and explanation of each proposed module are reasonable.\n\n[Significance] \n\n- As a study that improves upon existing research, I think there are many elements that can be utilized in follow-up studies.", "weaknesses": "- Compared to just using cross attention (CLIP embeddings), it seems that it needs to analyze various perspectives such as computational cost or memory.\n\n- Also, the analysis of the effectiveness of Expert Denoioser seems to be somewhat lacking.\n\n- There are experiments that show the quantitative effectiveness of each of the proposed modules, but the qualitative, in-depth analysis is somewhat lacking.\n\n- This paper is somewhat limited by the fact that comparative experiments were conducted with Zero123 only. I think it would be a better paper to compare NVS with a single image, or a diffusion-based model adapted to a given task, even if it is not exactly the same task.\n\t\t \t \t \t\t\n- There is no section 3.2.2, so there is also no need for section 3.2.1. It is better to use just section 3.2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zfkInVhKgv", "reviewer": "Reviewer_LBZY", "summary": "This paper proposes TOSS, which is an image-to-image diffusion model. Comparing to pioneering work like Zero123, TOSS is different because it's not only conditioned on input image and relative camera pose, but also an optional text prompt. \n\nSince single image to other views are naturally an ill-pose problem, the authors claim that text prompt can provide more detailed guidance, which allows for increasing plausibility and controllability, and consequently better quality. In TOSS authors also analyze the drawbacks of previous conditioning mechanism and propose a dense attention mechanism for improved performance. \n\nAuthors provide a valid pipeline for automatic captioning and train the TOSS model on the large-scale Objaverse dataset. The extensive experiments prove the validness of each components and the superior performance of the proposed TOSS model.", "questions": "In Sec4.5, what is the detail meaning of \"replace the SD v1.4 model\". Does it mean you replace the VAE encoder and the decoder? Does it replaced only in inference or in the training and re-train the model?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "In general, this paper is motivated, well-organized and provide valuable results. In detail I find paper's strength in several aspects:\n\n1. The design of dense attention mechanism and the necessity of text as input are well motivated, with clear reason and proof.\n\n2. The experiment part is sufficient. Extensive experiments have been done on not only for the comparison and main results, but also ablation on a wide range of components.\n\n3. The authors also propose some valuable practices about image-to-image diffusion model, for example the expert denoiser design.", "weaknesses": "### 1. Over-claim multiview-consistency ###\n\nI think the fundamental weakness for this paper is over-claim. In Sec.3.2.1, authors list two advantages of introducing text-prompt: plausibility and controllability, and I agree these two points. However, in the other parts of the paper, for example the Fig.1's caption, authors list *multiview-consistency* as another improvement, which I totally disagree.\n\nFundamentally, TOSS is like the Zero123 but with more input condition, it doesn't adopt some explicit design to guarantee the multiview-consistency like in SyncDreamer[1], so naturally it shouldn't be multi-view consistent. And although authors claim this in a lot of places in paper, I didn't find explanation towards this point.\n\nAlso, in Fig.13, while author titled it as \"Generated Multiview-consistent images\", it's actually not. For example in the provided microphone samples, I can clearly find it's not consistent across different views. So I think authors should revise the manuscript and avoid over-claim on being multivew-consistent.\n\n### 2. Emphasize the controllability ###\n\nAnother minor weakness is I hope authors can emphasize more on the TOSS's controllability. I would like to see the authors provide more visualization results where from the same input image, different text can generate different novel views, currently the results are mainly in supplementary and I think it's better to emphasize this ability and move the part to the main paper.\n\n[1] Liu, Yuan, et al. \"SyncDreamer: Generating Multiview-consistent Images from a Single-view Image.\" arXiv preprint arXiv:2309.03453 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)