ViewFusion: Learning Composable Diffusion Models for Novel View Synthesis
Authors: Bernard Spiegl
Andrea Perin
Stephane Deny
Alexander Ilin
Venue: Accepted by TMLR ()

Abstract: Deep learning is providing a wealth of new approaches to the problem of novel view synthesis, from Neural Radiance Field (NeRF) based approaches to end-to-end style architectures. Each approach offers specific strengths but also comes with limitations in their applicability. This work introduces ViewFusion, an end-to-end generative approach to novel view synthesis with unparalleled flexibility. ViewFusion consists in simultaneously applying a diffusion denoising step to any number of input views of a scene, then combining the noise gradients obtained for each view with an (inferred) pixel-weighting mask, ensuring that for each region of the target view only the most informative input views are taken into account. Our approach resolves several limitations of previous approaches by (1) being trainable and generalizing across multiple scenes and object classes, (2) adaptively taking in a variable number of pose-free views at both train and test time, (3) generating plausible views even in severely underdetermined conditions (thanks to its generative nature)---all while generating views of quality on par or even better than comparable methods. Limitations include not generating a 3D embedding of the scene, resulting in a relatively slow inference speed, and our method only being tested on the relatively small Neural 3D Mesh Renderer dataset. Code is available.

Published in Transactions on Machine Learning Research (05/2025)
ViewFusion: Learning Composable Diffusion Models for Novel View Synthesis
Bernard Spieg!* Aolto University
bernard.spiegl@ppooom...
Andrea Perin Aalto Universiby
andrea.perim@uualto,
Stéphane Deny
stephone.deng..nooggnme proceemyprro@mme
Aalto University
Alexander Ilin
alezander. deezander.iin@xspttmmaaa.ooo
System 2
Aolto Universily
Reviewed on OpenReview: https: openreview. net/ forum? id= amUisgrate
Abstract
Deep learning is providing a wealth of new approaches to the problem of novel view synthesis, from Neural Radiance Field (NeRF) based approaches end-to-end style architectures. Each approach offers specific strengths but also comes with limitations in their applicability. This work introduces ViewPasion, an end-to-end gemerative approach to novel view synt hesis with unparalleled flexibility. ViewFusion consists in simultancously applying a diffusion denoising step to any nunber input views scene, then combining the noise gradients obtained for each view with an (inferred) pixel-weighting mask, ensuring that for each region of the target view only the most informative input views are taken into account. Our approach resolves several limitations of previous approaches by (1) being trainable and generalizing across multiple scenes and object classes, (2) adaptively laking in a variable number of pose-free views at both train and test time, (3) generating plausible views even in severely underdetermined conditions (thanks to its generative naturo) all while generating views of quality on par or even better than comparable methods. Limitations include not generating a 3D embedding of the scene, resulting in a relatively slow inference speed, and our method only being tested the relatively small Neural 3D) Mesh Renderer dataset. Code available at httpss//ggituu..... htt.ttttbb.coo/rroooooo///eeeeee
1 Introduction
Novel view synthesis computer vision problem with a long research history. Traditionally, approaches that explicitly model the 3D space have been used such as voxels (Kim al. 2013). point clouds (Agarwal et al. 2011) meshes (Riegler & Koltun 2020). With advancements of machine learning techniques and capabilities, various methods based neural radiance fields (NeRFs) (Mildenhall et al. 2021 have emerged.
These approaches aim to represent a 3D scene implicitly by using an MLP to paramcterize it. Most recently, methods use an end-to-end, image-to-image approach where a collection of images of a scene is given to the model to produce a novel view the scene (Sun et al.) 2018 Dupont et al.) 2020 Sajjadi et al.) 2022b a 2023 Despite the extensive variety of methods, all the proposed approaches come with their drawbacke backs, such as (1) requiring expensive per-scene retraining and an abundance of input views, (2) an inability
"Corresponding author. Work done at Aalto University, prior appointment CERN and is unrelated to the position. Work done at Aalto University.
1

---

Published in Transactions on Machine Learning Research (05/2025)
Table 1: Comparison of features with previous methods. The features refer to the method's capability to: (1) operate in setting where pose information about input views is not available, (2) generalize across multiple scenes and classes without the need to be retrained, (3) make use of variable input view count both at inference and training time, (4) produce a variety of plausible views when dealing with underdetermined, fully occluded target viewing directions.
cb al.
(Sajjadi et al, 2022b
ViewFusion (Ours)
Lin et al. 2023
Yu et al. 2021
VIT for NeRF
Sitzmann
PixeINeRP
LPS 21221
SRT
1) Pose-free
VXKA
X x
HAZN
2) Generalization
Variable input
4) Generative
X
operate without pose information about the input views or (3) an inability to adapt to a variable number of input views at test time. Therefore, the aim of this work is to introduce an intuitive end-to-end architecture for novel view synthesis which resolves the aforementioned drawbacks of previous work.
We propose View Fusion, a novel approach that tackles the mentioned drawbacks all at once through a series of problem-specific design choices. Our method employs a diffusion probabilistic framework. We simultaneously apply diffusion denoising step to any number of input views of a scene, then combine the noise gradients obtained for each view with pixel-weighting mask, inferred specifically for every view, to ensure that for each region of the target view only the most informative input views are taken into account. Our method combination of multiple single-view denoising models for novel view synthesis, which, in
be understood addition to the noise predictions, produce weights used to aggregate the corresponding noise predictions of each of the single-view diffusion models during the denoising process. By training our method on a multitude of scenes and classes at once, we enable it to generalize without the need for retraining on every scene. Thanks to the stochastic nature of the diffusion process, the model is capable of performing well even in underdetermined settings (e.g. severe occlusion of objects or limited amount of input views) by providing a variety of plausible views. Our proposed solution does not require ordering nor any explicit pose information about the input views and, unlike the previous counterparts, once trained, the model is able to effectively handle inputs of arbitrary length. The ability to handle inputs of arbitrary length is thanks to the novel weighting mechanism that allows the model to weight views based on their informativeness and redundancy.
We evaluate our proposed approach on the Neural 3D Mesh Renderer (NMR) dataset (Kato et al. 2018 Chang et al., 2015) consisting of a wide variety of classes and input view poses. Through quantitative evaluation we show improved performance compared to relevant methods. We also qualitatively explore intermediate outputs of the model and confirm the soundness of our pixcl-weighting mechanism to infer and adaptively adjust the importance of each of the input views: the inferred weighting scheme aligns with our human intuition that input views closer to the target view should be more informative than the further ones.
Summarized, the main contributions of this work are (also see Tables and
a novel and intuitive approach to perform novel view synthesis, using specifically tailored weighting mechanism paired with composable diffusion,
a highly ficxible solution thanks to the model's ability to process unordered and pose-froe collections of images with variable length both at inference and training time, all while generalizing across a multitude of different classes,
2

---

Published in Transactions on Machine Learning Research (05/2025)
an inherent capability of the model to handle highly underdetermined (e.g. full occlusion) cases thanks to its generative capabilities,
a competitive performance while providing significant flexibility improvements.
2
Related Work
Novel view synthesis is a long researched topic with solutions ranging from explicit modelling of the 3D space to more recent NeRFs and end-to-end approaches. However, these solutions often come with various drawbacks that our approach aims to address.
Neural Radiance Fields (NERFs). NeRTs (Mildenhall et al. 2021 Yu et al., 2021 Lin et al, 2023) aim to perform novel view synthesis by optimizing an underlying continuous volumetric scene function using a sparse set of imput views. The volumetric scene function is represented by a neural network whose inputs are a spatial location in the form of position and viewing direction, and the output is color and volumetric density at a given point. Ultimately, once a NeRF model is trained, we can query all the possible spatial locations in order to obtain all possible views of the object. While NeRF's usually yield high quality results, they require a significant amount of training data for a single object and commonly do not generalize well across different objects, requiring objecl-dependent retraining. The need for per-scene retraining makes these models especially problematic when comes to online applications where view synthesis has to be performed on objects of previously unseen class. While there have been extensions that do not require per-scene retraining like Yu et al. (2021); Jiang et al. (2023), ; these approaches also rely explicit and precise camera 2 which are not always available, or lack the generative capabilities to handle single-view cases or scenarios in which objects are occluded (Jiang et al. 2023).
poses Yu et al.
End Novel View Synthesis. Using end-to-end based architectures is another commonly seen approach when it comes to novel view synthesis. For instance, Equivariant Neural Renderer (Dupont et al., 2020) aims lo explicitly impose 3D structure on learned latent representations by ensuring that they transform like real 3D scene. The encoded latents are then transformed before being passed to the decoder to produce the target view. Another example of an end-to-end approach that has shown promising performance for performing novel view synthesis is the Scene Representation Transformer (Sajjadi et al. 2022b) and its pose-free (Sajjadi et al. 2023) and object-centric (Sajjadi et al. 2022a) follow-ups. They focus on learning a latent representation of a scene by encoding a set of input images and passing it to a decoder to synthesise new views. A major drawback of the mentioned methods is their entirely deterministic nature, meaning that at inference time they do not have the ability to output a variety of plausible views when dealing with underdetermined scenarios, making them difficult. to use in the settings such as severe occlusion or limited amount of input views. [Sun al. (2018) deal with this issue by employing generative approach and self-learned confidence, but like the aforementioned NeRFs, they require explicit pose information to be provided for each of the input views.
Diffusson Probabilistic Models. Recently, models based on finding the reverse Markov chain transit in order to maximize the likelihood of the training data, commonly known as diffusion probabilistic models (Sohl-Dickstein et al. 2015), have seen extensive use for solving various text-conditioned generative tasks
Ho et al. 2020: Rombach et al.) 2022) including text-conditioned 3D synthesis (Poole et al. 2022 Li et al, 2023 Shi et al. 2023b). One of the main reasons for widespread use of diffusion models is their capability produce high quality outputs when conditioned on textual descriptions. Diffussonn models consist of stochastic diffusion process and deep neural network that parameterizes the denoising function used to perform the denoising procedure. Besides being used for generating images conditioned on text-based prompts, diffusion probabilistic models have seen applications in solving various other problems, including image-lo-inage novel view synthesis. These range from models that use pre-trained stable diffusion backbones such as Liu et al. 2023a)) and its extensions (Shi et al. 2023a Liu et al. 2023b Chen et al., 2025), to models trained from scratch such as Muller et al. (2023), which operates directly on 3D radiance fields or Watson ct al. (2022): Anciukevičuus et al.] (2023 2024); Müller et al. (2024); Tang et al. (2024), in which a novel view diffusion based models introduced to perform end-to-end novel view synthesis in image domain.
3

---

Published in Transactions on Machine Learning Research (05/2025)
While these methods provide good qualitative results, they often come with various drawbacks. In particular, approaches based on using a pre-trained backbone rely those backbones being freely and easily available. This is often not the case as authors of large models trained on extensive datasets refrain from making the weights and architectures publicly available, and instead provide access solely through an API, c.g. LLMs, large-scale diffusion models, elc. Furthermore, Müller et al. (2023) lacks the ability to generalize across multiple classes, Müller al. (2024) is pose-conditional, Watson et al. (2022); Anciukevičius al. (2023); Liu et al. (2023a]; Shi et al. [2023a]; Chen et al. (2025) cannot make use of additional views when they are available, Anciukevičtus et al. (2024) lacks the ability to extrapolate at test time beyond number of imput views available at training time and Tang et al. (2024) operates on fixed grid with predetermined number of input views, resulting in reduced flexibility. Therrfooee, even though these approaches offer high quality synthesized views, they all come with specific drawbacks that we aim to address.
3 Method
Our approach (Figure D employs a composable diffussin probabilistic framework to generate novel views. The model receives an unordered, pose-free free and arbitrarily long collection of imput views {x scene along with the target viewing dircction Adh. The model then predicts the scene viewed from the larget viewing direction Ad..
J.V of given
[xp]
sof tmax(W)
U-Net
U-Net
y--1
U-Net
Same Backhone
Reused und Paralletized
Figure 1: Architecture Overview. An arbitrary number of unordered and pose-free views coupled with the noise at timestep is denoised in parallel using a U-Net conditioned on timestep and target viewing angle At. The model then produces noise predictions and corresponding weights w; for timestep A composed noise prediction, computed as weighted sum of individual contributions, then subtracted from the previous timestep prediction. Ultimately, after T timesteps, fully denoised target view is obtained.
3.1 Architecture
Each input view is treated separately through identical streams. We feed each input view separately to an identical copy of the denoising backbone a diffusion model (see Appendix for a technical description of diffusion). We use U-Net as the denoising backbone, with the same architecture and hyperparameters Saharia al. (2022), unless specified otherwise below. Given variable set of input views fx; E 3 Je concatenate the noisy image timestep I, (y: 133xxxW77 ) along the channel dimension to each
4

---

Published in Transactions on Machine Learning Research (05/2025)
of the views, in order to produce collection of U-Net inputs (V, R (2022). Additionally, in order to globally condition the model on the target pose information, positional encodings [Vaswani et al., 2017) of the target pose Ae (single angle in radians) and timestep t (integer) are concatenated and jointly embedded using a simple MLP consisting of two layers one hidden layer with a sigmaid non-linearity followed by an output linear layer (dimennsionality 128). The conditional embeddings produced by the ML. are injected into the U-Net through feature-wise affine transformations (Perez et al., 2018) in all downscaling and upscaling ResNet blocks. Aúh is the angular disparity between the target view and the canonical pose of the object, i.. the front facing view of the object at 0º, as defined in the dataset (in Appendix D.1 we also explore relative canonical pose by defining Anb as angular disparity between the target view and the first imput view V1). We introduce the notation cf = (x, Ad, t) to denote a tuple containing all the inputs to the U-Net.
li=1: following Saharia et al.
At each denoising step, all streams are composed through an inferred weighting strategy. For set of input views fx, E 333Hxx Ji=1. a set of corresponding noise weighting masks {w; E R J is produced at every timestep by the U-Net backbone. Indeed, the U-Net outputs pairs of noise predictions é and weights Wi, computed by two different heads at the final layer of the U-Net (obtained by changing the output layer channel count from 3 in the standard backbone to 6), each of the same dimensionality as the noise prediction (same dimension as the image). The weights tensor W xg( is then normalized using softmar along the channel dimension. Next, these normalized weights are applied to the noise, and the weighted noise contributions are then summed, lorming the final noise prediction yr-1- Intuitively, the weights reflect per-pixel informativeness of each of the input views for producing the target view. The described architecture is trained end-to-end using Lg loss computed between the true noise and model's prediction. Algorithm shows the training procodure pscudocode. a denotes the noise schedule and I IT ay, further described in Appendix A
A Algorithm Composing View Contributions Training
repeat
2: (x, yo) q(x, Avb,))
(sample data point) } (sample a timestep)
3: t (Iiform((1,....,.
4:
I(0.T
(sample noise)
5:
e fo( Vrayo + v1 roc 76)
(predict the noise and weights} (normalize the weights)
6: w. softmaz( dim=0),
7: Take gradient descent step on
8: V. Volle-Ei. Em w. w.eell
9: until converged
At inference time (Algorithm 2 the denoising process is repeated for T timesteps. Like in training, after each timestep, the noise predictions are weighted and summed together. Then, they are passed as conditioning for the next timestep prediction. Ultimately, after timesteps, the final target view is produced.
Algorithm Composing View Contributions Inference
yT 10,1)
(sample starting noise]
2: for L = T...., do
3: =~ N( if 1, clse
(sample noise) (predict the noise and weights) (normalize the weights)
4:
è. .W - yo, yr, An, (,,,, 74)
5:
w; = softman W, dim
y: - Ei w. o&) +VI-au
6: y vas
7: end for
return Yo
3.2 A Probabilistic Interpretation of ViewFusion
Here we provide a theoretical framework to justify the design choices of ViewFusion (see Appendix C for extended version of this argument). We will do so by enforcing specific desiderala about the transition
5

---

Published in Transactions on Machine Learning Research (05/2025)
probability of the reverse diffusion process, in the specific context of pose-free novel view synthesis. First, given set of input views {x:}{=1, the transition probability of the reverse diffusson process should not depend on the specific order in which these imput views are fed to the model. Indeed, input views do not contain pose information, and thus cannot be ordered in any meaningful way. One way to enforce such permutation invariance is lo write the transition probability of the reverse diffusion process as sum contributions for each of the input views, where each input view contributes separately through an identical energy function E:
p(y:-1 &
exp( Exp(-E(yt-1.c))
(1)
This functional form permutation-invarian by construction (as also remarked by Zaheer et al. (2017)) )
hence does not depend on the order of the N imput views. In addition, this functional form can be applied
any number of views, allowing our model to flexibly deal with an arbitrary number of views. This functional
form can also be interpreted as mirture of eoperts, where cach individual view-conditioned stream acts one expert in predicling the reverse diffussio process.
Next, we show how the softmax weighting scheme in our approach directly derives from the functional form described above, without any further assumption. The update step of diffusion model is given by the score [Song et al. 2020b):
Vy log
(2)
By replacing p with the functional form proposed in equation 1, we show (derivation in Appendix C the score reduces to:
that
vy log p(y:((y-)1c)) = w. (-
(3)
where are the softmaxes over the energies Ej. This functional form for the update step is directly identifiable to the one we are using in our model, where (1) the weighting masks produced by our model correspond to the respective w,, and (2) the predicted noise for each view-dependent stream corresponds to its respective view-dependent score Vy(-B(yr-1- c()). With this equivalence, we establish that the parallel stream architecture of ViewFusion, combined with its softmax aggregation scheme, derives directly from reasonable assumptions on the functional form of the transition probability of the reverse diffusion process, namely that it should be niew-permttationninvaaaie nee-cconditioone--preiictooood through a mirture of experts model.
and more specifically that it should combine
4 Experimental Results
We evaluate our method on relatively small, but diverse dataset, NMR, consisting of variety of scenes and spanning multiple classes. We show that our model is capable of handling a wide variety of settings, while offering performances near or above the current comparable methods.
4.1 Dataset
Neural 3D Mesh Renderer Dataset (NMR). NMR has been used extensively in the previous works Lin et al. 2023 Sajjadi et al. 2022b Yu et al. 2021 Sitzmann et al. 2021 and serves good benchmark while keeping the computational footprint relatively low. The dataset is based on 3D renderings provided in Kato ct al. (2018) and consists of 13 classes (sofa, airplanc, lamp, telephone, vessel, loudspeaker, chair, cabinet, table, display, car, bench, rifle) from ShapeNetCore (Chang et al. 2015) that were rendered from 24 azimuth angles (rotated around the vertical axis) at a fixed elevation angle using the same camera and lighting conditions. The resolution of each image is x 64. In total, there are 44 different objects, split across training, validation and testing sets as follows: 31 k, k, 9 k. There are no overlaps in individual objects between the sets.
6

---

Published in Transactions on Machine Learning Research (05/2025)
4.2 Evaluation Procedure
In order to ensure good generalization performance both across multitude of classes well as variable input view count, we train our model by randomly picking the number of views that the model receives as conditioning, while simultaneously training across all available classes. At evaluation time we test our model both in fixed as well as variable input view count settings. Implementation details as well as training configurations are available in Appendix B.I. and the full evaluation procedure is described in Appendix B.2
B.I
We limit our model to receive anywhere between one and six input views at random during the training, since providing an abundance of views can make the problem overly easy and completely determined while also requiring significant increase in computing power to process all of the input views. The NMR dataset provides 24 views for cach object, viewed from the same clevation and rotated around the vertical axis. First, the number of views used for conditioning is uniformly sampled. Following this, we randomly select subset. of input views to be used for conditioning. We do not employ a specific sampling strategy, i.c. closer views to the target view are equally as likely as the further away ones.
However, even though we limit the training and inference to only up to six views, we also show that our approach is capable of generalizing to arbitrary, previously unseen view counts.
NB All the evaluations were performed the same model, which was limited to receiving between one and six views at random during training time.
4.3 Quantitative Results
We evaluate our model (Table 2 both singlo-view and variable-vien settings on commonly used metrics for novel view synthesis PSNR, SSIM (Wang et al.) 2004) and LPIPS (Zhang et al. 2018) on which we compare it to the most recent methods for novel view synthesis.
In order to ensure equivalence, we constrain our model to receiving only a single input view. This is effectively equivalent to turning off our learned weighting mechanism since all the weights scale to one after applying softmaz. Even in this constrained scenario, our approach is able to reach competitive performance in LPIPS when compared to the previous approaches.
Table 2: Quantitative Results. Comparison of evaluated metrics against comparable models for NMR. NB To ensure equivalence of the results when comparing against SRT and ViT for NeRF, our model is restricted receiving only a single input view at evaluation time. See Appendix B.2 for full evaluation procedure.
1PSNR
1SSIM
JLPIPS
Sitzzman al.
24.95
Pexe Yuu al.
2021
26.80
0.910
SRT Sajjadi al
22222
27.87
0.912
0.066
VIT
NeRF
2023
28.76
0.933
0.065
View Fusion (Ours)
single
26.0
0.883
0.053
View.Fusion (Ours)
up six
0.925
0.033
Additionally, we use the same model (without retraining) to compute the metrics for the setting where it receives anywhere between one and six views as input conditioning at random. By doing so, we reach an even better result in LPIPS, outperforming the current best approaches (to our knowledge), and on par with them when it comes PSNR and SSIM. This goes to show that our model is capable of effectively utilizing the availability of additional views. Better LPIPS results can be attributed to the fact that our model does not produce blur in areas of uncertainty (c.g. occluded arcas) thanks to its generative capabilities, unlike prior methods, which often suffered from blurry results when producing areas of uncertainty (Sitzmann et al. 2021 Yu et al. 2021 Sajjadi et al. 2022b). LPIPS also more accurately reflects human perception than the other two metrics.
We also explore how the performance scales the input view count incrementally increases. We observe that the model's performance consistently increases with the view count beyond the view count of six present.
7

---

Published in Transactions on Machine Learning Research (05/2025)
PSNR and S5IM w.r.t. the Input View Count
28.0
0.92
PSNR
SSIM
27.5
0.91
27.0
0.90
PSNR
26.5
0.89
SSIM
26.0
0.88
25.5
0.87
0.86
24.5
0.85
Input View Count
Figure 2: Performance w.r.t. the Input View Count. The model's performance consistently increases with the view count, even beyond the view count of six present training time.
at training time, as shown in Figure 2 In this setting we use a portion the test set for computational reasons.
4.4 Qualitative Results
In order to underline the flexibility of our approach, subject it to variety of different scenarios (Table D
Variable Input Length. One of the main advantages of approach is the ability to effectively make use of variable input view count, both during inference and training. Figure 3 Jshows that consistently able to produce high quality samples regardless of the imput view count. Additionally, thanks to the model being class agnostic, we use single model to produce the results across all of the classes
Adaptive Weight Shifting. By altering the target viewing direction, we show that the model shifts the weighting adaptively according to the informativeness of the input views that are provided (Figure Our woighting strategy allows the model to perform well even in scenarios where less informative views are given, that could normally act as a distraction. In Figure 4 we note that the views closest. to the largel view are selected by the weights, in accordance with our intuition that these views are most informative about the target view.
Severe Occlusion. Due the generative nature of our approach, the model is capable of producing several plausible views when required to generate target views viewed from direction that is occluded in the input views. Figure 5 shows outputs of the model conditioned on the same input view and diverse starting noise, reflecting scenarios where parts of the object present in the target view are heavily occluded in the input view.
Autoregressive 3D Consistency. Despite not imposing any explicit 3D consistency constraints, in Figure D we show that our approach is capable of maintaining 3D consistency through autoregressive generation even when provided solely with a single input view. We start by priming the model with a single input view, and incrementally rotate the target viewing direction produce novel views. During the autoregressive generation, we fully utilize the flexible context length by adding each consecutively generated view to the conditioning for producing the next view. By doing so, we ensure that the model is 3D consistent. with itself.
8

---

Published in Transactions on Machine Learning Research (05/2025)
x
Input Views
Gen.
G.T.
Figure 3: Variable Input Length. Our approach is capable of generalizing across different classes without retraining. The padded empty views are included for visualization purposes only and are not present during training or inference, as the model operates with an unordered, pose-free view collection of variable length within the batches.
Additionally, unlike the previous approaches which often suffer from error accumulations by the time they reach the last frame, model only clicits significant dissimilarities to the target when producing fully occluded parts (i.e. half way through the generative loop), which can not possibly be inferred from the initial input. Such behavior is caused by the the adaptive weighting putting more weight back on the initial view past the midpoint of generation and ensuring that the produced samples are consistent start to end.
Generalization to Unseen View Counts. The novel learnable weighting mechanism for composing the view contributions scales seamlessly to an arbitrary number views that even larger than the maximum view count presented during the training. In Figure we show that our model trained on up to six views sensibly extrapolates problem is clearly determined with that many views, it is important to note that the model is able to sensibly utilize each of the views' contributions to produce the target view, all while maintaining its performance.
when presented with collections consisting of upwards of 20 views. Even though the

---

Published in Transactions on Machine Learning Research (05/2025)
Input
Weights
Weights
Denoising
Denoising
Figure 4: Adaptive Weight Shifting. The model shifts its weighting adaptively based on the most informative input view w.r.t. the desired target output. Six views of the truck rotated by fixed angular increment are passed in, depending on the target view the model puts most emphasis on the closest views. Additionally, the model picks up the details from different views, e-g. the cargo bed and the back window of the truck are picked up from the fully rear facing input view in the example the left or, in the example on the right, the front hull area gets picked from the front facing input view.
Input
Gen.
Gen. 2
Gen.
Gen. 4
G.T.
Figure 5: Severe Occlusion. Our approach is able to handle severely underdetermined settings by generating variety of plausible view in cases where the target viewing direction is fully occluded. We prompt the model with the same front views several times generate a variety of plausible rear views.
10

---

Published in Transactions on Machine Learning Research (05/2025)
Input
Autoregressive Generation
Figure 6: Autoregressive 3D Consistency. Our approach is capable of maintaining 3D consistency through autoregressive generation even when provided solely with a single input view. We start by priming the model with a single input view, and incrementally rotate the target viewing direction to produce novel During the autoregressive generation, each consecutively generated view added to the flexible conditioning for producing the next view.
views.
Spr
Input Views
Gen.
G.T.
Figure 7: Gencralization to Unscen View Counts. In addition to taking an arbitrary number of input
views, the model performs exceptionally well even when presented with significantly more input views inference compared the maximum of six training time. Here condition our model trained on up six views on significantly larger number of views.
11

---

Published in Transactions on Machine Learning Research (05/2025)
5 Applications
This section explores potential application scenarios of ViewFusion based on findings of this work.
Generating 3D Representations. As shown previously in Section 4.4 our model is capable of autoregres- sively synthesizing views of object from all sides given only single input view. Such capability could be useful for wide variety of VR or AR applications, as well as when trying to build 3D representation of an object based on one or few images (e.g. for creating video game assets). This would, however, require pairing with an approach that goes from image to 3D representation domain, as our mothod currently operates only in the image domain.
Occlusion Prediction. Generative capabilities our model could be leveraged to generate multiple plausible views of an occluded object, making the approach particularly useful in situations where any kind of plausible view needed, even if it might not be ground truth, or in scenarios where the absolute correctness of the produced views is not of significant importance.
Dataset Augmentation. Recent findings of Abbas & Deny (2023) have shown that commonly used deep networks for image classification fail to classify objects correctly when they are presented in unusual poses (e.g. flipped upside down or rotated in the 3D space). Therefore, if successfully scaled up, our method could be leveraged to produce more realistic results which could then be used to augment the already existing datasets used to train large classifiers, with the goal of improving their performance in a wide variety of edge-cases.
6 Limitations and Future Work
Our current approach does not explicitly incorporate 3D semantics of scene, potentially posing a problem in situations where a quick, on-the-ft adaptation to a completely new, out-of-distribution scene is needed. When it comes to target pose conditioning, Appendix D. shows a promising step towards using relative canonical pose defined as the first input view, rather than defining it as the first, front facing view of the object at 09 Another limitation the trade-off between the generative power in underdetermined settings and the inference times for producing novel views, which scale linearly with the view count. Such scaling can make the model particularly slow when presented with significant amount of input views if the image resolution is significantly increased, especially when operating in the autoregressive mode. A potential way of fixing this would be to incorporate a well-established approach of performing diffusion in a latent space (Rombach et al., 2022) instead image space. To further increase the inference speed, DDIM (Song et al., 2020a] sampling strategy could be employed instead of DDPM. Furthermore, our method does not explicitly impose cointsaints to ensure 3D consistency due to the sLochastic nature of diffusion modelling. However, this can be partly alleviated by using autoregressive generation where each new view is also conditioned on previously generated views. Lastly, to unleash the full potential of our generative approach and to enable it to operate in real-workd scenarios, training on a more realistic and larger dataset, such as Objaverse (Deitke et al. 2023), CO3D (Reizenstein 2021 or MV MVImageNet (Yu et al., 2023) would be a good future direction. Relatedly, our method can easily be implemented on top of already existing diffusion-baase methods for novel view synthesis such as Liu
test our model on fairly limited and small NMR datasct. Therefore,
et al.
et al. 2023a
7
Conclusion
This work introduces ViewFusion, a flexible, posc-free generative approach for performing novel view synthesis using composable difTusion models. We propose a novel weighting scheme for composing diffusion models ensuring that only the most informative input views are taken into account for prediction of the target view, and enabling ViewFusion to adaptively handle an arbitrarily long and unordered collection of input views without the need to retrain. Additionally, the generative nature of ViewFusion enables it to generate plausible views even in severely underdetermined conditions. We believe that our approach serves
12

---

Published in Transactions on Machine Learning Research (05/2025)
valuable contribution when it comes to novel view synthesis, with potential of being applied to other problems as well.
Broader Impact Statement
We provide brief discussion potential impacts and considerations that approach entails.
Applications. As already outlined in Section 5 we believe that our generative model can be used for a wide varicty of purposes, such as various VR and AR applications, as well as when building 3D models of objects given one or few images.
Fake Content. This paper proposes generative method, which could potentially be used to produce images containing fake or misleading content. We believe that given the relatively small scale of our current model, it poses no immediate threat. However, in a scenario where the approach would be expanded to a significantly larger or a more realistic datasets, the risks could be mitigated by using the model a controlled environment.
Energy Consumption. We propose a method that requires a training procedure which can be computa- tionally expensive, particularly if applicd to larger dataset than NMR. Additionally, our current sampling procedure can require significant amount of computing power, especially if large collection of views is used as conditioning, Despite the potential computational footprint of our solution, our method offers unparalleled ficxibility without the need to retrain. Therefore, we believe that it still remains reasonably efficient, as training is by far the most expensive part of the procedure.
Acknowledgments
This work has been supported by a Helsinki Institute for Information Technology (HITT) grant to B. Spiegl, and an Academy of Finland (AoF) grant to S. Deny under the AoF Project: 3357590. We would like to extend our gratitude to Aalto Science-IT project for providing the computational resources and technical support.
References
Amro Abbas and Stéphane Deny. Progress and limitations of deep networks to recognize objects in unusual poses. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 37, pp. 160-168, 2023.
Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, Ian Simon, Brian Curless, Steven M Seitz, and Richard Szeliski. Building rome in a day. Communications of the ACM, 54(10):105-112, 2011.
Titas Anciukevičius, Zexiang Xu, Matthew Fisher, Paul Henderson, Hakan Bilen, Niloy J Mitra, and Paul
Guerrero. Renderdiffusion: Image diffusion for 3d reconstruction, inpainting and generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12608-1266 2023.
Titas Anciukevičius, Fabian Manhardt, Federico Tombari, and Paul Henderson. Denoising diffusion via amage-based rendering. arXiv preprint arXiv:2402.034 03445, 2024.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-ried 3d model repository. arXiv preprint ar Xiv: 1 512. 03012, 2015.
Yabo Chen, Jiemin Fang. Yuyang Huang, Taoran Yi, Xiaopeng Zhang, Lingxi Xie, Xinggang Wang, Wenrui Dai, Hongkai Xiong, and Qi Tian. Cascade-zerole zero123: One image to highly consistent 3d with self promppr ncarby views. In European Conference on Computer Vision, pp. 311-330. Springer, 2025.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ebsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. Proceedings of the IEEE/CVF conference on computer vision and pattern recognation, pp. 2023.
13

---

Published in Transactions on Machine Learning Research (05/2025)
Emilien Dupont, Miguel Bautista Martin, Alex Colburn, Aditya Sankar, Josh Susskind, and Qi Shan. Equivariant neural rendering. In International Conference on Machine Learning, pp. 2761-2776 PMLR, 2020.
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. Advonces in Neural Information Processing Systems, 33:6840 6851, 2020.
Hanwen Jiang, Zhenyu Jiang, Yue Zhao, and Qixing Huang. Leap: Liberate sparse-view 3d modeling from camera poses. arXiv preprint or 111::23310004400 2023.
Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of liffusion-basse generative models. Aduances in Newral Information Processing Systems, 35:26565-26511 2022.
Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3907-3916, 2018.
Byung-soo Kim, Pushmeet Kohli, and Silvio Savarese. 3d scene understanding by voxel-crf. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1425-1432, 2013.
Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. Xix preprint or eix:22911 1122911.0022 1.06214. 2023.
Kai-En Lin, Yen-Chen Lin, Wei-Sheng Lai, Tsung- Yi Lin, Yi-Chang Shih, and Ravi Ramamoorthi. Vision transformer for nerf-based view synthesis from a single input image. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 806-815, 2023.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-33 Zero-shot one image to 3d object. arXiv preprint (tt.:22003.11322 2023a.
Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview-consiitentt images from a single-view image. arXiv preprint ar XrX1i:2309.03440 2023b.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1): 99-106, 2021.
Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulo, Peter Kontschieder, and Matthias Nießner. Diffif: Rendering-guide 3d radiance field diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 4328 4338, 2023.
Norman Müller, Katja Schwarz, Barbara Rössle, Lorenzo Porzi, Samnel Rota Bulò, Matthias Nießner, and Peter Kontschieder. Multidiff: Consistent novel view synthesis from a single image. In Proceedings of the LEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10258 10268, 2024.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: with general conditioning layer. In Proceedings of the AAAI conference artificial intelligence, volune 32, 2018.
Visual reasoning
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiu preprint Xrx::22009. 14988, 2022.
Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Tabatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In Proceedings of the IEEE/CVI International Conference on Compuler Vision, pp. 19901-10911,, 2021.
Gernot Riegler and Vladlen Koltun. Free view synthesis. In Computer Vision-ECCo 2020: 16th Europeon Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part XIX 16, pp- 623 640. Springer, 2020.
14

---

Published in Transactions on Machine Learning Research (05/2025)
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent, diffusion models. In Proceedings of the IEEE/CV conference. on computer vision and pattern recognition, pp. 10684 10695, 2022.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Imago-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-10, 2022.
Mehdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetic, Mario Lucic, Leomidas J Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer. Advances in Neural Information Processing Systems, 35:9512-9524, 2022a.
Mchdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučié, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation transformer: Geometry-free novel view synthesis through set-latent scene reprosentations. In Proceedings of the IEEE/CVT Conference Computer Vision and Pattern Recognition, pp. 6229 6238, 2022b.
Mehdi SM Sajjadi, Aravindh Mahendran, Thomas Kipf, Etienne Pot, Daniel Duckworth, Mario Lučié, and Klaus Greff. Rust: Latent neural scene representations from unposed imagery. In Proceedings of the IEFE/CVT Conference on Computer Vision and Pattern Recognition, pp. 17297-17306, 2023.
Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. +2ro123++:: single image to consistent multi-view diffusson base model. ar preprint ar WrXh:22310151100 2023a.
Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. ar preprint arXiv:2308. 16512, 2023b.
Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evalustion rendering. Advances in Neural Information Processing Systems, 34:19313 119813-1332,,,. 2021.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using zonequilibrium thermodynamics. International conference on machine learning, pp. 2256 2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ar preprint urXiv:2010.02502 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arX.w:2011.1356, 2020b.
Shao-Hua Sun, Minyoung Huh, Yuan-Hong Liao, Ning Zhang, and Joseph J Lim. Multi-view to novel view: Synthesizing novel views with self-learned confidence. In Proceedings of the European Conference Computer Vision (ECCV), pp. 155-171 2018
Shitao Tang, Jiacheng Chen, Dilin Wang, Chengzhou Tang, Fuyang Zhang, Yuchen Fan, Vikas Chandra, Yasutaka Furukawa, and Rakesh Ranjan. Mvdiffusion++: A dense high-rcsolution multi-view diffusion model for single view 3d object reconstruction. In European Conference on Computer Vision, pp- 175-191. Springer, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in newral information processing systems, 30, 2017.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEFE transactions image processing, 13(4):600 (3(4)2600-612, 2004.
15

---

Published in Transactions on Machine Learning Research (05/2025)
Daniel Watson, William Chan, Ricardo Martin-Brualla, Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi. Novel view synthesis with diffusion models. orXiv preprinl or arXiv:2210.04 042688 2022.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one few images. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4778-4577 2021.
Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, al. Mvimgnet: A large-scale dataset of multi-viww images. Proceedings of the IEEE/CVT conference computer vision and pollern recognilion, pp. 9150 9161, 2023.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. Advances in neural information processing systems, 30, 2017.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang- The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586 595, 2018.
16

---

Published in Transactions on Machine Learning Research (05/2025)
A Diffusion Probabilistic Models
In this appendix we briefly introduce the technicalities of vanilla diffusion probabilistic models (Ho et al, 2020) based on Saharia et al. (2022) and using the same notations as the rest of the paper, but without the modifications introduced Section R
Diffusion probabilistic models consist of forward diffusion process during training and a corresponding reverse denoising process the inference time. The forward process consists in gradually adding Gaussian noise to the image over T timesteps, which can be described following:
p(y:lr-1) = Vorye,,, (1-o0)))
(4)
p (o) II (xx.lve11)
(5)
where an is the noise schedule hyper-parameter. The noise is added up to the point where it impossible to tell yr from the Gaussian noise. The forward process at each step can also be marginalized as:
ply:lo)) My v (1 - I)
(6)
where N = -I: ag.. By applying Gaussian parametrization of the forward process, obtain closed form expression for the posterior distribution of yr-1 given (yo.y.):
ply:-1lyo. -V(yc-1lu,,
(7)
in which / - 1-%, Yo + Var(
1-%t Yt and x² ( Having defined all the necessary aspects Iー
to perform diffusion, can separate it into the training and inference procedures. During the training, the model acts denoising function and learns to invert the forward process. That means that given the noisy image y, defined as:
= Vryo + VI- I
the goal is to recover the original, target image Yo- Therefore, the deep neural network model is parameterized fx ;,7), meaning it is conditioned the input x, noisy image ÿ, and the noise level at given time-step The objective of the training procedure is to maximize weighted variational-lower bound on the likelihood (Ho et al. 2020) and is given by:
E (xy)) Be, fo(x, V7y0 + V 76,
Pseudocode for the training procedure is given in Algorithm
Algorithm Training Forward (Noising) Process
1: repeat
2: (x, y(x,y)
3:
t~ Uniform(f1,...,
e~,,(
5: Take gradient descent step
6: V. fe(x, Varyo + /1 te, - c|
until converged
In order to perform inference, we want to perform a reverse process over T iterative refinement steps, i.e. the goal is to go from a randomly sampled Gaussian noise back to the image by iterative denoising. To do so, we first need to approximate yo by utilizing Equation g to obtain:
1 (yt V1 (xl(
8o
(10)
17

---

Published in Transactions on Machine Learning Research (05/2025)
Now substitute Yo into p(ye-1lyo. from Equation in order to parameterize the mean of po po(y:-1les, as: (x(x Var yz
1
vT folx,
az
(11)
By setting the variance of po (y x) to (1 - -a),, each step of the iterative reverse process can be computed 88:
Yt-1
oz
fo(x, Ye: 70) vI- OGEt
(12)
Pseudocode describing the inference procedure is given in Algorithm
Algorithm Inference Reverse (Denoising) Process
1: Yr (0,
2: for = T, T...., ....1.0 do
10 if 1, clse
4: y=-1 voi
Yc vi yol + vI- -z2
1-m
end for
6: return Yo
B Implementation Details
B.1 Architecture and Hyperparameters
Following Karras et al. (2022) linear noise scheduling is applied for the diffusion process spanning (le- 6, 0.01) over 2000 timesteps, both for training and inference.
We base U-Net architecture Saharia et al. (2022) with modifications listed in Section 3.1
We train the model using T2 loss computed between the loss prediction and true noise. Furthermore, a learning rate scheduler is used combination with Adam optimizer. The learning rate starts at 5e-5 with a 10k steps as a warm-up following which it peaks at 1c-4. The model is conditioned on one to six input views and trained for 710k steps using batch size of 112 and 4xV100 GPUs. The total training time using this sctup amounts to approximately 6.5 days.
Listing shows PyTorch pseudocode for aggregating the view contributions at each diffusion step, given an arbitrary, unordered and pose-free collection of input views.
inference time, we run the model for 2000 timesteps which takes around minutes and does not depend on the amount of views used for conditioning (as long as they in the memory) since the streams are treated as a batch. Using single 32GB V100, we are able process batch size of 28 with up to six conditional input views, meaning that our model is able to process up to 168 64x64 images at time.
B.2 Evaluation Details
In order to ensure consistency of our evaluation process given the stochastic generative nature of our model, and while maintaining a reasonably low computational footprint, we repeat the evaluation procedure several times using the same model. For singlo-view setting, we evaluate the same model three times over the whole test. dataset, by randomly picking an input view and an arbitrary target for each object. Table 2 reports the mean metrics of this procedure. We omit reporting standard deviations directly in Table 2 as they are orders of magnitudes lower than the metrics themselves, namely +3.18 for PSNR, 44.78c-4 for SSIM and +1.41e-4 for LPIPS, respectively. We perform a single run where the model receives up to six views, as it is significantly more expensive than the single-view setting and its results cannot. be directly compared to prior methods.
It important note that our evaluation procodure differs slightly from evaluation procedures of Sajjadi et al. 2022b): Lin et al. (023) where for randomly picked input view, all other 23 views are generated. We
18

---

Published in Transactions on Machine Learning Research (05/2025)
avoid performing such procedure as observe stable results are already obtained from the separate runs, but also to maintain a low computational footprint our sampling procedure can take significant amount. of time to be computed over the whole dataset.
Listing 1 PyTorch Pseudocode for Contribution Aggregation
create delimiter indices unstacking
outputs
# vieu_count of shape (B, 1) number of input views for each sample in the batch view dee__delimteer torch. cumsum (vien_count, 0) tollstt view dlew_delmmiters... insert 0)
noise_level = extract (self. gammas, t, (1shape= 1)) to
# prepare shapes of conditioning, noise, angles and levels;
10 # fron (B, 23, G, H, W) -> ((V_ + V_B), c, H, M); where v_n is the input view count for each sample 11 x_stacked torch. concatenate( [x_i[i, :idx] for i, 1dx in enumerat((ioo.cout]]]] dim=0, (xto
16 y_t_ stacked torch. repeat, interleave iet vieu_count, dim-0)
17
noise angle_stacked torch. repeat. reperrrrtetteeeeeeeeeeee intereeeeeeeeeangglil (angle, vieN_count, dim=0)
torch. reaererreeetttteleevveeee interleave (noise_level. vieu_count, dim-0)
18
10
20
unet_
self (
21 torch cat stackdd y_t_ s_t_stac,ed], dim-1),
22
23
noise_level_stane stacked,
24
)
25
all, logits unet untt_output [:. :3, ...J, unet_output::: [:, 3:, ...)
# weights and noise padded; shape (B, max (V_1, logits_padded = torch mn. utils. rnn (
V_B), c, H, W)
[ [ogits[idxi::s
for idx1, idx2 in pepprriee_dllimittrrst::.. viev_delimiters [1:1)
},
33
batch, first-True,
34
padding value- ('-iitt (*-inf"),
-inf padding becomes after softmax
35
36
weights softmax F. softmax (lagits (lagits_padded, dim=1) podde_ torch. nn. utils. rnn sequencc)
87
38
[
39
noise_ale all [1dx1:1dx21
40
for idx1, idx2 in zip woiew_dolimitors (1-1), viev_deliniters (1:))
41
}.
42
Batch_firat=Tie
43
)
noise_ noise_weighted = noise_padder veights_softant
noise = noise_ poise_neighted..o sum (dim=1)
C A Probabilistic Interpretation of ViewFusion
We hereby elucidate the design choices of the ViewFusion approach by means of a probabilistic interpretation. In the following, we will assume specific expression for the transition probability of the reverse diffusion process, where we will recover a weighted composition of per-view noise gradients.
We use the following notation:
1. yt-1 and ye, the images produced by the diffusion process at times - and t, respoctively;
2. / being the time variable;
19

---

Published in Transactions on Machine Learning Research (05/2025)
3. [x: being the set of N conditioning images, each detailing different pose of the object of interest;
4. Au being the conditioning target angle.
For brevity, we contain all the conditional information in tuple cf == Yx, Aw,
We want to model the transition probability of the reverse diffusion process
p(y--1lc').
Since {x:)= set, the expression of the probability p(ye-1/c;) should not depend on the order of the conditioning images, nor their number. We can consider single function E applied separately to each of the N views, and have these terms contribute to the final probability via summation. In formula,
N
p(y:-1/c") x Σ Exp(-E(yy-1.c)))
With this prescription, in accordance to Zaheer et al. 2017); the probability now a function that does not depend on the order of the N views, and can be applied to any number of views (in their notation, we have o= ex) and p= Id). In the machine learning context, this formulation is sometimes referred to as a Minture of Experts (MoE), where the experts here correspond to the different input views, and the output, is the probability distribution of the reverse diffusion step. This is also known, in the context of physics, as Boltamann distribution. The analogy implies that the N views act as states of the system, each of them with an associated energy, and the probability of each state is proportional to the exponential of the negative of this energy. These states then combine their influence on the reverse diffusion process paths via their summation.
We are now interested in the score,
Vy lygpp((y-)lee).
this is the quantity which the model is trained to prodict, following Song ct al. (2020b) We perform the computations directly:
Vy log logp(y--lcc) = Vy log
expl ( - F((+--1,c)))
(13)
1=1 Vy Σ N
exp( EEyy, c()
(14)
Σ exp( -E(y,-1, ct)
expl - 1,,
(15)
=1 (xxp-FF(y-11:0 c())
exp(-E((y-1, Σ exp(-E(yy-1, cf))
Vy(-F(yy-1.0)))
(16)
where we recognize the softmax function. For case of writing and reading, we call
exp(-E(yx-i,ct))
W; N Σ 1 exp((EE(y_x)))))
(17)
Then
vy log poyp(yy11cc)) Σ w, W.
(18)
and so we recover the weighting of the contributions of each conditioning image.
While, in principle, knowledge of E enough to completely characterize p (as cach of the w, expressible function of all the N energies), in this work we predict separately the weights w, and the gradients of the energies (also called scores).
20

---

Published in Transactions on Machine Learning Research (05/2025)
D Additional Results
D.1 Relative Canonical Pose
Defining the canonical view the first, front facing view of the object at 0" as given by the dataset assumes similar objects to have the same canonical pose, resulting in close relation between the semantics and the camera pose. Instead of assuming the canonical view to be defined by the dataset, here we test our model's performance when canonical pose is defined relative to the first input view. We assume the first of the input views, 11, is the reference view and define A& as angular disparity between the target view and the first input view V1. The model trained from scratch under this premisc.
We observe that approach handles this case successfully well, producing sensible results (see Figure 8). However, the metrics computed validation subset of NMR dataset lower: 31.17 (canonical) 26.54 (relative) in PSNR and 0.95 (canonical) vs. 0.89 (relative) in SSIM. We don't compute LPIPS at validation time. The comparably lower performance can be attributed to this setup posing considerably more difficult task.
Input Views
Gen.
G.T.
Figure 8: Relative Canonical Pose. Instead of defining Ay as the front facing view of the object as defined in the dataset we assume the first of the input views, VI, is the reference view and define Arb as angular disparity between the target view and the first input view V1.
21

---

Published in Transactions on Machine Learning Research (05/2025)
D.2 Generalization to Unseen Objects
We test our method's generalization to previously unseen classes by training the model from scratch and omitting the car class entirely from the training set. Then, we run autoregressive inference using an image of a car and show that the model is able to rotate car up to 180 fairly reasonably, but beyond that it collapses (starting from rear facing view) as shown in Figure 9 While the results are far from perfect, it is interesting to observe that the model still captures relatively meaningful information from images of unknown objects, especially given that the entire car class was not present the training set.
Input
Autoregressive Generation
Ground Truth
Figure 9: Autoregressive out of Distribution Generation. We first omit the car class entirely from the training set. We then once again prime the model with a single input view of a car, and incrementally rotate the target viewing direction produce novel views. During the autoregressive generation, each consecutively generated view is added to the flexible conditioning for producing the next view.
D.3 Weighting Scheme Ablation
Here we justify the design decision to use weighting mechanism by performing a set of ablation experiments.
In the first experiment, we train the model from scratch using simple averaging to combine the noise contributions and completely omit the weighting scheme. We observe that the averaged model is consistently trending lower than the weighted model on the validation metrics as shown in Figure I0
PSNR Weighted vs. Averaged
SSIM for Weighted vs. Averaged
31
Weighted Averaged
Weighted Averaged
30
29
0.92
28
SSIM
27
0.88
26-
0.86
24.
0.84
0.82
200000 250000 300000 350000
400000 450000 500000 550000 600000 Step
200000
250000
300000 350000 400000 450000 500000 550000 600000 Step
Figure 10: Weighting vs. Simple Averaging. We train the model from scratch using simple averaging to combine the noise contributions and completely omit the weighting scheme. We observe that the averaged model is consistently trending lower than the weighted model on the validation metrics, with a more significant difference PSNR.
In the second experiment, we simply disable the weighting scheme of an already trained model and just averaged the noise contributions. We note that our model's capability to operate well in a single view setting, which is equivalent to the weighting scheme being off, also ensures that each of the individual streams in a multi view setting produce sensible result on their own. Then, the weighting scheme helps with picking out the best information from each of the streams meaning that simple averaging should still produce sensible
22

---

Published in Transactions on Machine Learning Research (05/2025)
results. Replacing the weighting with averaging in already trained model also results in performance drop on validation set both in PSNR 27.38 (weighted) vs. 25.88 (averaged) as well as in SSIM 0.90 (weighted) vs. 0.88 (averaged).
23


Reviews:
[{"review_id": "8pj65rWWwY", "reviewer": "Reviewer_ahEw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The underlying idea of combining the information from several instances of a single-image NVS diffusion model during sampling is elegant, and to the best of my knowledge novel in this application\n- The specific approach of learning a noise-weighting at each pixel (determining how much each single-view u-net affects the output) is also interesting (and again novel in this setting)\n- Quantitative results on ShapeNet renderings show the proposed approach exceeding three baselines (Light Field Nets, pixelNeRF, SRT, ViT-for-NeRF) on the standard reconstruction metrics (PSNR, SSIM, LPIPS)\n- Qualitative results appear high quality, with sharp and plausible synthesised novel views that appear consistent with the input image\n- There are also results demonstrating that the model behaves generatively, i.e. can sample diverse (but still plausible) completions for ambiguous, self-occluded regions\n- There is a fairly detailed discussion of limitations\n- Overall the text is clear and readable; the paper is well structured; the figures are appropriate\n\n**Weaknesses**\n\n- Experiments are only conducted on ShapeNet (specifically, the renderings from NMR); this is a rather small and simplistic dataset by modern standards. It would have been good to see experiments on a more challenging dataset – either more diverse synthetic data (Objaverse or similar), or more complex single-class data (CO3D, MVImageNet or similar)\n- There are no ablation experiments, or experiments using variations of the proposed model, to justify its design decisions. In particular, the novel fusion approach is interesting, but it'd be good to see quantitatively the benefit of using the predicted weights (rather than equal weighting), and also to the see performance of an identical architecture (on identical data) but using a more 'traditional' fusion approach (e.g. concatenating a fixed number of input views, or performing cross-attention)\n- There is no qualitative comparison against the baselines, making it difficult to intuitively understand how the proposed method performs in comparison\n- A key claim (and motivation for the architecture) is the support for varying numbers of views; however this is only backed up with qualitative evidence in the evaluation. It would be good to see quantitative results (maybe a graph?) on how view-synthesis performance varies with the number of views, with both fewer and more views than used during training. Presumably there is some 'overfitting' to the view count seen during training, even if the model adapts relatively smoothly to other counts?\n- The claim in sec. 5 that the method supports generation of explicit 3D representations (rather than just independent images that might be 3D-inconsistent across output views) is justified only by referencing qualitative results on autoregressive image generation. To justify this claimed application properly, an attempt should be made to actually perform a reconstruction on these images (e.g. using NeRF) and render this from interpolated viewpoints\n- The same is true of the claim that the method can be used for data augmentation – this may be true, but there is no evidence – and synthetic shapenet renderings can be created almost for free, so on the specific domain where the method is demonstrated, this does not seem useful\n- The baselines are all non-generative – being deterministic, they will necessarily tend to make blurry predictions in uncertain regions, and thus the comparison is not entirely fair. It'd be good to include some existing generative approach as an additional baseline\n- Although the approach is generative, there is no quantitative measurement of diversity – there are three qualitative examples in fig 4, but nothing more", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a generative method for novel view synthesis based on diffusion models. The base model is a standard diffusion u-net that predicts one output view given an input view and relative pose. However the method fuses information from several model instances during both training sampling, allowing varying numbers of input images from different viewpoints, and avoiding any order dependence. Specifically, a weighting is predicted in addition to the noise, and used to determine the influence of different input views at each output pixel. The method is evaluated on the NMR dataset of ShapeNet renderings, where it achieves better performance than three baseline approaches.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Justify or weaken the not-fully-supported claims mentioned above under 'weaknesses'\n\nAdd qualitative results for baselines\n\nIf possible, include some ablation / variant experiments to more clearly justify the design decisions made in the proposed approach", "broader_impact_concerns": "There is a broader impact statement present; however this work does not raise any specific concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "D70NZf8ztQ", "reviewer": "Reviewer_duNQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n1. The problem of processing an arbitrary number of input images for novel-view images is interesting.\n2. The idea of combining the predicted noises from multiple zero123 models is simple but plausible.\n\nWeakness:\n1. Some claims are not accurate: the proposed method still requires pose information to define the pose difference $\\Delta \\psi$. However, the paper claims the method is able to handle pose-free images.\n2. The results of experiments are not impressive enough. The results all are based on synthetic data while the original Zero123 shows the ability to rotate a 2D object even in an oil painting style.\n3. The writing of the paper could be improved. The first section of the method describes lots of implementation details but what we need to understand is the high-level idea of the paper. The introduction is not a good overview of the proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a new model to generate a novel view image of an object from an arbitrary number of images of this object. The key idea is to adopt multiple Zero-123-like architectures to process each input view. Then, the predicted noises are averaged by a predicted weight for the denoising. In this scheme, the proposed method is able to process an arbitrary number of input views to generate novel view images.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Improve the writing. The key idea of this paper is not very complex. I think we can say that the proposed method could be a combination of multiple Zero123 models to denoise a single-view image.\n2. I would suggest directly fine-tuning Zero123 on Objaverse and demonstrating some generalization ability in real-world examples instead of just synthetic data.\n3. If we use some input views, then it is possible to directly apply NeRF/3DGS. Some recent works like GaussianObject are able to reconstruct the object in just 4 views.\n4. If we generate images in an autoregressive manner, how the errors accumulate in the whole process is unknown. It would be better to check the 3D reconstruction quality so that we can better see the 3D consistency of the proposed method.\n5. Many related works about multiview diffusion are not discussed like MVDream, SyncDreamer. And some sparse-view reconstruction methods are not discussed either, like Instant3D.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GzkU7YNryj", "reviewer": "Reviewer_yyPb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. It allows the input of unordered and pose-free collections of images with variable length, showing the advantage compared to the NeRF-based baselines.\n2. The method is simple and direct. It may be easy to be applied in other works.\n\nWeaknesses:\n\n1. The paper lacks discussions with some relevant works like Diffusion-based [1] and NeRF-based [2]. Especially,[1] proposes very similar idea to this paper, which also achieves novel view synthesis with unordered and pose-free collections of images with variable length by fusing the input views. It should be revisited whether the idea and declared contributions are still interesting in the field.\n\n2. The evaluation is somewhat weak. First, the method is just evaluated on NMR with a resolution of 64x64, of which the data diversity is far from the prevailing datasets like Objaverse and Google Scanned Object. And the resolution is also much lower than the prevailing setting of 256x256 and 512x512. It is doubtful whether the proposal is scalable to larger data scenarios. Second, the baselines are somewhat outdated. The newest baseline in the paper is ViT for NeRF, which is released on arxiv in 2022.07. It's expected to verify if the performance is competitive with the advanced methods like mentioned in 1. \n\n3. I'm curious about the solution for the different camera coordinates between scenes. In 3.1, it said, \"∆ψ is the angular disparity between the target view and the canonical pose of the object, i.e. the front facing view of the object as defined in the dataset\". That's to say, the current solution seems to hypothesize the canonical pose of similar objects to be the same, like all the cars should have a same canonical pose like facing the front of the car, while the canonical pose of a table may be from another specific angle. This defines a tight relation between the semantics and the camera pose. However, as the method is pose-free, it should be learned by the model. This is weird, which heavily restricts that all the scenes in the training dataset have a well-calibrated canonical pose. And as the pose is ambiguous, the model seems not to be very robust for out-of-domain semantics, which makes it hard even to specify a target view.\n\n[1] Tang, Shitao, et al. \"Mvdiffusion++: A dense high-resolution multi-view diffusion model for single or sparse-view 3d object reconstruction.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.\n\n[2] Jiang, Hanwen, et al. \"LEAP: Liberate Sparse-View 3D Modeling from Camera Poses.\" The Twelfth International Conference on Learning Representations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a new multi-view diffusion method to synthesize novel views from pose-free and length-varied image series. The main contribution is to predict the noise as well as a weight according to each input image and the target pose, and use Softmax to merge these noises for the current timestep. Experiments on NMR datasets show that it gets competitive results compared to some generative NeRF-based methods.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Major:\n\n1. Add discussions and comparisons with the advanced relevant methods.\n\n2. If available, add more evaluations with different datasets and resolutions to verify the scalability.\n\n3. Clarify the handling of camera poses.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Two of the three reviewers recommended acceptance, describing the approach as elegant, interesting, and simple. The reviewers also indicate some skepticism regarding ShapeNet-only experiments and missing competition with industry-style approaches that are more resource-intensive, but they also note that the paper is very direct about its limitations -- even stating some within the abstract, which is rare and useful. The AE sides with the majority and recommends acceptance without further review necessary, but the authors are encouraged to integrate the work presented in the rebuttal, as the reviewers found these additions helpful.
Decision: Accept (poster)