Introducing 3D Representation for Dense Volume-to-Volume Translation via Score Fusion
Authors: Xiyue Zhu
Dou Hoon Kwark
Ruike Zhu
Kaiwen Hong
Yiqi Tao
Shirui Luo
Yudu Li
Zhi-Pei Liang
Volodymyr Kindratenko
Venue: ICML 2025 poster (2025)
Keywords: Diffusion models
3D medical image generation
video generation
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: We find it hard to train 3D volume translation models in inverse problems that require high accuracy, such as super-resolution. We  effectively introduced 3D representation by ensmbling 2D models' results with a 3D model.

Abstract: In volume-to-volume translations in medical images, existing models often struggle to capture the inherent volumetric distribution using 3D voxel-space representations, due to high computational dataset demands. We present Score-Fusion, a novel volumetric translation model that effectively learns 3D representations by ensembling perpendicularly trained 2D diffusion models in score function space. By carefully initializing our model to start with an average of 2D models as in existing models, we reduce 3D training to a fine-tuning process, mitigating computational and data demands. Furthermore, we explicitly design the 3D model's hierarchical layers to learn ensembles of 2D features, further enhancing efficiency and performance. Moreover, Score-Fusion naturally extends to multi-modality settings by fusing diffusion models conditioned on different inputs for flexible, accurate integration. We demonstrate that 3D representation is essential for better performance in downstream recognition tasks, such as tumor segmentation, where most segmentation models are based on 3D representation. Extensive experiments demonstrate that Score-Fusion achieves superior accuracy and volumetric fidelity in 3D medical image super-resolution and modality translation. Additionally, we extend Score-Fusion to video super-resolution by integrating 2D diffusion models on time-space slices with a spatial-temporal video diffusion backbone, highlighting its potential for general-purpose volume translation and providing broader insight into learning-based approaches for score function fusion.

Introducing 3D Representation for Dense Volume-to- Colume-to-Volume Translation via Score Fusion
Xiyue Zhu Dou Hoon Kwark
Ruike Zhu'
Kaiwen Hong' Yiqi Tao Shirui Luo? Yudu Li¹
Zhi-Pei Liang"
Volodymyr Kindratenko 12
https://score-fusiin thub.
Abstract
TPOM
Ours
(4) Model Stracture Conpansuu with IPDM
In volume-to-volume translations in medical im- ages, existing models often struggle to capture the inherent volumetric distribution using 3D voxel- space representations, due to high computational dataset demands. We present Score-Fusion, a novel volumetric translation model that effec- tively learns 3D representations by ensembling perpendicularly trained 2D diffusion models in score function space. By carefully initializing our model to start with average of 2D mod- els as in existing models, we reduce 3D train- ing to fine-tuning process, mitigating compu- tational and data demands. Furthermore, we ex- plicitly design the 3D model's hierarchical lay- ers learn ensembles of 2D features, further en- hancing efficiency and performance. Moreover, Score-Fusion naturally extends to multi-modalin settings by fusing diffusion models conditioned on different inputs for flexible, accurate integra- tion. We demonstrate that 3D representation essential for better performance downstream recognition tasks, such as tumor segmentation, where most segmentation models are based on 3D representation. Extensive experiments demon- strate that Score-Fusion achieves superior accu- racy and volumetric fidelity in 3D medical image super-resolution and modality translation. Addi- tionally, we extend Score-Fusion to video super- resolution by integrating 2D diffusion models on time-space slices with spatial-temporal video diffusion backbone, highlighting its potential for eureral-purpose volume translation and providing broader insight into bearning-basee approaches for score function fusion.
M-...
Difictionn Medele
Oethation Nodels
Weighted Average
Vasalize Coupasion Supar
GT
Output
32:33
t TRDM
4)-
13:11
TPDM
TPOR
Cark
TrOR
Cark
TPON
Figure 1: Comparison between TPDM (left) and Score- Fusion (right). Score-Fusion learns to ensemble pre-trained diffusion models with 3D model, effectively utilizing 3D representations. Our model thus shows better 3D realism and demonstrates superior accuracy and realism metrics.
1. Introduction
Dense volume-to-volume translation is critical vol- umetric medical imaging, such as magnetic resonance imaging (MRT) and X-ray computed tomography (CT). It addresses various inverse problems of image reconstruc- tion (Hai-Miao Zhang, 2020; Liang et al., 2020; Wang et al., 2020), handling sparse (Kerstin et al.; Mardani et al., 2019), limited (Chung et al., 2023b; Lyu et al., 2020), and/or noisy (Chung et al., 2023b; Yang et al., 2019) imaging data. It also supports image synthesis, such as multi-contrast MRI (Zhang al., 2022: Dar al., 2018: Shi al., 2021; Wolterink al., 2017), CT-ulrrasound (Vedula et al., 2017). and MR-histopathology (Leroy et al., 2021). In addition, by using the time axis as the third axis, video data can be considered as dense volumes. Therefore, dense volumetric translation can be useful video data. Potential applica- tions include video super-resolution (Li al., 2023a; Zhou
Uniieersity of Illinois Urbana-Champaign "atioona Center for Supercomputing Applications. Correspondence to: Xiyue Zhu <xiyuez2 edu..
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Introducing 3D Representation for Dense Volume-to-Volun Translation via Score Fusion
et al., 2024; Li et al., 2025), video denoising (Zhang al., 2023b; Fu et al., 2024), and video editing (Chai al., 2023; Feng et al., 2024).
two-stage training strategy: (1) It first trains multiple 2D dif- fusion models (Saharia et al., 2021) in perpendicular planes. (2) It then utilizes 3D fusion network to produce the final translation in each diffusion step. Meanwhile, the Score- Fusion model is designed to start with a weighted average of 2D models following TPDM(Lee et al., 2023), which reduces 3D training to a fine-tuning process. The hierarchi- cal layers of the 3D model are also reformulated learn the ensemble of 2D features, further enhancing training efficiency and performance. Additionally, by ensembling diffusion models conditioned on various input modalities, Score-Fusion seamlessly supports multi-modality fusion.
3D voxel-space representation has great potential to play a critical role in volume-to-volume translation tasks for various reasons. Firstly, medical images are inherently 3D- dense volumes. Using 3D representation enables us to di- rectly model the entire 3D distribution. Additionally, the 3D latent diffusion model (LDM) is common technique to ac- celerate diffusion models. However, compared to LDMs, the 3D voxel-space diffusion model maintains better features in high-freyuency details, which are essential in inverse prob- lems that require highly accurate prediction, such as super- resolution. Moreover, volumetric translation models enable numerous downstream image processing tasks, ranging from image reconstruction (Meng et al., 2021) to analysis (Akrout et al., 2023; Fernandez et al., 2022)). Most models in anal- ysis tasks, such as tumor segmentation( gggmentaaion(Hatamizzaee et al., 2022), are trained with 3D volumes using voxel-space 3D representations. Therefore, volume translation models using 3D representation may generate images that are more accu- rate when processed with such downstream task models.
Similarly, we find that time-space planes, i.e. x-t and y-t planes, can be crucial in video modeling, as demonstrated in previous works (Al-Sumaidace et al., 2023; Otroshi- Shahreza et al., 2022). However, they remain underexplored in current video diffusion models. By extending Score- Fusion to video super-resolution, we trained additional 2D diffusion models on time-space planes and successfully demonstrated that introducing representations learned from time-space planes can enhance video super-resolution.
The mathematical intuition of Score-Fusion lies in the prop- erties of diffusion models and their associated score func- tions (Song et al., 2021). As the score function models the gradient of the probability distribution, it is inherently suit- able for an ensemble. Previous works (Chen et al., 2024; Chung et al., 2022) have also demonstrated this by showing strong performance with straightforward weighted averag- ing of score functions. Consequently, Ccore-Fusion replaces the weighted averaging process with 3D network, effec- tively incorporating 3D representations. To the best of our knowledge. Score-Fusion performs diffusion model fusion in the score function space, which provides new insights for diffusion model ensembling. Score-Fusion can also func- tion as a plug-and-play mechanism compatible with various combinations of 2D models from previous studies (Chung et al., 2022; Chen et al., 2024; Li et al., 2024), consistently delivering performance improvements across various 2D backbones.
However, previous works (Dorjsembe et al., 2024; Lee et al., 2023) highlight substantial challenges to utilize 3D repre- sentation in volumetric translation model for their increased demands for computational resources and large datasets that are costly to acquire in medical imaging. To the best of our knowledge, within the domain of 3D medical image inverse problems, no fully 3D models have demonstrated su- perior accuracy over 2DD-baeed models due to these practical limitations. Weight inflation (Liu et al., 2023b) proposes de- signing a 3D network of the same size as 2D models, which is a promising approach given the rich 3D context and strong pre-trained 2D models. However, this is generally infeasi- ble with existing 2D diffusion models (Saharia et al., 2022), which require around 500 GB of GPU memory for batch size 1 training and potentially an extremely long training time. As result, current 3D diffusion models (Dorjsembe et al., 2024) are designed to be much smaller with insufficient capacity to demonstrate competitive performance in inverse problems. Recent advances in volume-ta toluuue transla- tion have introduced methods that combine perpendicular 2D diffusion models (Lee et al., 2023; Chen et al., 2024), achieving improved accuracy and volumetric consistency. However, these methods cannot model the distribution of the entire volume since the generated images are produced by an averaging of the 2D networks without 3D representations, resulting in limited realism in 3D.
Score-Fusion has been evaluated in various MRI image pro- cessing tasks on the BraTS (Baid et al., 2021) and HCP (DC et al., 2013) datasets, including image super-resolution and modality translation. Our experimental results demonstrate that Score-Fusion performs superior volume translation over current taae-ooftthe-art (SoTA) models in accuracy, realism, and downstream task performance. By learning to ensem- ble perpendicular 2D models conditioned on different input modalities, Score- shows strong performance with- out retraining new 2D models. Moreover, Score-Fusion has been extended to video super-resolution on the vide- oLQ dataset (Chan al., 2022). Score-Fusion demonstrates the importance of time-space plane representation in video
To effectively introduce 3D representations into volumetric translation, we present Score-Fusion, a pioneering model for volumetric translation that directly and effectively cap- tures the distribution of 3D volumes. Score- Fusiin adopts
2

---

Introducing 3D Representation for Dense Volune-to-Volume Translation via Score Fusion
super-resolution by showing improvement in temporal con- sisteney and realism.
diffusion models have become useful research topic. Most current works use weighted averages ensemble different branches of diffusion models (Cheng et al., 2023; Lee et al., 2023). Collaborative Diffusion (Huang et al., 2023) is a learning-based ensembling method for diffusion; it trains an auxiliary model to estimate the confidence score for each branch of diffusion and ensemble based on the score, In this work, our approach uses information across all branches of diffusion models. This under-explored approach provides new insights for advancing diffusion model ensembling.
2. Related Work
3D medical image generation and translation. Attempts have been made to generate dense 3D volumes for medical imaging. Direct 3D-based diffusion models (Dorjsembe et al., 2024; Liu al., 2023b) face difficulties due to high computational and dataset demands, resulting in moderate accuracy in tasks like super-resolution. Patch-wise, slice- wise, or cascaded generation strategies have been utilized to accommodate high-diman sigh-dimensional data( Uzuuova et al., 2020). However, in such models, initial inaceuracies in the low- resolution base are propagated during the refinement stages and the patch-based refinement often struggles with main- taining global consistency across the image. Latent 3D models (Dorjsembe et al., 2024; Zhu et al., 2023a; Khader et al., 2023) have been exploited to compress the 3D data into a low-dimensional latent space and train diffusion mod- els with this compressed latent space. However, the process of reducing dimensionality also has high computational and dataset demands and can lead to substantial reconstruction errors. Sequential slice generation from autoregressive mod- els (Peng et al., 2023; Zhu et al., 2024) or simultaneous multiple slice generation may mitigate this issue of error accumulation over slices. Yet, these approaches suffer from challenges in maintaining coherence for long-range struc- tures. More related to our approach, integrating multiple 2D models trained along perpendicular directions is a promis- ing approach. TPDM (Lee et al., 2023) first proposes to combine two perpendicular 2D diffusion models to improve 3D imaging, where the weighted average of scores from pre-trained 2D models estimates the score function of 3D model. Building on this conCept,TOSM (Li et al., 2024) and MADM (Chen et al., 2024) further improve the model performance by including 2D models in all three directions and using multiple consecutive 2D slices in 2D models. These models generate highly accurate results by effectively leveraging the high-resolution information in each 2D plane.
3. Score Fusion in 3D
Problem formulation. We formulate volume-to-volume translation as conditional sampling problem. Specifically, let x E R& xbyxby be an input medical image volume, and let y X be the corresponding target volume to be generated, where h,, b2, and by denote the spatial dimen- sions of the volume. Our objective is to learn conditional distribution p(y x) that accurately captures the volumetric structure in three dimensions. The input volume x may consist of low-resolution data and/or a different imaging modality.
3.1. Overall Framework of Score-Fusion
We designed the Score-Fusion as conditional dif- fusion model. Following DDPM and Palette (Ho et al., 2020; Saharia et al., 2022), our model gradually adds Gaussian noise to the target image in the train- ing dataset during the forward or diffusion process as: q (t-)):-1) = -N (yrs v1 - B, (3)) IT_ ((y:ly.-1), where yo ~ q(y) is the target im- age and Be is the variance of noise added at timestep (. The forward process produces sequence of increasingly noisy variables .I.... yr, after sufficient noising steps, the process reaches a pure Gaussian noise, i.e., yr ~ NNO, I).
((yTl9o) =
During training. our denoising diffusson model, eg (Yt,,,)), (), is trained to predict the noise added into y. given yr- Demonstrated effective in existing work (Saharia et al., 2021), the sampling process can be guided by concatenating the noisy image Yt with condition x. The loss used to optimize es (yt,, x, t) is: [ee(varyo 0, = I (1 1 Be), and we sample yo, ~ p(yo, e <~ 10.
Model ensembling. Ensemble techniques, which include bagging, boosting, and stacking, have been further devel- oped through specialized algorithms like Random Forest, AdaBoost, XGBoost, and Mixture of Experts (MoE). These techniques also demonstrated remarkable efficacy in medi- cal image analysis. particularly in brain tumor segmentation (Hatamizadeh et al., 2022; Zhou et al., 2019), hypertension detection (Fitriyani et al., 2019). and kidney stone identifica- tion (Kazemi& & Mirroshandel, 2018). More recent research underscores the potential of ensembles as an effective strat- egy for scaling up large models(Jiang et al., 2024).
ate, 2, t) - ell where
During sampling in the reverse or generative process, we also follow Palette (Saharia et al., 2022) to generate im- ages by iteratively removing the added noise in the se- quence yr-1, y1, Yo. from standard Gaussian prior yr ~ I). In addition, inspired by (Song et al., 2022; Chung et al., 2023a; Song et al., 2024), we ex- plore self-consistency for solving inverse problems. More
Diffusion model ensembling. Recently, diffusion models have shown great success. The ensembling methods for
3

---

Introducing 3D Representation for Dense Volume-to- Volume Translation via Score Fusion
Alignment Modules
Alignment Module 1
Hetarchcal
Algemmnt Nocause
30 t
20 oftixer on model
Hierarchical 2D feature
2a Nuturas
3Dfukkn
Dessiang Rosati Piok
Sindd lepat
Alignment Module
Sirreer Tured
Denotsing Staps
Alignment Module n
Figure 2: Overview of the Score-Fusion. At each denoising step, two pre-trained 2D models provide initial estimations of the scores a slice-wise manner. Subsequently, a 3D network learns to integrate these estimations via 3D representation extracted from 3D input and aggregated 2D scores. In addition, the 3D model is initialized to output an average of 2D scores. Moreover, The 3D network layers also reformulated to learn ensemble of aggregated and aligned 2D features. These designs accelerate and stabilize the 3D training process.
specifically, in each diffusion sampling step, we estimate noise with our denoising model eo ( x. t). Therefore, we have the estimated jo(() (4) at the t-th denoising step as Ju(t): = (y:- v1 ar (Yt. x, / In inverse prob- lems, conditional input æ is obtained through known linear degradation process a = Ay. At each diffusion step, we project the estimated (t) to a plausible Go(t), such that
3.2. 2D Score Models
2D(b) 2D(a) and Egb. are trained on The 2D diffusion models, Ego two perpendicular slices of the volumes using standard conditional diffusion framework (Saharia et al., 2021). We take gradient descent steps on the following objectives for both 2D diffusion models during training:
= Ago. by (o( <- (t) AT (AAAT) -1 (A) - xr). We provide more details in Sec. H. After the consistency projection, we obtain y1-1 by adding noise back: y1-1 =
2D(a) V Na soa (voli, d, : a/ :).0) el2
(I)
V vool|
20 (y ( m 1.),
6b
(ot-10(( + VT 01-66
Here, i and j are the indices for the slices along two per- pendicular planes, which are sampled uniformly: ~ (0,
Score-Fusion Models. The key component of this work lies in our denoising network eg (3r. x. t). In particular, our model Eg consists of two 2D diffusion denoising models, 2D))) and 3D diffusion denoising model, C0 with 0a. 0g. and 03D being their trainable parameters, respectively. The 3D network is conditioned two 2D diffusion models, trained to capture 2D image distributions along orthogonal planes to provide complementary views of the volumetric data. The 3D network is carefully initialized to start with a weighted average of 2D networks' estima- tion, such that Score-Fusion has the same performance with TPDM(Lee et al., 2023) before any 3D training. This design effectively constrains the 3D model, reduces the 3D training to fine-tuning process, and thus promotes faster and sta- bilized training convergence. Additionally, the hierarchical representations from the 2D models are introduced to the layers in the 3D model with alignment projection layers. In this way, the 3D model's layers are reformulated to learn an ensemble of pre-trained 2D models' representations, in- stead of learning representations from scratch. Therefore, the training of the 3D model is further accelerated and stabi- reference. We refer to this hybrid 2D/3D volumetric generative model as Score-Fusion Fig. 2 provides a schematic overview of Score-Fusion.
j ~ Uniform(0, ..., b3). After proper training, the high-capacity 2D model can provide decently accurate estimation of e for every volume slice.
2D(a) and Cgb
3.3. 3D Fusion Model
The 3D ensembling model, <0 is trained to fuse the pre- trained 2D diffusion models to capture the desired volu- metric image distributions. In this stage, we first obtain the inference results, yr2D(a) and y/2(()) from the 2D dif- fusion models, Ega sliced directions:
2D(a)
by iterating through the and Egb
2D(b)
y 2D()) [, E
2D(a)
(y [. i, als (,,,)) for i [0, b2)
y 22())) [.,
Eab (yi J. alt (...t) t) for j [0, by)
20(b)
(2)
During training of the 3D diffusion model, both 2D mod- els return Y's, which contains the predicted score and a hierarchical feature map of the model: }2D((a) (22D(a) F2D(o)), 222 (22D(0) F2D(b1).
lized by using the aligned 2D representation
The 3D model is designed to learn an ensemble of multi- ple 2D models' score estimation with 3D representation. Specifically, at each diffusion step, the 3D model takes as input the original image æ. the noisy intermediate state y:. and the aggregated score estimation é obtained from the 2D
4

---

Introducing 3D Representation for Dense Volume-to-Voluun Translation via Score Fusion
diffusion models. Furthermore, aggregated feature maps FF from the 2DD models are incorporated as supplementary information as in Fig. 2. Formally speaking, the 3D net- work is trained to perform the ensembling process using the following formulation using an 1.2 loss:
2023a), zero-initialized convolution layer at the end of the model smoothly initializes w and R as all-zero vectors. This makes the 3D training a fine-tuning process starting with an average weighting strategy, and thereby stabilizing the 3D model training. The pseudo-code for training and inference with Score-Fusion is provided in Algorithm and Algorithm 2.
Vesp 20 yy. α, 222(a)) g-2(1), .) 2
where yo, @ ~ p(yo x). and E (O,, Although, the inference results from 2D models. y-2((a) and y2D(b)) al- ready help the training of the 3D model, 3D ensemble model
3.4. Score-Fusion for Video Super-resolution
We extend the Score-Fusion framework to the task of video super-resolution. Since most existing diffusion -based video super-resolution methods incorporate both spatial and tem- poral features, we adopt MGL.D- (GLRD-VSR (Yang et al., 2024) as our 3D model to fuse scores. MGLD-VSR builds upon 2D latent diffusion model pretrained on the spatial (x-y) plane using image datasets and introduces inter-frame guid- ance to both the diffusion model and decoder, enabling temporal consistency in the output. However, like other diffusion-basst based video super-resolution methods, MGLD- VSR does not explicitly learn representations in the time- space planes, despite prior works emphasizing its impor- tance (Al-Sumaidaee et al., 2023; Otroshi-Shahres et al., 2022).
still needs to be trained from scratch. To improve train- ing speed, we initially pre-train the model on 3D patches, (yo, . = crop(yo. .²), and then fine-tune it on the full vol- umes. Due the translation invariance of our convolution- based networks, on the patches results in a decently good network initializa- tion, thereby effectively improving the training convergence. While existing works, such as (Wang et al.. 2023), could po- tentially enhance this patch-wise diffusion training process, we leave such optimizations for future work.
empirically find that naively pre-train
In this work, the network architecture of the 3D model, Egsp is 3D Unet-like denoising model with ttme-step embeddings, the 3D input æ, the noisy target yı, and the noise estimated by the 2D models 22D(e), 2DD()) as the in- put of the 3D model. In the encoder, each downsampling block is enriched with corresponding feature maps from the features of both 2D models, F2D(a), and F2D(6), At each hierarchical level, the feature maps are first processed with MLP-based alignment layers, which align the 2D features with the 3D model and map them to an appropriate shape. The feature maps are then injected into 3D layers, providing the 3D layer with a reference to fused 21) features. Hence, the 3D layers are reformulated to learn an ensemble of the aggregated 2D features, which is easier than learning repre- sentation from scratch. In addition, using the feature maps mitigates the risk of information bottlenecks between the 2D and 3D stages, which could otherwise limit the performance. Additionally. rather than directly outputting the predicted noise, our 3D U-Net-like model produces two components: a weight vector w. used to ensemble the estimations from the 2D models, and residual term R, which is directly esti- mated by the 3D model. These two outputs are combined to form the final prediction:
To address this limitation, we train two additional 2D dif- fusion models directly in the video latent space by slicing video latent embeddings along time-space planes. The de- noising outputs from these time-space models are then used auxiliary conditions to fine-tune the 3D model. This in- tegration allows the model to benefit from both time-space representation learning and cross-plane features.
3.5. Multi-modality Fusion
In volumetric translation for medical imaging, the condi- tions for translating a new image can be multifaceted, For instance, DDMM-Synth (Li et al., 2023b) suggested using both MRI and low-resolution CT scans to produce high- resolution CT images. Training a separate model for each possible combination of input conditions would result in exponential time complexity, making it generally impracti- cal. On the other hand, training a foundation model to sup- port various translation tasks also requires larger datasets, which are hard to acquire in medical imaging. Therefore, a model that can efficiently integrate pre-trained models across diverse conditions provides significant advantages. Score-Fusion achieves this by naturally integrating multiple diffusion models, each conditioned on individual modalities, through 3D network architecture that functions similarly to fusing two 2D models described in 3.3. This approach leverages the accelerated 3D) training of Score-Fusion. To further enhance the speed of multi-modality fusion, we em- play a smaller variant of our model, adjusting the number of channels in each layer.
3D (...) = (0.5 + ( w) + +0.5 - w))--w)) + +XR (4) Egpp
where À is a hyperparameter, whereas and R are of the same size as the target noise c E Rab, This design enables the model to dynamically select the more reliable 2D estimation based on 3D context and allows the 3D model to contribute 3D-specific content R. Meanwhile, a tunable weight parameter, X. controls the model's 's reliance on the 3D output, R. In addition, inspired by ControlNet (Zhang et al.,
5

---

Introducing 3D Representation for Dense Volume-to-Volun Translation via Score Fusion
Algorithm 1 Training of Score Fusion
In addition, we investigate Score-Fusion's generalizability to different datasets by applying it, alongside related base- lines, to super-resolution task on the FLAIR modality of the HCP dataset (DC et al., 2013). This demonstrates Score- Fusion's 's potential for broader applicability. We present both quantitative and qualitative results for the HCP dataset in Section D.
1:
repeat
2:
(x) ~ yo)
sample from dataset i pretrain on patch
3: if pretrain then
4:
Yo) = crop(x,yo) end if
5:
6:
~~ Uniform(,, (0, T) (O,,T)
7:
0 b2 do
8:
Y-2 i c 6aa
2D(a)
Baselines. We reproduced diverse baseline methods across diverse set of established 2D and 3D translation models to ensure comprehensive comparison. For slice-wise 2D models, we utilized Pix2pix (Isola et al., 2017) as the rep- resentative GAN-based method, U-Net (Ronneberger et al., 2015) for supervised regression. Palette (Saharia et al., 2022) as a diffusion-based approach, and 12SB (I.iu et al., 2023a) for aptimal-translation based modeling. Similarly, for 3D- based baselines, used Pix2pix3D (Isola et al., 2017), U- Net3D (Ronneberger et al., 2015), Med-DDPM (Dorjsembe et al., 2024) (or Palette3ID). As stated in Sec. 1, Med-DDPM uses a small denoising network and thus demonstrates lim- ited performance. In addition, we used Palette-2.5D for another baseline, which uses multiple consecutive 2D slices as input. Several existing approaches closely related to our method combine multiple pre-trained 2D diffusion models in perpendicular orientations, demonstrating enhanced per- formance over other baselines. For instance, TPDM (Lee et al., 2023) combines two 2D diffusion models trained on perpendicular planes. To support modality translation, we adapted the TPDM's 2D backbone to Palette (Saharia et al., 2022) architecture in place for DPS (Chung et al., 2023a). Furthermore, TOSM (Li et al., 2024) employs three perpendicularly trained 2D diffusion models, whereas MADM (Chen et al., 2024) uses three 2.5D diffusion mod- els. We perform a hyper-parameter search on the super- resolution task on the BraTS dataset for all baselines.
9:
end for
10:
for
to by do Y2 [. t Coa
2D(b) (yr (:, 5, x((,),)).t)
11:
12:
end for
13:
Take a gradient descent step on
14:
Vesp 60 (yi, 42D(a)) 42D(1)) .) - -
15:
until converged
Algorithm 2 Inference of Score Fusion
1: (az) ~ >p())
D sample from dataset
2: yr ~ ().
3: for to TT.........dd
4:
for f0 do [,,,;]++ + Eoa
5:
2D(e)
(2n i.,,,
6: end for
7:
for j bs...... do
8:
220(0) [t:n]]] Eau
20(b)
(3 [. 31.1))).1
9:
end for
10: 3D = 30 (Ve, x, (
p2n()) t)
11: go- + u-vT-mia2
get current estimation of y0
12: if Inverse problem then
13:
g0 + 9o- - A (AA²) (Ago - x)
14: end if
15: V/-1 1iu + VT 1 a:-16
16:
end for
17:
return Yo
4. Experiments
Model Architecture and Variants. To make a fair compar- ison, we use pre-trained 2D models from TPDM, TOSM, and MADM utilizing an existing 3D diffusion model ar- chitecture, Med-DDPM (Dorjsembe et al., 2024). The TPDM-based model is our primary model as it achieves 30% faster inference speed and smaller model size relative to the TOSM-based model, as shown in Tab. 10. Meanwhile, MADM-based and TOSM-based models are heavier vari- ants that yield performance gains across all metrics. These consistent improvements in all three variants demonstrate that our approach can serve plug-in alug-in-ann in-and-play mecha- nism for multiple combinations of 2D/2.5ID model back- bones and existing 3D model architectures. We also include a more detailed model architecture in Sec. F.
4.1. Experimental Setup
Datasets. We conducted experiments using the BraTS 2021 training dataset (Baid et al., 2021), which includes 1,251 volumetric brain scans with tumors across 4 modalities: FLAIR, TI, Tlce, and T2. We randomly divided the dataset into a 0.8:0.2 split for training and evaluation purposes, al- lowing its use for downstream tasks as well. Each scan was center-cropped to a dimension of 192x 192x 152 to re- move the blank background. For training 2D models, we sliced the 3D volumes two directions- directions-ttrasverrse and sagittal planes for both TPDM baselines and Score-Fusion. In the super-resolution experiment, FLAIR images were downsampled using [4x4x4] average pooling. For modal- ity translation, Tice images served as inputs with FLAIR images as targets. Additionally, we investigated a multi- condition task, using both low-resolution FLAIR and Tice images as input to predict high-resolution FLAIR images.
Metrics. We used multiple metrics to assess both the accu- racy and realism of generated MRI images. For accuracy, used peak signal-to-noise ratio (PSNR) and the structural similarity index measure (SSIM), which are widely used in
6

---

reconstructs more realistic details with smoother Introducing 3D Representation for Dense Volume-to- Volume Translation via Score Fusion
Ground Truth
Super-Resalution
Modality Translation
Both Conditions
Input
Ours
TPDM
Inpux
Ours
TPDM
Ours
TPDM
Figure 3: Visual comparison of generated samples for three different conditions. The first three rows show axial view slices from different MRI volumes. Neither Score-Fusion nor TPDM have 2D model trained in this direction. The last three rows show slices for the same MRI volume in all three views. edges and fewer artifacts.
medical imaging. To evaluate perceptual quality and realism, we used the maximum mean discrepancy (MMD) (Gretton et al., 2012) and the Fréchet inception distance (FID) met- rics (Heusel et al., 2017). Lower MMD/FID scores imply the generated images are more realistic. evaluate the FID score, following common practice (Dorjsembe al., 2024; Sun et al., 2022), we adopted the same pre-trained model (Chen al., 2019) to extract features and calculate the FID metrics in the feature space. Because diffusion- based models exhibit inherent stochasticity, we further as- sess uncertainty by performing inference multiple times with different noise realizations e. From these runs, we calculate voxel-wiss means and standard deviations, thereby provid- ing uncertainty-awer metrics. Quantitative and qualitative results for these metrics can be found in Sec. B.
ation quality under various conditions and provides compar- isons with other methods. The first two columns show the performance for super-resolution and modality translation, and the last columns show the model performance when using both conditions. Our approach excels in faithfully recovering intricate high-frequency details, particularly in tumor-affected areas where such details are complex and often underrepresented. In the super- super-resolution task. from the zoomed-in panel, Score-Fusion clearly distinguishes tissue boundaries across various tissue types, including tu- mor and white/grey matter. In the modality translation task, the distribution of contrast difference between modalities is also better captured, as stated in Sec. B. Furthermore, our method demonstrates superior volumetric consistency, while the bascline model exhibits noticeable artifacts. In the visualization of Score-Fusion, the fidelity of tissue texture and sharpness along all three orthogonal directions are well preserved, even though the 2D diffusion models in Score- Fusion are trained only on transverse and sagittal planes. Our model reconstructs tumor regions with clearer margins, fewer artifacts, and higher resolution for samples containing tumors, providing superior performance in all three orthogo-
4.2. Experimental Results for Medical Image
We showcase the performance of Score-Fusion in solving various translation problems, including 4x super-resolution, modality translation, and conditioned on both conditions in Fig. 3. We also include more randomly selected samples for more variants of our model in Sec. C. Fig. 3 shows the gener-

---

Introducing 3D Representation for Dense Volume-to-Volun Translation via Score Fusion
nal directions. Overall, Score-Fusion generates images with higher accuracy, realism, and volumetric consistency. Tab. summarizes the quantitative results of each translation task. Score-Fusion clearly surpasses other baseline models in most metrics, showing superior fidelity, structure, texture preservation, and noise suppression performance.
4.5. Training and Inference Speed.
We show Training and Inference Speed in Tab. 5. Previous 3D diffusion method struggles to use 3D representation, with extremely long training time (120 GPU days) and sub- optimal accuracy performance as in Tab. 1. In contrast, Score-Fusion effectively introduced 3D representation in just 16 days of extra 3D training time on top of TPDM. Score-Fusion-snie small further accelerates the 3D training for times, achieving 30x more efficient than 3D diffusion baselines.
4.3. Downstream Task
We further evaluated the performance of Score-Fusion on a downstream task: tumor segmentation, where high- quality input modalities are crucial for accurately seg- menting complex structures. Using the BraTS 2021 tu- mor segmentation dataset, we applied a pre-trained Swin- UNeIR (Hatamizadeh et al., 2022) model with four modal- ities, replacing the ground-truth FLAIR modality with in- ferences from each model. Segmentation performance was assessed on Tumor Core (TC), Whole Tumor (WT), and Enhancing Tumor (ET) regions using two metrics: Dice score and Recovery rate. The Dice score measures seg- mentation quality, while the Recovery rate quantifies each model's segmentation score recovered from low-quality FLAIR to the generated FLAIR, with segmentation using ground truth (GT) FLAIR as the upper bound and down- sampled FLAIR as the lower bound. The Recovery rate defined as: Recovery Rate = (Prediction - Downsample) / (Ground- Truth- Downsample), where Prediction refers to the segmentation performance using predicted FLAIR, Downsample is the performance with downsampled FLAIR, and Ground Truth is the performance with ground-truth FLAIR. As shown in Tab. 2, our methods consistently out- performed other methods. Our method also qualitatively results in smoother tumor edges and more accurate struc- tures demonstrated in Fig. 9. (see Sec. for more details).
4.6. Video Super-resolution Results
Dataset. We extend Score-Fusion to video super-resolution. We treat videos as 3D dense volumes composed of axes: x-axis, y-axis, and t-axis. We adopted sliding window strategy on the time axis to get dense volumes of the same size. Following MGLD-VSR, we train our model on the REDS (Nah et al., 2019) dataset and evaluate on the Vide- oLQ (Chan et al., 2022) dataset.
Experiment Setup. Following previous works (Yang et al., 2024; Li al., 2023a; 2025), we adopt latent diffusion method, composed of a video auto-encoder and diffusion model in the latent space. For the video autoencoder, we use an off-the-shelf spatial-temporal autoencoder from MGLD- VSR (Yang et al., 2024). We use Score-Fusion in the latent diffusion model by using a 3D latent model to fuse two perpendicular 2D latent models. For the 3D model, we also utilize the pretrained spatial-temporal latent diffusion model from MGLD- VSR (Yang al., 2024). Such settings allow us to make fair comparison with MGLD-VSK For 2D models, we use pretrained models from Stable_ SR (Wang et al., 2024), which share a similar architecture and model size with MGLD-VSR. During training. the hierarchical layer of the 3D model learns to incorporate the 2D features. Following Ilama-adapter, we use a vero-initialized gate on the 2D features, such that the model starts with its original states as in MGLD-VSR
4.4. multi-moddlity fusion
As discussed in Sec. 3.5, the Score-Fusion not only merges 2D models trained in different directions but also effectively integrates models pre-trained under various single condi- tions when faced with new combinations of input modalities given pre-trained 2D models every single condition. We show the model's performance in Tab. 3. TPDM uses weighted average for all 2D models, demonstrating lim- ited performance. Using a 3D model of the original size, Score-Fusion learns fuse scores estimated in two different conditions, demonstrating competitive performance without re-training 2D models. Score-Fusion-small further improves training speed with marginal performance drop to flexibly support
Method
DOVER(1)
MGLD
0.748
Score-Fusion-MGLD
0.755
Table 4: Quantitative results for Video Super-Resolutin
We present our quantitative results in Tab. 4. using DOVER (Wu et al., 2023) metrics. DOVER focuses on video quality assessment by evaluating technical and aes- thetic perspectives, which proves to be highly aligned with human preference. Our results show performance improve- ment, demonstrating the positive impact of time-space plane representations in video super-resolution.
fusion. The models on both condi- tions (last 2 rows) show the metrics when re-training every 2D model on both conditions using an early fusion strategy, representing an upper limit of multi-modality fusion perfor- mance. All training experiments are performed on Nvidia
RTX A A100-40G GPUs.
8

---

Introducing 3D Representation for Dense Volume-to- Volume Translation via Score Fusion
SR
MT
both condition
Method
PSNR(1)
SSIM(T)
MMD())
())
PSNR
SSIM
MMD
FID
PSNR
SSIM
MMD
FID
Pix2pix(Isola 2017)
28.75
0.889
512.2
25.9
22.25
0.812
8989.0
577.6
31.78
0.923
133.9
11.8
U-net(Ronneherger al., 2015)
30,32
0.579
917.2
58.9
23.74
0.846
1829.0
320.3
33.58
0.931
83.5
36,6
Paletto(Saharia et al., 2022)
29.26
0.894
40.9
13.5
22.68
0.784
284.4
85.9
33.6
0.939
34,9
9.3
12SB(Liu et al., 2023a)
27.51
0.860
2644.5
47.7
20.75
0.738
35774.5
1343.6
31.3
0.905
1313.6
12.0
Dalote-3D et ul., 2024)
28.48
0.320
4222
88.7
Not Working
24.98
0.297
15926.0
463.1
Pix2pix-3D(ssola et al., 2017)
29.54
0.866
516.5
22.77
0.784
1974.0
342.2
31.86
0.900
87.81
56.2
U-net-3D(Ronneberger al., 2015)
31.23
0.892
115.6
23.43
0.809
487.5
273.8
32.95
0,922
43.3
43.9
Polete-2.5D(Saahrri et al. 2022)
29.76
0.834
35.37
12.3
23,04
0.728
1141.19
258.6
25,89
0.819
2858.37
138.92
TPDM 2023)
32.23
0.922
29.35
17.3
25.35
0.868
176.3
185.5
35.12
0.945
14.5
22.2
Scon--FuioonTPDDM
33.24
0,944
13.77
8.31
25,26
0.882
154.9
48.2
36.24
0.961
7.52
5.8
TOSM(Li ul., 2024)
32.76
0.932
24.17
24.87
25.66
0.881
1018.91
209.5
35.44
0.947
8.32
14.53
Score Cone-Fusion-TOSM
33.30
0.945
13.62
6,51
25.24
0.882
138.47
136.06
36.51
0.963
5,926
3.72
MADMi(Chn et al., 2024)
33.02
0.945
30.92
35.64
25.47
0.874
1419.4
251.4
35.21
0.946
8.21
13.6
Score-Fusson-MADMM
33.31
25.13
0.876
192.51
130.33
36.37
0.964
5.44
3.81
Table 1:
Quantitative evaluation of Score-Fusion on BraTS dataset. Best metrics are highlighted in bold. The proposed model achieves better accuracy (PSNR/SSIM) given more 3D context than their corresponding variant in most tasks. Moreover, thanks 3D representation, Score-Fusion achieves significantly better 3D realism (MMD/FID). We demonstrate the standard deviation and uncertainty metries in Tab. 6.
Training (GPU days)
Training (GPU days)
Inference (minutes/volume))
Method
Method
Dice (%)
Recovery (%)
31) Palette
120
0.6
TC
WT
ET
TC
WT ET
TPDM
16
0
1.72
Ssore-Fusion
16
16
2.34
GT FLAIR
82.71 89.17 81.20
Score- Sreee-Fuionn-maan
16
4
1.92
Downsampled GT
82.30 86.82 80.30
Table 5: Training and Inference time results, GPUs are
SR
TPDM
82.49 87.77 80.49 46.27
40.46
20.62
A A100-40G. More complete comparison Tab. 10
TOSM
82.52 87.21 80.80 54.55
16.51 55.28
Slore-Fusion-TPDD 82.69 87.85
80.94
93.71
43.80 71.69
Score-Fusion-TOSNM 82.59
87.86
80.87
70.14
44.38
63.94
5. Conclusion
MT
3D voxel-space representation can be essential in medical image translation and generation for both volumetric real- ism and downstream task performance. However, existing models struggle to use 3D representation due computa- tional challenges and data scarcity. In this work, have introduced Score-Fusion to effectively introduce 3D voxel space representation into 3D medical image translation by fusing estimations from slice-wise 2D models in the score function space. Several key designs, including average- initialization, feature map fusion, patch- mutch-wise pre-training. In addition, our model integrated the strengths of both 2D and 3D diffusion models. Score-Fusion provides strong insights for diffusion model ensembling as the first work to adopt learning- based fusion in the score function space. Empirical evaluations on various 3D MRI image translation tasks, including super-resolution and modality translation, have shown that Score-Fusion achieves unmatched accu- racy, volumetric realism, and downstream task performance. In addition to computational and memory efficiency, the approach offers considerable flexibility in merging models conditioned different domains.
TPDM
77.28 77.74 78.37
TOSM
77.94 79.21 78.64
Score-Fusion- TPDMM
77.88
78.51 78.22
Score-Fusion-TOSS. 78.84
78.73 79.52
both condition
TPDM
82.45 87.69 80.74 36.31 37.25 49.28
TOSM
82.54 87.27 80.82 57.81
19.25 57.79
Score-Fusion-TPDS
82.46 87.91 80.74 38.66 46.67 48.97
Sooee-Fusion-TOS.. 82.61 87.98 80.89 75.35 49.69 65.72
Table 2: Segmentation performance with the FLAIR modal- ity replaced by model predictions.
Training Time
Method
PSNR
SSIM
MMD
(GPU days)
TPDM
32.43
0.929
25.05
0
Score-Fusion-sman
35.34
0.956
8.64
4
Score-Fusion
35.6
0.958
8.82
16
TPDM-both cond
35.12
0.945
14.5
16
Fusion-both, botheen-botheeond cond
36.24
0.961
7.52
32
Table 3: Multi-modality fusion results for Score-Fusion.
9

---

Introducing 3D Representation for Dense Volume-to-Volun Translation via Score Fusion
Impact Statement
genomic classification. arXiv preprint arXiv:110.002314 2021.
This paper presents work that aims to advance the field of Machine Learning and its application in medical imaging and video processing. There are several Limitations of our work. Unlike some other multi-stage models (Zhu et al., 2023b; Huang et al., 2023), Score-Fusion struggles with joint end-to-end training due to the substantial computa- tional demands of simultaneously managing high-capacity 2D models and the volumetric complexities of 3D tasks. In addition, the model's dependency on patchwise pre-training for efficient 3D model learning presents limitations for tasks requiring the integration of long-range spatial information, such large-area inpainting and compressed sensing MRI. Therefore, Score-Fusion may require longer training for such tasks. There are also many potential societal conse- quences our work. However, direct application of our method to medical imaging should be approached with cau- tion.
Chai, W., Guo, X., Wang, G., and Lu, Y. Stablevideo: Text- driven consistency-awe aware diffusion video editing, 2023.
Chan, K. C., Zhou, S., Xu, X., and Loy, C. C. Investigating tradeoffs in real-world video super-resolution. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
Chen, S., Ma, K., and Zheng, Y. Med3d: Transfer learn- ing for 3d medical image analysis. arXiv preprint arXiv:1904.0m 1904.00625, 2019.
Chen, T., Hou, J., Zhou, Y., Xie, H., Chen, X., Liu, Q., Guo, X., Xia, M., Duncan, J. S., Liu, C., and Zhou, B. 2.5d multi- averaging diffusion model for 3d medical image translation: Application to low-count pet recon- struction with ct-less attenuation correction, 2024. URL https://arxivv org h/tg///b///// 2436.08374.
Acknowledgement
Cheng, S.-I., Chen, Y.-J.. Chiu, W.-C., Tseng, H.-Y., and Lee, H.-Y. Adaptively-realistie image generation from stroke and sketch with diffusion model. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 4054-4062, January 2023,
This work used the Delta system at the National Cen- ter for Supercomputing Applications through allocations CIS230243 and CIS240171 from the Advanced Cyber- infrastructure Coordination Ecosystem: Services & Sup- port (ACCESS) program. which is supported by Na- tional Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296. We also thank Xiaoyue Li (xiaoyue98 @zju.edu.cn) and Shen Yuan Miihen77@illmoiis for the valuable discussion.
Chung, H., Sim, B., Ryu, D., and Ye, J. C. Improving diffusion models for inverse problems using manifold constraints, 2022.
Chung, H., Kim, J., Mccann, M. T., Klasky, M. L.,, and Ye, J. C. Diffusion posterior sampling for general noisy inverse problems, 2023a.
References
Akrout, M., Gyepesi, B., Holló, P., Poór, A., Kincsö, B., Solis, S., Cirone, K., Kawahara, J., Slade, D., Abid, L., et al. Dise data augmentation for skin disease classification: Impact across original medical datasets to fully synthetic images. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 99-109. Springer, 2023.
Chung, H., Lee, E. S., and Ye, J. C. Mr image denoising and super-resolution using regularized reverse diffusion. IEEE Transactions on Medical Imaging, 42(4):922-934, 2023b. doi: 10.1109/TM.2022222666
Dar, S. U. H., Yurt, M., Karacan, L., Erdem, A., Erdem, E., and Çukur, T. Image synthesis in multi-conttast mri with conditional generative adversarial networks. IEEE Transactions on Medical Imaging, 38:2375-2388, 2018. URL https: //api. wttptttiisccccottoooorrrrreeeeee 3655281.
Sumaaiaiee,,, S., Abdullah, M., Al-Nima, R., Dlay, S., and Chambers, J. Spatio-temporal modelling with multi-gradien features and elongated quinary pat- tern descriptor for dynamic facial expression recogni- tion. Pattern Recognition, 142:109647, 2023. ISSN 0031-3203. doi:
DC, V. E., SM, S., DM, B., TE, B., Y., K, K, U., and WU- Minn HCP Consortium. The wu-minn human connec- tome project: an overview, 2013.
109647. URL sttenc//Ariieee//i///00122222222
com/
Baid, U., Ghodasara, S., Mohan, S. Bilello, M., Calabrese, E., Colak, E., Farahani, K., Salpatth-Cramee J., Kita- mura, F. C., Pati, S., et al. The rsma-asnr-mice miccai brats 2021 benchmark on brain tumor segmentation and radio-
Dorjsembe, Z., Pao, H.-K., Odonchimed, S., and Xiao, F. Conditional diffusion models for semantic 3d brain mri synthesis. IEEE Journal of Biomedical and Health Infor- matics, 2024.
10

---

Introducing 3D Representation for Dense Volume-to-Volune Translation via Score Fusion
Feng, R., Weng, W., Wang, Y., Yuan, Y., Bao, J., Luo, C., Chen, Z., and Guo, B. Ccedit: Creative and controllable video editing via diffusion models. CVPR, 2024.
Kazemi, Y. and Mirroshandel. S. A. A novel method for predicting kidney stone type using ensemble learning. Artificial intelligence in medicine, 84:117-126, 2018.
Fernandez, V., Pinaya, W. H. L.,, Borges, P., Tudosiu, P.-.., Graham, M. S., Vercauteren, T., and Cardoso, M. J. Can segmentation models be trained with fully synthetically generated data? In International Workshop on Simulation and Synthesis in Medical Imaging, pp. 79-90. Springer, 2022.
Kerstin, H., Teresa, K., Erich, K., P., R. M., K. S. D.,
Thomas, P., and Florian, K. Learning variational net- work for reconstruction of accelerated mri data. Mag- netic Resonance in Medicine, (9(6):3055-3077 doi: 10. 10.10022/mm..269777 URL ptttps://onineee cal.yy..om/oo//bbl http://ooaabb//...oo/ https//1.10000000 1002/mrm. 269.7.
Fitriyani, N. L., Syafrudin, M., Alfian, G., and Rhee, Development of disease prediction model based on en- semble learning approach for diabetes and hypertension. Ieee Access, 7:144777-144789 2019.
Khader, F., Müller-Franzes, G., Tayebi Arasteh, S., Han, T., Haarburger, C., Schulze-Hagen, M., Schad, P., En- gelhardt, S., Baeßler, B., Foersch, S., et al. Denoising diffusion probabilistic models for 3d medical image gen- eration. Scientific Reports, 13(1):7303, 2023.
Fu, Z., Guo, L., Wang, C., Wang, Y., Li, Z., and Wen, B. Temporal as plugin: Unsupervised video denoising with pre-trained image denoisers, 2024. URL https: //arxiv. org/ abs 2409/ 110099..1220 11256.
Kuleshov, V., Fenner, N., and Ermon, S. Accurate uncertain- ties for deep learning using calibrated regression, 2018. URL httpp:://aaaee
org/ abs// 1807 00263.
Gretton, A., Borgwardt, K. M., Rasch, M. J., Schölkopf, B., and Smola, A. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
Lee, S., Chung, H., Park, M., Park, J., Ryu, W.-S., and Ye, J. C. Improving 3d imaging with pre-trained per- pendicular 2d diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10710-10720, 2023.
Mai-Miao Zhang, B. D. A review on deep learning in med- ical image reconstruction. Journal of the Operations Research Society of China, 8(2):311, 2020. doi: 10.1007/ $40305-0099-0028--4 URL https:/ http://wooosss.. shu. edu..cn/EW/bbstracc//rre
Leroy, A., Shreshtha, K., Lerousseau, M., Henry, T., Estienne, T., Classe, M., Paragios, N., Grégoire, V. and Deutsch, E. Magnetic resonance imaging virtual histopathology from weakly paired data. In Atzori, M., Burlutskiy, N., Ciompi, F., Li. Z., Minhas, F., Müller, H., Peng, T., Rajpoot, N., Torben-Nielsen, B., van der Laak, J., Veta, M.. Yuan, Y., and Zlobec, I. (eds.), Proceedings of the MICCAI Workshop on Computational Puthology, volume 156 of Proceedings of Machine Learning Re- search, pp. 140-150. PMLR, 27 Sep 2021. URL https: //proceedings mlr wtw...preesvvviv 56/ leroyylallllll html.
175.l.1 shtml.
Hatamizadeh, A., Nath, V., Tang, Y., Yang. D., Roth, H., and Xu, D. Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images, 2022.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models, 2020.
Li G., Ji, J., Qin, M., Niu, W., Ren. B., Afghah, F., Guo, L., and Ma, X. Towards high- and efficient video super-resolution via ting, 2023a.
Huang, Z., Chan, K. C. K., Jiang, Y., and Liu, Z. Collabora- tive diffusion for multi-modal face generation and editing, 2023.
data overfit-
Li, X., Shang, K., Wang. G., and Butala, M. D. Ddmm- synth: A denoising diffusion model for cross-modal med- ical image synthesis with sparse-view measurement em- bedding. arXiv preprint arXiv:2303. 202.b.
Isola, P., Zhu, J.,Y., Zhou, T., and Efros, A. A. Image-to- image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vi- sion and pattern recognition, pp. 1 1134-1134, 2017.
Li, X., Liu, Y., Cao, S., Chen, Z., Zhuang, S., Chen, X., He, Y., Wang. Y., and Qiao, Y. Diffvsr: Revealing an effective recipe for taming robust video super-resolution against complex degradations, 2025. utthttps: abs/2501. 10.10
Jiang, A. Q. Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., de las Casas, D., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, L. R., Saulnier, L., Lachaux, M.-.,, Stock, P., Subramanian, S., Yang, S., Antoniak, S., Scao, T. L., Gervet, T., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mixtral of experts, 2024.
Li, Z., Wang, Y., Zhang, J., Wu, W., and Yu. H. Two-and- a-half order score-based model for solving 3d ill ill-posed

---

Introducing 3D Representation for Dense Volume-to-Volun Translation via Score Fusion
inverse problems. Computers in Biology and Medicine, 168:107819, 2024.
Saharia, C., Ho, J., Chan, W., Salimans, T., Fleet, D. J., and Norouzi, M. Image super-resolution via iterative refinement, 2021.
Liang, D., Cheng, J., Ziwen, K., and Ying, L. Deep magnetic resonance image reconstruction: Inverse problems meet neural networks. IEEE Signal Processing Magazine, 37: 141-151, 01 2020. doi: 10L.0///22....99555555
Saharia, C.,, Chan, W., Chang, H., Lee, C., Ho, J., Salimans, T., Fleet, D., and Norouzi, M. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 conference proceedings, pp. 1-10, 2022.
Liu, G.-H., Vahdat, A., Huang, D.-A., Theodorou, E. A., Nie, W., and Anandkumar, A. 19sb: Image-to-image schrödinger bridge. ICML, 2023a.
Shi, Z., Mettes, P., Zheng, G., and Snoek, C. Frequency- supervised mr-to-ct image synthesis. In Deep Generative Models, and Data Augmentation, Labelling, and Imper- fections: First Workshop, DGM4MICCAI 2021, and First Workshop, DALJ 2021, Held in Conjunction with MICCAI 2021, Strasbourg, France, October 1, 2021, Proceedings I. pp. 3-13. Springer, 2021.
Liu, Y., Dwivedi, G., Boussaid, F., Sanfilippo, F., Ya- mada, M., and Bennamoun, M. Inflating 2d convolu- tion weights for efficient generation of 3d medical im- ages, Computer Methods and Programs in Biomedicine, 240:107685, October 2023b. ISSN 0169-2607. doi: 10. 10.1016fj.cmp..2000 cmmp..002. org/10, 101 6 cmpb. 2023. 107685.
URL http://dx.dot
Song, B., Kwon, S. M., Zhang, Z., Hu, X., Qu, Q., and Shen, L. Solving inverse problems with latent diffusion models via hard data consistency. ICLR, 2024.
Lyu, Q. Shan, H., and Wang, G. Mri super-resolution with ensemble learning and complementary priors. IEEE Transactions Computationaa Imaging, (:615-624, 2020. doi: 10. 10.1 10.10 296421..
Song. Y., Durkan, C., Murray, I., and Ermon, S. Maximum likelihood training of score-based diffusion models, 2021.
Song, Y., Shen, L., Xing, L., and Ermon, S. Solving inverse problems in medical imaging with score-based generative models, 2022.
Mardani, M., Gong, E., Cheng, J. Y., Vasanawala, S., Za- harchuk, G., Alley, M., Thakur, N., Han, S., Dally, W., Pauly, J. M., et al. Deep generative adversarial networks for compressed sensing automates mri IEEE Trans Med Imaging, 2019.
Sun, L., Chen, J., Xu, Y., Gong. M., Yu, K., and Batmanghe- lich, K. Hicrarchical amortized gan for 3d high resolution medical image synthesis. IEEE journal of biomedical and health informatics, 26(8):3966-3975, 2022.
Meng, Z., Guo, R., Li, Y., Guan, Y., Wang, T., Zhao, Y., Sutton, B., Li, Y., and Liang, Z.P.. Accelerating t2 map- ping of the brain by integrating deep learning priors with low-rank and sparse modeling. Magnetic Resonance in Medicine, 83(3): 1467, March 2021. ISSN 0740- Publisher Copyright:
Uzunova, H., Ehrhardt, J., and Handels, H. Memory- efficient gan-based domain translation of high reso- lution 3d medical images. Computerized Medical Imaging and Graphics, 86: 86:101801, December 2020. ISSN 0895-6111. doi: 10.1016/j.copmeeding 1010.101801 URL http://dk doi. 1016// compmedimag. 2020. 101801.
3194. doi: 20.10022/mm.2285 @ 2020 International Society for Magnetic Resonance in Medicine Copyright: Copyright 2020 Elsevier B. All rights reserved.
Nah. S., Timofte, R., Gu, S., Baik, S., Hong, S., Moon,
Vedula, S., Senouf, O., Bronstein, A. M., Michailovich, O. V., and Zibulevsky, M. Towards ct-quality ultra- sound imaging using deep learning. arXiv preprint arXiv: 2017.
G., Son, S., and Lee, K. M. Ntire 2019 challenge video ruppr-ies: Methods and results. In CVPR Workshops, June 2019.
Otroshi-Shahreza, H., Amini, A., and Behroozi, H. Feature- based no-reference video quality assessment using extra trees. IET, 2022.
Wang. G., Ye. C., and De Man, B. Deep learning for tomographic image reconstruction. Nature Ma- chine Intelligence, 2:737-748. 12 2020. doi: 10.1038/ -22256-000000223--
Peng, W., Adeli, E., Bosschieter, T., Park, S. H., Zhao, Q., and Pohl, K. M. Generating realistic brain mris via con- ditional diffusion probabilistic model. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pp. 14-24. Springer, 2023.
Wang, J., Yue, 7.,, Zhou, S., Chan, K. C., and Loy, C. C. Exploiting diffusion prior for real-world image super- resolution. 2024.
Wang, Z. Jiang, Y., Zheng, H., Wang, P., He, P., Wang, Z., Chen, W., and Zhou, M. Patch diffusion: Faster and more data-efficient training of diffusion models, 2023. URL https://arxiv. arg/abs//2n 12526.
Ronneberger, O., Fischer, P., and Brox, T. U-net: Con- volutional networks for biomedical image segmentation, 2015.
12

---

Introducing 3D Representation for Dense Voluue-to-Volume Translation via Score Fusion
Wolterink, J. M., Dinkla, A. M., Savenije, M. H., Seevinck, ct
P. R., van den Berg, C. A., and Iguu, I. Deep synthesis using unpaired data. In Simulation and Synthe- sis in Medical Imaging: Second International Workshop, SASHIMI 2017, 7, Held in Conjunction with MICCAI 2017, Québec City, QC, Canada, September 10, 2017, Proceed- ings 2. pp. 14-23. Springer, 2017.
Wu, H., Zhang, E., Liao, L., Chen, C., Hou, J. H., Wang, A., Sun, W. S., Yan, Q., and Lin, W. Exploring video quality assessment on user generated contents from aesthetic and technical perspectives. In ICCV, 2023.
Yang, X., He, C., Ma, J., and Zhang, L.. Motion-guided latent difffusion for temporally consistent real-world video super-resolution. 2024.
Yang, Z., Zhuang, X., Sreenivasan, K., Mishra, V., Curran, T., and Cordes, D. A robust deep neural network for de- noising task-based fmri data: An application to working memory and episodic memory. Medical Image Analysis, 60: 60:101622, 11 2019. doi: m0.10166..eedia221199.
Zhang, H., Li, H., Dillman, J., Parikh, N., and He, L. Multi- contrast mri image synthesis using switchable cycle- consistent generative adversarial networks. Diagnostics, 12:816, 03 2022. doi: 10.3390/diagnoosiicss22200 20448811.
Zhang, L.,, Rao, A., and Agrawala, M. Adding conditional control to text-to-image diffusion models, 2023a.
Zhang, Z., Jiang, Y., Shao, W., Wang, X., Luo, P., Lin, K., and Gu, J. Real-time controllable denoising for image and video, 2023b. URL //arxiv org/ /rggs//bbb3.. 6425.
Zhou, C., Chen, S., Ding, C., and Tao, D. Learning contex- tual and attentive information for brain tumor segmenta- tion. In Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: 4th International Work- shop, BrainLes 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Revised Se- lected Papers, Part II 4, pp. 497-507. Springer, 2019.
Zhou, S., Yang, P., Wang, J., Luo, Y., and Loy, C. C. Upscale-a-video: cemporal-consister diffusion model for woald video super-resolution, 2024.
Zhu, L., Xue, Z., Jin, Z., Liu, X., He, J., Liu, Z., and Yu, L. Make-a-volume: Leveraging latent diffusion models for cross-modality 3d brain mri synthesis, 2023a.
Zhu, L.,, Codella, N., Chen, D., Jin, Z., Yuan, L.., and Yu, L. Generative enhancement for 3d medical images. arXiv preprint arXiv:2403,12883 2024.
Zhu, X., Zyrianov, V., Liu, Z., and Wang, S. Mapprior: Bird's s-eye view map layout estimation with generative models, 2023b.
13

---

Introducing 3D Representation for Dense Volume-to- Volume Translation via Score Fusion
MT
both condition
Method
PSNR(1)
SSIM(1)
())
PSNR
SSIM
MACE
PSNR
SSIM
MACE
32.23
0.922
25.35
0.868
35.12
0.945
TPDM(Lec et al., 2023)
53.16
279.3
48.72
0.0066
+ 0,000300
1 0.2363
1 0.00112
1 0.0154
0.00056
33.24
0.944
25.26
0.882
36.24
0.961
Ours-TPDM
44.63
268.3
38.49
+ 0.0298
0.000261
0.664
+ 0.00357
+ 0.0389
= 0.000249
32.76
0.932
25.66
0.881
35.44
0.947
TOSM(Li et al., 2024)
51.68
221.7
46.32
+ 0.02157
+ 0.000434
+ 0.2703
+ 0.00132
+ 0.0173
+ 0.00073
33.30
0.945
25.24
0.882
36,51
0,963
Ours-TOSM
42.89
200.8
36.67
+ 0.0296
+ 0.000240
= 0.668
+ 0.00317
= 0.0364
= 0.000210
33.02
0.946
25.47
0.874
35.21
0,946
MADM(Chen al., 2024)
83.20
251.4
47.89
+ 0,0276
+ 0.000436
0.2573
+ 0,00143
- 0.0165
0.00063
33.31
0.945
25.13
0.876
36.37
0,964
Ours-MADM
42.03
234.6
37.15
0,0308
0.000278
+ 0.667
+ 0.00342
+ 0.0379
+ 0.00226
Table 6: Quantitative evaluation of Score-Fusion on BraTS dataset with uncertainty metrics. Our models' performance boost is significant, given low standard deviations. Our model can also estimate uncertainty better through the standard deviation obtained by inference multiple times.
A. Overview
In this supplementary material, we first discuss the uncertainty awareness results performed by our model and baselines by rumning the inference multiple times in Sec. B. We provide more randomly selected results (We do exclude samples with low-quality GT) for more baselines and our variants Sec. C. Then, we provide our super-resolution result in an additional dataset, HCP dataset (DC et al., 2013) in Sec. D. We provide ablation studies on key techniques in Sec. We also provide more details on training and inference, including detailed model architecture in Sec. F and training/inference speed in Sec. G. We finally introduce more detailed method for self-consistency projection in Sec. H and downstream task results in Sec. I.
B. Uncertainty Awareness
As with most diffusion-based models, our models and some of our baselines can have uncertainty estimations. To study this uncertainty, we perform inference five times for each sample our validation set. This gives us 5 PSNR and SSIM values for each data sample. We then calculate the standard deviation (std) of the PSNR and SSIM for each sample and include the mean std across the entire validation set in Tab. 6. This further validates that our performance boost in PSNR and SSIM is significant. For the main variant, TPDM and Ours- TTDM, in the super-resolution task, we have 1.01 boost in PSNR, which is much larger than the std of PSNR for both models (0,0066 and 0.0298). Even for MADM and Ours-MADM, where we have the most marginal PSNR boost, the boost is still 0.3, around 10 times larger than the std for both models (0.0308 and 0.0276).
contrast, in modality translation, the std is significantly larger since the uncertainty in this task is much larger than in others, indicating the PSNR drop is not as significant. In fact, previous work (Saharia et al., 202 argues that PSNR prefers blurry results, and highly diverse and realistic results typically have low PSNR in tasks with high uncertainty.
In addition, this inference also provides mean μı; and std estimation for each voxel. We use Mean Absolute Calibration Error (MACE) (Kuleshov et al., 2018) to measure the uncertainty awareness of our model and baseline. MACE measures the absolute difference between the predicted uncertainty and the actual error, as shown in Eq. 5.
1 N
MACE
0
1y:
As demonstrated in Table 6, all variants of our model exhibit lower MACE values compared to their respective baselines. This indicates that the standard deviation (std) predicted by our model, derived from multiple inferences, provides more accurate estimation of the true error relative to the ground truth. Consequently, our model exhibits improved uncertainty awareness. For qualitative results in uncertainty awareness, we demonstrate our model's 's results with the uncertainty map and error map across various tasks and variants in Fig. 14 15 16 17 18 19 20 21. As shown in the figures, the uncertainty map aligns well with the actual error map, demonstrating decent uncertainty awareness for all models. Notably, our model

---

Introducing 3D Representation for Dense Volume-to-Volune Translation via Score Fusion
usually has higher uncertainty in modality translation tasks in Fig. 16 and 17. In the modality translation task, the p(y (2) should have high variance in overall contrast. Our model outputs samples that highly diverse in overall contrast, indicating that Score-Fusion is able to model the target 3D conditional distribution p(y|a) better. In contrast, our baselines tend to output the mean estimation for overall contrast, demonstrating higher PSNR but limited capability of generating diverse and realistic results.
C. Additional qualitative result
We show results for the variants that show the best metrics. Namely, we show results for MADM and Ours-MADM in the super-resolution task in Fig. 14 and 15, and show results for TOSM and Ours-TOSM for the other two tasks in Fig. 18 19 16 and 17. We include more samples in all 3 views in Fig. 20 and Fig. 21 in super-resolution tasks in addition to Fig. 3. Each figure contains 2 sample volumes, each of which contains visualizations in all three views in three rows. We show all results with uncertainty and error maps.
Similarly to Fig. 3, we find that both MADM and TOSM demonstrate similar artifacts as TPDM high-frequency details due a direct averaging the score function. In contrast, Score-Fusion consistently demonstrates better 3D consistency and realism across all views by introducing pixel-space 3D representation and networks to replace the weighted averaging in the score function space. For example, in the 6-th row of Fig. 14 16, and 18. the results from baselines are blurry at the top left part of the brain, whereas Score-Fusion shows more smooth and consistent results.
D. Result on HCP dataset
We present our super-resolution results on the FLAIR modality in the HCP (DC al., 2013) dataset to show our model generalizable across datasets. The HCP dataset consists of 1251 MRI volumes with resolution of 192x152x152. In contrast to the BraT's dataset, HCP comprises healthy brains with brain tumors. Experiments results in Tab. show that our model shows around 1.5 performance boost in PSNR. We also present the qualitative results in Fig. 4, including all 3 views. Again, Score-Fusse Fusion shows better 3D consistency and realism. For example, in the top-right part in the third view, our baseline demonstrates jittering and artifacts, while our model produces more realistic detail and smoother edges.
Method
PSNR
SSIM
MMD FID(Ie-4)
TPDM
28.17
0.890
81.81
35.70
Ours-TPDM
29.62
0.914
67.96
22.12
Table 7: Super-resolution result in HCP dataset.
TRDM
Durs
GI
Figure 4: Qualitative results in HCP dataset. The input is a 4x4x4 downsampled version of the ground truth.
2

---

Introducing 3D Representation for Dense Volume-to-Volun Translation via Score Fusion
E. Ablation Studies
We provide an ablation study Tab. on the key design elements for Score-Fusion, which includes: (1) Feature merging:
The 2D models not only contribute their outputs but also pass their feature maps the 3D model. (2) Finetune: We initially pre-train the model on 3D patches and then fine-tune on the full volume speed up training. (3) Consistency: Inspired by
DPS (Chung al., 2023a) and score-SDE (Song al., 2022), we implement self-consistency projections each denoising step. All these designs show performance gain in the super-resolution task. In addition, we benchmark the smaller variant in Sec. 4.4 for comparison, which shows a moderate performance drop compared to our best model.
Smaller model
Cousistency
Finetune
Feature
PSNR
SSIM
MMD
32.83
0.935
25.45
32.88
0.94
18.84
32.97
0.941
15.24
33.04
0.942
17.76
33.24
0.944
13.77
0.937
16.78
Table Ablation studies of additional design elements Score-Fusion. F. Detailed model architecture
In this section, we show detailed model architecture for 2D, 3D, and the smaller variant of the 3D model in Tab. 11, Tab. 12, and Tab. 13, respectively. In addition, we show other related syper-parameters in Tab. We modified the architecture of the 2D diffusion model from Palette (Saharia et al., 2022) and the 3D models from med-ddpm (Dorjsembe et al., 2024).
Given the differences in problem setting and dataset between our work and that of Palette, we conduct a comprehensive hyper-parameter search based on the super-resolution tasks. This search explores various configurations, including the number of channels, transformer layers, and learning rate, among others. The hyper-paramett search is conducted to optimize the performance of our baseline models, Palette2D, Palette3D, and Palette2.5D, Tab. 1. While such a search could potentially enhance the performance of our proposed model, we do not perform a hyper-parae prree-parmetter search to optimize the performance of Score-Fusion, TPDM, TOSM, and MADM. This practice ensures fair comparison between our model and their corresponding baselines, TPDM, TOSM, and MADM. Moreover, this shows that our model can be plug-in plug-in-ande aan-and--lay mechanism for existing pre-trained 2D and 3D model architecture.
Table 9: Other hyper-parameters
Parameter
2D Network
3D Network
Batch size
4
Diffusion steps
1000
1000
Inference steps (DDIM)
50
50
Noise scheduler
Linear
Linear
Learning rate
0.00005
0,0001
Optimizer
Adam
Adam
G. Training and inference speed
We present training inference speed in Tab. 10. All experiments are done with RTX A100-40GB GPU. Since Score-Fusion needs to train an additional model on top of the baselines, our training time inevitably higher. We need 16 GPU days to train our 3D models, which results in a 16-day increase in training time for most model variants compared to their corresponding baselines. Our models are also relatively slower in inference since we need to perform inference for an additional 3D model. However, as mentioned in Sec. 1, the 3D model is naturally limited in size due computational challenges in training. Therefore, 3D inference is more efficient than slice-wiee wise 2D inference. As result, the increase in inference time is significantly smaller than in training. As shown in Tab. 10, our 3D model is around 30% faster than one 2D model and, therefore, leads to 36% increase in inference time for Ours-TPMM and 26% for Ours-MADM and Ours-MADM.
Moreover, we find that the -PDed models are significantly faster than other variants of the models. Given the advantage of computational efficiency, we use TPDM and Ours-TPDM as our main variables for the model and the baseline.
3

---

Introducing 3D Representation for Dense Volume-to-Volune Translation via Score Fusion
Furthermore, perform more complete ablation study, the smaller 3D model decreases the inference and training time of the 3D model by 75% while showing a consistent performance boost over TPDM and moderate performance drop compared to Ours-TPDM shown in Tab. 1. Moreover, compared to 3D Palette baseline, our model effectively decreased 3D training from 120 GPU days to 16/4 days, addressing the computational challenge of 3D diffusion training.
Table 10: Training and Inference time for each model, GPUs are A with 40G memory.
Training time
Inference time
Time
(GPU days)
(minutes per volume)
2D Palette
0.85
2D I2SB
5
4.58
3D Pix2pix
0.0398
3D Unet
6
0.0398
3D Palette
120
0.6
TPDM
16
1.72
Ours-TPDM
32
2.34
Murs-TPDM-smal
20
1.92
TOSM
24
2.55
Ours-TOSM
40
3.23
MADM
36
2.76
Ours-MADM
52
3.56
H. Details for consistency projection
In this section, we provide the exact definition and detail for selfscoonistency projection mentioned in Sec. 3.1. In this work, we address the inverse problem using a diffusion model with consistency projections. The goal is to recover a high-resolution image, y. from its low-resolution observation æ. which is obtained through linear degradation process. Specifically, the degradation process is modeled as: .Ay.
In the 3D case, the degradation operator A represents downsampling operation that reduces the resolution of volume by factor of 4 along each spatial dimension (x, y and z) and resizes it back to the original resolution. This means that each voxel in the low-resolution volume x corresponds to the average of a [4x4x4] region in the high-resolution volume y. 12h xb2 xbs xb2xbs downscales the high-resolution volume y
Specifically, let y, x E Re xb2x ba, The operator matrix into the oow-resolution volume x by averaging over [4x4x4] blocks of voxel of y. Therefore, A is a sparse matrix where each non-zero entry corresponds to the average of a block of [4x4x4] voxels in y being averaged to form block of voxel in x. Therefore, A is 1 for the places where x and y belong to the same block, and A would be 0 elsewhere:
((i,,,,k, g.)) = 64
if pp, q, r) E block
(6)
A((i, j. (p. (.) =
Otherwise
Since we are doing average over [4x4x4], afi,j, k) and u(p. q,r) r) are in the same block if and only if i//4 == p/ ///
1//4- 0-/4==oh g//4, and k//4- =- 1//4=-r//// r//4.
In our diffusion process, we use (t) < (t) - (F -1 (A x) to make every of our mean prediction of Jo a plausible estimation with æ = (t)(
To compute matrix multiplication more efficiently in a super-resolution setting, we actually use (t) + (o - (Ago -x) in our code. This works in the average pooling downsample because AAy Ay since A represents the degradation process composed of average pooling followed by resizing the image back to its original resolution.
I. Details in downstream task
In Section 4.3, we evaluate tumor segmentation performance using three types of FLAIR inputs: the ground truth FLAIR modality, 4x downsampled FLAIR modality (as described in Section 4. 4.1), and the 4x super-resolution FLAIR prediction
4

---

Introducing 3D Representation for Dense Volume-to-Volun Translation via Score Fusion
the downsampled FLAIR. Accurate tumor segmentation is crucial in medical imaging, and its performance heavily relies the quality of the input data. It requires High-quality inputs for precise localization and delineation of tumor boundaries, while, depending its degradation level, the degraded inputs could significantly lower segmentation accuracy and reliability. We use robust pre-trained segmentation model, SwinUNet (Hatamizadeh et al., 2022), which takes four modalities (TI, Tice, T2, and FLAIR) as input. For this downstream task, our objective is to assess how well the models can recover segmentation performance when working with degraded inputs. Segmentation is performed with other modalities with ground truth inputs and a FLAIR input from the ground truth FLAIR, downsampled FLAIR, or the model-predicted FLAIR. Note that because there is no degraded FLAIR modality available in the modality translation task, only dice scores reported. For other tasks, including the super-resolution and both condition tasks, performance is measured using two metrics: (1) Dice Score, the primary metric of the segmentation model, and (2) Recovery Rate, measure of how well model predictions improve upon degraded FLAIR inputs. The recovery rate is calculated as:
Prediction Downsample
Recovery Rate
Ground Truth Downsample
where Prediction refers to the segmentation performance using predicted FLAIR, Downsample is the performance with downsampled FLAIR, and Ground Truth is the performance with ground truth FLAIR.
Fig. 5, 6, 7 illustrate the Dice score and Recovery rate comparisons across tumor categories. Dashed lines represent the lower and upper bounds. They show that segmentation performance with the predicted FLAIR modality from Score- Setrr--Fuiion-basee models outperforms other methods, as Score-Fusion-basee models are constantly positioned higher than others.
We also show qualitative results the tumor segmentation task, on TPDM, TOSM, and Score-Fusion built based on these two models. Fig 8 and Fig show the results in super-resolution, Fig 10 and Fig 11 show the results in modality translation, and Fig 12 and Fig 13 show the results given both conditions. In Fig 8, 13, 12, and 13, in the sagittal plane, we observe that our models help segmentation model capture branch coming out of the whole tumor, indicated by red bounding box. This branch is only partially captured or entirely missed in predictions from other models. This shows that our model yields more precise predictions, allowing the tumor segmentation model to delineate the entire tumor boundary more accurately.
5

---

Introducing 3D Representation for Dense Volume-to- Volume Translation via Score Fusion
Dice Score Comparison for Super Resolution
X
TPDM
X
TOSM
07.06
ours: TOSM 44.30%
Ours-TPDM
87
X Ours-TPDM 43.80%
ours-TOSM
Upper Bound
07.77
TPOM 40.46%
Lower Bound
87.21
TOSM 51%
85.82
0%
02.71
100%
curs-TPOM
TOSM 54.55%
82.49
46.27%
01.20
100%
80.94
ours-TPOM 71.69%
80.87
ours-TOSM 63.94%
80.80
TOSM 66.28%
00.49
TPDM
20.62%
80.30
0%
TC
WT
ET
Figure 5: Comparison Dice scores and recovery rates for super-resolution. The value on the represents the Dice score, while the value on the right represents the recovery rate.
6

---

Introducing 3D Representation for Dense Volume-to- Volume Translation via Score Fusion
Dice Score Comparison for Modality Translation
TPDM
89.17
X
TOSM
79.21
X
X
OUrs-TPDM
78.73
X
OUrs-TOSM
ours-TOSM
Upper Bound
78.51
X
ours-TPDM
Lower Bound
77.74
TPOM
82.71
78.04
X
ours-TOSM
77.94
TOSM
ours-TPOM
77.28
81.20
78.64
78.37
78.22
ours-TPOM
TC
WT
ET
Figure 6: Comparison of Dice scores and recovery rates for modality translation. The value on the left represents the Dice score.

---

Introducing 3D Representation for Dense Volume-to- Volume Translation via Score Fusion
Dice Score Comparison for Both Conditions
X
TPDM
09.17
100%
TOSM
87.98
ours-TOSM
49.69%
X
ours-TPDM
07.91
ours-TPOM
46.67%
X
ours-TOSM
Upper Bound
87.69
TPDM
Lower Bound
87.27
TOSM 19.25%
82.71
100%
82.61
X
ours-TOSM 75.35%
82.54
X
TOSM 57.81%
82.46
ours-TPOM
38.66%
82:45
36.31%
81.20
100%
80.89
ours-TOSM 85.72%
TOSM 57.79%
80.74
TPDM 49.28%
80.77 04
ours-TPDM 48.97%
80.30
TC
WT
ET
Figure 7: Comparison of Dice scores and recovery rates for both conditions. The value on the left represents the Dice score, while the value on the right represents the recovery rate.

---

Introducing 3D Representation for Dense Vollue-to-Volume Volume Translation via Score Fusion
Original FLAIR
Downsampled FLAIR
Ours-TPDM
TPDM
Figure 8: Qualitative results for the downstream task, tumor segmentation, in super-resolution task.
Original FLAIR
Downsampled FLAIR
Ours-TOSM
TOSM
Figure 9: Qualitative results for the downstream task, tumor segmentation, super-resolution task

---

Introducing 3D Representation for Dense Vollue-to-Volume Volume Translation via Score Fusion
Original FLAIR
Ours-TPDM
TPDM
Figure 10: Qualitative results for the downstream task, tumor segmentation, in modality translation task
Original FLAIR
Ours-TOSM
TOSM
Figure 11: Qualitative results for the downstream task, tumor segmentation, in modality translation task
10

---

Introducing 3D Representation for Dense Vollue-to-Volume Volume Translation via Score Fusion
Original FLAIR
Downsampled FLAIR
Ours-TPDM
TPDM
Figure 12: Qualitative results for the downstream task, tumor segmentation, in both-condition task
Original FLAIR
Downsampled FLAIR
Ours-TOSM
TOSM
Figure 13: Qualitative results for the downstream task, tumor segmentation, in both-condition task

---

Introducing 3D Representation for Dense Volume-to- Volume Translation via Score Fusion
Table 11: Architecture for 2D diffusion model. Each ResnetBlock consists of conv2D layers of the same channel and
skip connection. All ResnetBlocks are used with time embedding.
layers
parameters
input
Conv3d
in _n_ch: out_cht ch: 64, kernel: 3x3, stride: 1, pad: 1
Linear
in in_ch:66,, out_ch: 256
Time_ Embed
Activateion
Swish in_ in__ch:25, out_ch: 256
Linear
ResnetBlock
in_ch:64, out_ch: 64
ResnetBlock
in in_ch:64, out jh_ 64
Downsamplc_block block__
Downsample(Conv30)
in in_ch:64, out_ch: sttrkre))l:33,strdde::))))
ResnetBlock
in_ch:64, out_ch: 128
Downsample_bn block
ResnetBlock
in in_ch:128 out out_ch: 128
Downsample(Conv//
in_ 128, out_ch: 128,kernel:3x3. stride:2
ResnetBlock
in_ch:128, out_ch: 256
Downsample_ block
ResnetBlock
in ch:256, out ch: 256
Downsample(Convãd)
in_ch:256, oltooutt_hhhhh ch: 256.kerne1:3x3, stride:2
ResnetBlock
in_ch:256, out_ch- ch: 512
Downsample_bles block_ 4
ResnetBlock
in in_ch:511, out out_ech: 512
ResnetBlock
in_ch:512, out_ch:: 512
Middle
ResnetBlock
in in_ch:512, out ch: 512
Resnet lock
in_ch:512,, out ch: 512
Upsample_block_1
ResnetBlock
in_ch:512, out_ch: 512
Upsample
Conv3d and Finterpolate
ResnetBlock
in in_ch:512, out out_ch: 256
Upsample_block.
ResnetBlock
in_ch:256, out_ch: 256
Upsample
Conv3d and Finterpolate in ch:256, out_ch: 128
ResnetBlock
Upsample_blooe block___llkk333
ResnetBlock
in in_ch: 128, out_ch, 128
Upsample
Conv3d and Finterpolate
ResnetBlock
in mn_ch:128, out_ch: 64
Upsample_block block_4
ResnetBlock
in in_ch:64, out out_ch: 64
Normalize
64 nn.
Out
Activation
Conv3d
in ch:64, out ch: 1, kernel: 3x3, stride: 1, pad:
12

---

Introducing 3D Representation for Dense Volume-to- Volume Translation via Score Fusion
Table Architecture 3D diffusion model. Each ResnetBlock consists of 2 conv3D layers of same channel and a skip connection, All ResnetBlocks are used with time embed with an embedding layer, as well as gradient checkpoint
layers
parameters
input
Conv3d
in__hh ch: out _h: 64, kernel: 3x3, stride: 1, pad: 1
Linear
in_ch:66, out out_ch: 256
Time_Embed
Activateion
nn.SiLU
Linear
in ch:256, out _h: 256
ResnetBlock
ch:64, out_ch: 64
ResnetBlock
in mn_ch:64, out_ch: 64
Downsample_bloc block
cictuue_iijeetee_fii from__D
in in_ch:64, out out_ch: 64)
Downsample(Conv330
in in_ch:64, out_ch: 64,kerne1:3x3, stride:2) stridee
ResnetBlock
in in_ch:64, out_ch: 128
Downsample_block.
ResnetBlock
in_ch:128, out_ch: 128
Feature Feature_injeteee
injeteed from 2D 2D
in in_ch:128, out_ch 128)
Downsample(Conv/ll
in_ch: 128, out_ch: kerneeeee::33, stride:2 striddd
RosnetBlock
in_ch: out_ch: 192
Downsample block 3
ResnetBlock
in mn_chh192. _ut_ 192
Feature ceattre_injeeced. from_ 2D
in 1b_chh192, out_ch: ch: 192)
Downsample(Convããd
in ch: 192, out out__thht 192,kemel:3x3, stride:2
ResnetBlock
in in__ch:192, out_chi _ht_ch:: 256
Downsample_bleck block____lk__4_ 4
ResnetBlock
in_ch:256, out_ch: 256
Feature sictie___jeeeeedfoooe from_2D
in_ch:256, out_ch: 256)
ResnetBlock
in in_ch:256, out __t 256
Middle
ResnetBlock
in_ch:256, out_chi 256
ResnetBlock
in in_ch:256, out_ 256
Upsample_block
ResnetBlock
in in_ch:256, out_ch: 256
Upsample
Conv3d and Finterpolate
ResnetBlock
in ch:256, out_ch 192
Üpsample_block block_
ResnetBlock
in_ch:192. out_ch: 192
Upsample
Conv3d and Finterpolate
ResnetBlock
in mt_cch192. out_ 128
@ysample_blocke
ResnetBlock
in mn_ch:128, out_ch: 128
Upsample
Conv3d and Finterpolate
ResnetBlock
in_ch: out_ch: 64
Shhample_blocke 4
ResnetBlock
in_ch:64, out_ch: 64
Normalize
64
Out
Activation
nn.SiLU
Conv3d
in ch:64, out ch: kernel: 3x3, stride: 1, pad:
13

---

Introducing 3D Representation for Dense Volume-to- Volume Translation via Score Fusion
Table 13: Architecture for the smaller variant of 3D diffusion model. Again, each ResnetBlock consists of conv3l layers of the same channel and skip connection. All ResnetBlocks are used with time embedding with an embedding layer, as well as gradient checkpoint. We used smaller number of channels for each layer and omitted the feature injection from 2D
layers
parameters
input
Conv3d
in_ch: out_ch: 32, kernel: 3x3, stride: pad: ]
Linear
in in_ch:32, out_ ch: 128
Time Time_Embed
Activateion
nn.S
Linear
in [n__ch:128, out ch: 128
ResnetBlock
in_ mt__ch:2,, out sut_chh: 32
ResnetBlock
in_ch:32, out out_chhtthhhooo __t 32
Nownsample__loock
Downsample(Conv33
in mn_3h:32, outoooottooooo _ut__h:: 32,kermel:3x3, stride:2)
ResnetBlock
in in_ch:32, out cut 64
Downsample_block
ResnetBlock
in_ch:64, out_ch: 64
Downsample(Conr/e
in ch:64, out out_ch: 64.kermel:333, stride:2
ResnetBlock
in mn_ch:64, out ch: 64
Downsample_block -loockkkle_bbokk33 3
ResnetBlock
in in_ch:64, out_cht ch: 64
Downsample(Conn.30
in ch:64, out ch: 64.kernel.3x3, stride:2
ResnetBlock
in in__ch:3,, out out_ch: 128
Downsample_block,
ResnetBlock
in_ch: out out_ch: 128
ResnetB
jn__ch128 out ch: 128
Middle
Resnet
iin_ch: 128, out _ut 128
Resnet
in_ch: 128,, out_ch: 128
Upsample_blaos block_1
ResnetBlock
in_ch: c281 out out_ 128
Upsample
Conv3d and Finterpollte
ResnetBlock
in_ch:128, out_ch: 64
Upsample block
ResnetBlock
ch:64, out _ut_ch: 64
Upsample
Conv3d and Finterpolate in mn_ch:64, out_ch: 64
ResnetBlock
Upsample_bloce block 3
ResnetBlock
in ch:64, out ch: 64
Upsample
Conv3d and Finterrpolate
ResnetB
in mn_ch:64, out cht 32
Upsample
@ossapllebbloc___ block_4 4
ResnetBlock
in_ch:32, ch: 32
Normalize
64
Out
Activation
Sn.
Conv3d
in in_ch:32, out_ch: kernel: 3x3, stride: 1, pad:
14

---

Introducing 3D Representation for Dense Volume-to-Volume Volume Translation via Score Fusion
Ground Truth
MADM
Ours- MADM
Input
Result
Uncertainty
Error
Result
Uncertainty
Error
Figure 14: Uncertainty awareness results on super-resolution task
15

---

Introducing 3D Representation for Dense Volume-to-Volume Volume Translation via Score Fusion
Ground Truth
MADM
Ours- MADM
Input
Result
Uncertainty
Error
Result
Uncertainty
Error
Figure 15: Uncertainty awareness results on super-resolution task
16

---

Introducing 3D Representation for Dense Volume-to-Volume Volume Translation via Score Fusion
Ground Truth
TOSM
Ours-TOSM
Input
Result
Uncertainty
Error
Result
Uncertainty
Error
Figure 16: Uncertainty awareness results on modality translation task
17

---

Introducing 3D Representation for Dense Voluue-to-Volume Volume Translation via Score Fusion
Ground Truth
TOSM
Ours-TOSM
Input
Result
Uncertainty
Error
Result
Uncertainty
Error
Figure 17: Uncertainty awareness results on modality translation task
18

---

Introducing 3D Representation for Dense Voluue-to-Volume Volume Translation via Score Fusion
Ground Truth
TOSM
Ours- TOSM
Input
Result
Uncertainty
Error
Result
Uncertainty
Error
Figure 18: Uncertainty awareness results given both conditions
19

---

Introducing 3D Representation for Dense Volume-to- Voluue-to-Volume Volume Translation via Score Fusion
Ground Truth
TOSM
Ours- TOSM
Input
Result
Uncertainty
Error
Result
Uncertainty
Error
Figure 19: Uncertainty awareness results given both conditions
20

---

Introducing 3D Representation for Dense Volume-to- Volume Translation via Score Fusion
Ground Truth
TPDM
Ours- TPDM
Input
Result
Uncertainty
Error
Result
Uncertainty
Error
Figure 20: Uncertainty awareness results on super-resolution for TPDM and Ours-TPDM
21

---

Introducing 3D Representation for Dense Volume-to- Volume Translation via Score Fusion
Ground Truth
TPDM
Ours- TPDM
Input
Result
Uncertainty
Error
Result
Uncertainty
Error
Figure 21: Uncertainty awareness results on super-resolution for TPDM and Ours-TPDM
22


Reviews:
[{"review_id": "YO1htPc8nD", "reviewer": "Reviewer_ELBW", "summary": "The paper proposes to improve 3D representation learning of medical image volumes in diffusion models. Unlike earlier methods that ensemble 2D models by averaging their weights, the proposed method does “fusion in score function space”. This is shown to improve. They show improved performance in downstream tasks such as segmentation. Additionally, “ensembling in score function space” is also shown to result in feasible training of large 3D Diffusion Models by initializing using pretrained 2D models and then fine-tuning the 3D model. The synthetic generation quality is also shown to be better both via qualitative examples and quantitative image quality metrics.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer ELBW, \n\nWe sincerely thank you for taking the time to review our paper and for your insightful and positive assessment, noting that \"the paper is very well written and easy to follow with good synthesis of literature\" and \"the contribution is well motivated.\"  Here is our response to your questions and reviews.\n\n## Weakness 1: challenge-specific metrics for tumor segmentation:\nWe have added lesion-wise Dice scores and 95% Hausdorff Distance (HD95) metrics below. Following [1], we set thresholds (150, 250, and 100 for TC, WT, and ET) to remove small predicted segmentations across all the evaluations. \n\n\nBoth Condition Results:\n\n| Model | lesion-wise Dice scores | HD95 |\n|:--:|:---: |:---:|\n| - |TC ---- WT ---- ET | TC ---- WT ---- ET |\n| TPDM | 0.779 0.689 0.742 | 18.107 18.937 32.011 |\n| Ours-TPDM| 0.783 0.692 0.745 | 17.864 18.790 32.046 |\n| TOSM | 0.779 0.677 0.745 | 18.287 20.223 32.287 |\n| Ours-TOSM | 0.781 0.698 0.748 | 17.428 17.982 30.638 |\n\nModality Translation Results: \n\n| Model | lesion-wise Dice scores | HD95 |\n|:--:|:---: |:---:|\n| - |TC ---- WT ---- ET | TC ---- WT ---- ET |\n| TPDM | 0.756 0.653 0.736 | 25.085 23.854 39.001 |\n| Ours-TPDM | 0.760 0.685 0.751 | 19.193 21.788 36.925 |\n| TOSM | 0.751 0.624 0.734 | 21.254 21.227 37.359 |\n| Ours-TOSM | 0.758 0.671 0.744 | 20.837 22.406 35.755 |\n\nSuper Resolution Results:\n\n| Model | lesion-wise Dice scores | HD95 |\n|:--:|:---: |:---:|\n| - |TC ---- WT ---- ET | TC ---- WT ---- ET |\n| TPDM | 0.778 0.688 0.746 | 18.079 18.894 31.793 |\n| Ours-TPDM | 0.780 0.692 0.745 | 18.389 18.157 33.842 |\n| TOSM  | 0.777 0.660 0.747 | 18.180 21.043 32.052 |\n| Ours-TOSM | 0.778 0.690 0.746 | 17.953 18.589 31.888 |\n\nOur methods show improvements over the baselines in most cases, with a particularly notable gain for WT lesion segmentation under the Modality Translation task. Moreover, lowered Hausdorff distances suggest that our method helps the segmentation model capture subtle lesion boundaries more accurately, indicating more precise morphological alignment. This further demonstrates that our model's generated results are more friendly for downstream segmentation models.\n\n[1] Ferreira, A.,et al. How we won brats 2023 adult glioma challenge? just faking it! enhanced synthetic data augmentation and model ensemble for brain tumour segmentation. arXiv preprint arXiv:2402.17317.\n \n## Weakness 2: sparse-view reconstruction in CT: \n\nThanks for raising this. We highly agree that sparse-view reconstruction in CT can substantially add weight to the claim of better representation learning.. We evaluated our method on sparse-view reconstruction with CT data:\n\nQuantitative results for sparse-view CT:\n| Model | PSNR | SSIM | MMD |\n|:---:|:---:|:---:|:---:|\n| TPDM | 33.78 | 0.922 |  58.2  |\n| Ours-TPDM| 34.53 | 0.931 |  20.78 |\n\nWe perform a sparse-view reconstruction task on the CHAOS dataset[1] with 36 views. The CHAOS dataset contains CT/MRI images of the liver. Each CT volume has 78-289 slices, each slice 512x512. During training, we cropped the third axis to have a common shape of 64. During inference, we adopted a simple sliding window strategy along the third axis. We resized each CT slice to 256x256, following TPDM[2].\n\nThese results on CT data demonstrate our method's effectiveness beyond MRI, enhance the generalizability of Score-Fusion to other modalities, and help support the claim of “medical image” in the title.\n\nAgain, we sincerely appreciate your thorough review and constructive suggestions, which have helped us improve our work.\n\n[1] CHAOS - Combined (CT-MR) Healthy Abdominal Organ Segmentation Challenge Data, A.E. Kavur et al\n\n[2] TPDM.", "claims_and_evidence": "Yes, compared to multiple baselines, improved representation learning and subsequently improvement in downstream tumor segmentation task is shown.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "Yes, the experiment design and analysis looks sound.", "relation_to_broader_scientific_literature": "The paper seems to have gone a step further in merging diffusion models to obtain a better 3D representation learning from pretrained 2D models. Whereas earlier methods simplify the problem of learning 3D data distribution by simplifying into product of 2D distributions, the proposed method ensembles the estimations from the 2D models to obtain weight vector and residual terms as learnable parameters for fine tuning.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "The paper is very well written and easy to follow with good synthesis of literature.  \n\nThe contribution is well motivated and shows good synthesis performance and improvement on downstream tasks. The possibility to train a full capacity 3D diffusion models without having to resort to compromises such as diffusion in latent space rather than full 3D volume space is very enticing. \n\n \n\nIn the downstream BraTS tumor segmentation task, it is advisable to report challenge-specific metrics such as Lesion-wise Dice Score (rather than Dice Score over the entire image) that penalizes not only overlap but also heavily penalizes missing lesions. Additionally, 95% Haussdorff Distance may also be reported. \n\nAdditional, downstream tasks such as sparse-view reconstruction and other volume-to-volume translation tasks could have been added to substantially add weight to the claim of better representation learning.", "other_comments_or_suggestions": "n/a", "questions_for_authors": "Again, to reiterate, Additional, downstream tasks such as sparse-view reconstruction and other volume-to-volume translation tasks could have been added to substantially add weight to the claim of better representation learning.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2j0zXRA0n1", "reviewer": "Reviewer_AnYS", "summary": "The authors study medical volume-to-volume translation, presenting Score Fusion, a 3D volumetric translation model. The model is based on a fine-tuning process, which starts from an average of 2D models. The method is tested on multiple tasks on two medical datasets, being compared with a number of approaches.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer AnYS,\n\nWe want to thank you for your feedback noting that \"the topic is interesting and method is timely.\" and providing constructive suggestions for our paper.\n\n## Experimental Question 1, multi-scale SR: \n\nThanks for raising this point. We agree that multi-scale super-resolution is common practice. We tried to address your concern about performing super-resolution at multiple scales and to demonstrate the generalizability of different super-resolution scales. We performed experiments on 2x and 8x SR. Here are the quantitative metrics. (The 4x results are the original results presented in the paper.)\n\nQuantitative Results for 2x Super-resolution:\n\n| Model | PSNR | SSIM | MMD |\n|:---:|:---:|:---:|:---:|\n| TPDM | 33.22 | 0.929 | 23.48 |\n| Ours-TPDM| 34.61 | 0.947 | 10.67 |\n\nQuantitative Results for 8x Super-resolution:\n\n| Model | PSNR | SSIM | MMD |\n|:---:|:---:|:---:|:---:|\n| TPDM | 28.45 | 0.903 |  69.17  |\n| Ours-TPDM| 29.38 | 0.924 |  46.44  |\n\nWe can see that our model consistently outperforms baselines across multiple super-resolution scales for all three metrics, showing the generalizability to different super-resolution scales.\n\n## Question 2 Missing references: \nWe appreciate your suggestions [A, B] and will incorporate these in our final version:\n\n## Question 3, Typos: \nThank you for pointing this out. We will ensure consistency in section title capitalization in the final version and also check for other typos.\n\nAgain, we greatly appreciate your time and effort in reviewing our work and providing positive feedback and constructive advice.", "claims_and_evidence": "The claims are supported by empirical evidence.", "methods_and_evaluation_criteria": "The proposed solution is reasonable and efficient.", "theoretical_claims": "N/A.", "experimental_designs_or_analyses": "It is typical to perform super-resolution at multiple scales. This should be performed by the authors. Aside from this, I did not find any flaws in the experiments.", "relation_to_broader_scientific_literature": "The topic is interesting and method is timely.", "essential_references_not_discussed": "There are some relevant references on medical image-to-image translation that are not acknowledged, e.g. [A, B]\n\n[A] Haimour, Fatima, Rizik Al-Sayyed, Waleed Mahafza, and Omar S. Al-Kadi. \"Bidirectional brain image translation using transfer learning from generic pre-trained models.\" Computer Vision and Image Understanding 248 (2024): 104100.\n\n[B] Ristea, Nicolae-Cătălin, Andreea-Iuliana Miron, Olivian Savencu, Mariana-Iuliana Georgescu, Nicolae Verga, Fahad Shahbaz Khan, and Radu Tudor Ionescu. \"CyTran: A cycle-consistent transformer with multi-level consistency for non-contrast to contrast CT translation.\" Neurocomputing 538 (2023): 126211.", "other_strengths_and_weaknesses": "The paper is mostly easy to follow.", "other_comments_or_suggestions": "Section titles are not consistently capitalised.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DZfjYZvsjD", "reviewer": "Reviewer_vsot", "summary": "In this work, the authors focus on using diffusion models for 3d volume-to-volume translation tasks such as super resolution and modality translation. Since 3d volumes (characteristic of medical data) are too big to computationally run a diffusion model, the authors first train two diffusion models in perpendicular axes. Then, they train a 3d diffusion model which is able to fuse both views to reconstruct the desired volume.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer vsot,\n\nThank you for your insightful review of our work. Here is our response:\n\n## Experimental Question 1, LDM for diffusion framework: \nThe reason why we are not using LDM is related to our task setting.\n\n__Intuitively,__ the main aim of Super-resolution is to recover/generate the high-frequency detail from a lower resolution image. In both LDMs, the high-frequency details are mainly generated with an autoencoder, which may result in worse performance in super-resolution. Therefore, we chose DDPM over LDM, in which the high-frequency detail is carefully modelled with a diffusion process, although LDM could potentially improve computational efficiency. \n\n__Empirically,__ we find that the 3D LDM's autoencoder is usually unreliable. At an early stage of this project, we tried 3D LDM using medical diffusion[1]. Although the model can reconstruct the image with semantically correct and realistic results, the reconstruction error is already pretty large. resulting in a __~25__ reconstruction PSNR. (We get a reconstruction PSNR of __26.23__ on a subset of the validation set of the BraTs dataset). We do acknowledge there are more advanced 3D LDM recently, e.g. MAISI[3], but they rely on much more data (more than 20 datasets combined for MAISI) to train a reliable auto-encoder model, which makes an unfair comparison between models.\nIn addition, if we use perpendicular 2D LDM, their diffusion process will be in different perpendicular 2D latent spaces, which makes the fusion of 2D scores difficult. Therefore, we do not use 2D LDMs such as Make-A-Volume[2].\n\nAnother reason is that Score Fusion is the first work to explore a learning-based method for score function fusion to the best of our knowledge. We believe that using a more standard and vanilla version of the diffusion model can provide more solid and generalizable insights in terms of score-function fusion.\n\nIn future work, we may consider extending score fusion to latent diffusion models in a text-to-image setting, where semantic information is better supported by the LDM. We will also include the negative results for 3D LDM and the discussion in the future version of our paper. Thank you for your insightful review. \n\n[1] Medical Diffusion: Denoising Diffusion Probabilistic Models for 3D Medical Image Synthesis.\n\n[2] Make-A-Volume: Leveraging Latent Diffusion Models for Cross-Modality 3D Brain MRI Synthesis.\n\n[3] MAISI: Medical AI for Synthetic Imaging\n\n## Experimental Question 2, Statistical significance: \n\nWe have already included the std values in the Supplementary Material in Table 5, confirming that our improvements are statistically significant. In the future version of our paper, we will show the results more directly in the experiment section for clarity.\n\nAgain, we appreciate your recognition of our work's potential for real-world applications. We would like to express our sincere gratitude for taking the time to review our paper", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "- Yes, the authors conduct experiments on tasks like super resolution and modality translation which are common tasks in medical volume-to-volume translation.\n- They evaluate on metrics like PSNR, SSIM, FID which are standard metrics.\n- They evaluate both tasks on two datasets: Brats and HCP\n- Authors also conduct downstream task (medical segmentation) to justify the superior quality of their generated volumes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "- Good experiments. See Method section above for more details.\n- Is there a reason why the authors did not considere LDM for the diffusion framework? Because DDPM-based models have image size constraints in 2D (authors had to use 192x192 sizes which is small).\n- The standard deviation and t-test of the results was not provided. Hence, unsure if the quantitative performance is statistically significant or not.", "relation_to_broader_scientific_literature": "This work relates to practical medical volume generation. Existing works in literature focus on 2D slices which has limited applicability in the real setting. Since the authors' work is in 3D, it has potential for real-world usage.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "My score is reduced mainly due to the comments in the Experiments section. If the authors can address this, then i can consider increasing the score after discussing with other reviewers as well.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "none.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vQ6fq2wTpE", "reviewer": "Reviewer_XUPB", "summary": "The authors present Score-Fusion, a volumetric translation model that learns 3D representations by assembling perpendicularly trained 2D diffusion models in score function space. It can reduce the 3D training computational cost and data demand, and the results is comparable in different downstream tasks. However, limitations exist.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer XUPB,\n\nThank you for your detailed and constructive feedback. Here is our response to the raised questions and concerns:\n\n## Weakness (1-2): Generalizability beyond brain MRI: \nThank you for your insightful review. We highly agree that results on datasets beyond brain MRI can significantly strengthen our paper. \nWe evaluated our method on sparse-view reconstruction with CT data:\n\nHere are the quantitative results for sparse-view CT:\n\n| Model | PSNR | SSIM | MMD |\n|:---:|:---:|:---:|:---:|\n| TPDM | 33.78 | 0.922 |  58.2  |\n| Ours-TPDM| 34.53 | 0.931 |  20.78 |\n\nWe perform the sparse-view reconstruction task on the CHAOS[1] with 36 views. The CHAOS dataset contains CT/MRI images of the liver. Each CT volume has 78-289 slices of resolution 512x512. During training, we cropped the third axis to have a common shape of 64. During inference, we adopted a simple sliding window strategy along the third axis. We resized each CT slice to 256x256, following TPDM[2].\nThese results on the CT data demonstrate our method's effectiveness beyond MRI, showing the generalizability of Score-Fusion to other modalities (CT). It also demonstrates our method's effectiveness on a different organ (liver). The results help support the claim of “medical image” in the title. \n\nWe acknowledge that there are many other modalities, such as ultrasound, and there are other organs, such as the breast. The results on these datasets will also enhance our claims. Unfortunately, given the limited time during rebuttal, we are not able to include even more dataset results and we leave these applications to future work. \n\nHowever, we believe it is a common practice to use two modalities, especially CT and MRI, for papers targeting volumetrics medical image translation, according to multiple previous works, including TPDM[2], ScoreSDE[3], DiffusionMBIR[4], etc.\n\n[1] CHAOS - Combined (CT-MR) Healthy Abdominal Organ Segmentation Challenge Data, A.E. Kavur et al\n\n[2] TPDM.\n\n[3] Solving Inverse Problems in Medical Imaging with Score-Based Generative Models, Song et al.\n\n[4] Solving 3D Inverse Problems using Pre-trained 2D Diffusion Models, Chung et al.\n\n## Weakness 3: Non-isotropic medical imaging:\nAs mentioned above, our new experiments on CT data are trained on 256x256x64 data due to the nature of the CHAOS dataset. Due to the time limit of the rebuttal, we did not get results on the fastMRI dataset yet, where the resolution is 320x320x20. __We plan to include the fastMRI results in the second round of rebuttal. We hope this can still be considered.__\n\nAs an extension of score-fusion, we tried to extend Score-Fusion for video super-resolution. We used MGLD-VSR[5] as the 3D model and used two perpendicular models on the time-space slices (i.e., (x-t) slices and (y-t) slices). Due to the capacity requirement of the video model, we are only using 5 consecutive slices in this problem, resulting in a 1280 x 960 x 5 or 2560 x 1920 x 5 resolution for the volume. This can potentially help address the concern of extreme resolution. Here are the quantitative results on the VideoLQ dataset[6]. We use DOVER[7] as a metric, which is a spatial-temporal metric for video quality assessment.\n\n| Model | DOVER($\\uparrow$) |\n|:---:|:---:|\n| MGLD|  0.748 |\n| Ours-MGLD| 0.755 | \n\nAlthough the video results might be beyond the scope of the rebuttal, we would be glad if these preliminary results could help address the concern and could be included in the current paper.\n\n[5] Motion-Guided Latent Diffusion for Temporally Consistent Real-world Video Super-resolution\n\n[6] Investigating tradeoffs in real-world video super-resolution\n\n[7] Exploring Video Quality Assessment on User-Generated Contents from Aesthetic and Technical Perspectives\n\n## Question 1: \nSee weakness 1-2\n\n## Question 2-3:\nNetwork architecture: We have already included a detailed model architecture in Supplementary Material section F and Tables 10, 11, 12. We will emphasize this by pointing to the Supplementary Material section in the experiment section of the main paper for clarity.\n\n## Question 4:\nWe acknowledge that a lot of modality translation is a task setting that is not directly useful in the medical imaging domain, such as MRI to CT, where MRI lacks certain information due to physical properties, as the reviewer mentioned. \n\nHowever, from a multi-modal learning perspective, efficiently training a multi-modal model by fusing knowledge from multiple single-modal models can be useful due to the difficulty in training a foundation model. We will modify the discussion on this part in our final version of the paper.\n\nAlso, we want to emphasize that our problem setting ( i.e., low resolution flair + T1ce => high resolution flair) does not have the issue that the reviewer mentioned. \n\nI hope the above rebuttal addresses your concerns. Again, we would like to express our sincere gratitude for taking the time to review our paper and providing insightful questions.", "claims_and_evidence": "I have concerns about the claim, since the authors made very broad claims about “Medical Image” in general. \n1. A concern is that the experimentations were done on brain only. Brain is well-known to be relatively the best structured and clean among all organs, compared to breast, liver, prostate, etc. \n2. The experimentations were done on single modality of medical image - MRI only, without any evidences from CT, ultrasound, Mammogram, etc.\n3. The authors ignoring the physical limitations of the medical imaging. It seems like the authors are treating all medical imaging as isotropic or nearly-isotropic. However, many of the MRI sequences do not generate such results with isotropic properties. For example, prostate MRI. T2Tse prostate MRI suffers from 320x320x20 dimensionality, which means higher resolution in x-y plane (320x320), and much lower resolution (20x320) in both x-z and y-z planes. In this case, I would doubt if the authors’ methodology still works or not.\nTherefore, although the authors have done experimentations on two different downstream tasks, the organ & image modality & image physics limitations let me feel concerned about the generalizability of the proposed method, and the validity of the claims made by the authors.", "methods_and_evaluation_criteria": "The concerns are stated under “Claims and Evidence” together.", "theoretical_claims": "Yes. About Diffusion models and model fusions.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "More efficient way of using 2D Diffusion models to mimic 3D Diffusion models’ results, in Brain and in MRI specifically.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. About generalizability, I have mentioned some of my concerns under “Claims And Evidence”, please response to them\n2. I’m not clear of the network architecture, although you mentioned it is UNet-like. I don’t think you need to describe everything about the parameters you set in this network, but apparently the current description is not enough.\n3. “To further enhance the speed of multi-modality fusion, we employ a smaller variant of our model, adjusting the number of channels in each layer.” Would like to learn more about this.\n4. “For instance, DDMM-Synth (Li et al., 2023) suggested using both MRI and low-resolution CT scans to produce high resolution CT images. Training a separate model for each possible combination of input conditions would result in exponential time complexity, making it generally impractical …” I hold my opinion regarding the discussions here. T2 MRI -> T1 MRI image translation, and the T2 MRI -> CT image translation, are VERY different as MRI and CT have different physical properties. T2->T1 is within the same image modality, however, MRI->CT is using two different image modalities. From my understanding, the statement made by Li et al. 2023 was majorly becuase - at least you have to get some physics information of CT so that you can create an “accurate” high-resolution CT image that won’t impact the diagnosis. For example, it is possible that some tiny lesions could be identified by the CT but cannot be identified by the MRI. If not providing ANY CT information, the translation results is useless in terms of diagnosis.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper initially received mixed reviews. After rebuttal and reviewers' discussion, all the reviewers provided positive ratings as the concerns have been adequately addressed by the rebuttal. As a result, this work received all positive ratings. Meta-reviewer agreed with the reviewers' recommendation and made final decision.
Decision: Accept (poster)