Towards General Modality Translation with Contrastive and Predictive Latent Diffusion Bridge
Authors: Nimrod Berman
Omkar Joglekar
Eitan Kosman
Dotan Di Castro
Omri Azencot
Venue: NeurIPS 2025 poster ()
Keywords: Modality Translation
Diffusion Models
General Cross-Domain Generation
Latent Space
Primary area: deep_learning
TL;DR: The paper introduces a general framework for modality translation using latent diffusion bridges, enabling translation across arbitrary modalities without shared dimensions

Abstract: Recent advances in generative modeling have positioned diffusion models as state-of-the-art tools for sampling from complex data distributions. While these models have shown remarkable success across single-modality domains such as images and audio, extending their capabilities to *Modality Translation (MT)*, translating information across different sensory modalities, remains an open challenge. Existing approaches often rely on restrictive assumptions, including shared dimensionality, Gaussian source priors, and modality-specific architectures, which limit their generality and theoretical grounding. In this work, we propose the Latent Denoising Diffusion Bridge Model (LDDBM), a general-purpose framework for modality translation based on a latent-variable extension of Denoising Diffusion Bridge Models. By operating in a shared latent space, our method learns a bridge between arbitrary modalities without requiring aligned dimensions. We introduce a contrastive alignment loss to enforce semantic consistency between paired samples and design a domain-agnostic encoder-decoder architecture tailored for noise prediction in latent space. Additionally, we propose a predictive loss to guide training toward accurate cross-domain translation and explore several training strategies to improve stability. Our approach supports arbitrary modality pairs and performs strongly on diverse MT tasks, including multi-view to 3D shape generation, image super-resolution, and multi-view scene synthesis. Comprehensive experiments and ablations validate the effectiveness of our framework, establishing a new strong baseline in general modality translation. For more information, see our project page:  https://sites.google.com/view/lddbm/home.

Towards General Modality Translation with Contrastive and Predictive Latent Diffusion Bridge
Nimrod Berman 1,2 Omkar Joglekar 1,3
Eitan Kosman'
Dotan Di Castro!
Omri Azencot
Bosch AI Center Ben-Guriin University of the Negev
Technical University of Munich
Abstract
Recent advances in generative modeling have positioned diffusion models state- of-the-art tools for sampling from complex data distributions. While these models have shown remarkable success across single-modalty domains such images and audio, extending their capabilities to Modality Translation (MT), translating in- formation across different sensory modalities, remains an open challenge. Existing approaches often rely on restrictive assumptions, including shared dimensionality. Gaussian source priors, and modality-specific architectures, which limit their gen- erality and theoretical grounding. In this work, we propose the I Latent Denoising Diffusion Bridge Model (LDDBM), general-purpose framework for modality translation based on latent-vant variable extension of Denoising Diffusion Bridge Models. By operating in shared latent space, our method learns bridge between arbitrary modalities without requiring aligned dimensions. We introduce a con- trastive alignment loss to enforce semantic consistency between paired samples and design Jomain-agnosti, encoder-decoder architecture tailored for noise pre- diction in latent space. Additionally, we propose a predictive loss to guide training toward accurate cross-domain translation and explore several training strategies improve stability. Our approach supports arbitrary modality pairs and performs strongly on diverse MT tasks, including multi-view to 3D shape generation, image super-resolution, and multi-view scene synthesis. Comprehensive experiments and ablations validate the effectiveness of our framework, establishing new strong baseline in general modality translation. For more information, see our project page: https: google wwt:://ee/wdd///////e/./e/aceaee
1 Introduction
Generative modeling has progressed rapidly in recent years, with diffusion models emerging as powerful framework for sampling from complex, high-dimensional distributions [57]. These models have achieved impressive results across range of domains, including images, audio, and video 30 g 21]. Their inherent conditioning module naturally lends itself to the next step: Modality Translation (MT), the task of mapping information across different sensory modalities, such as from text image or from audio to video 72 68 2. Unlike traditional generation tasks, MT often involves bridging modalities that differ significantly in dimensionality. structure, and semantics, posing unique challenges for modeling shared representations and consistent cross-modal synthesis.
Recent works on distribution translation, notably Denoising Diffusion Bridge Models (DDBMs) 75). offer principled framework for modeling transitions between arbitrary distributions. In parallel, diffusion models have been widely applied to MT tasks such as text-to-image 49] and text-to-3D generation 46 (6), typically by conditioning on auxiliary signals and exploiting modality-specific inductive biases. However, these approaches often assume translation between distributions of similar dimensionality or insert strong data biases into their design. This limits their applicability to more general MT scenarios involving heterogeneous modalities, e.g., translating from images to 3D
Equal contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

structure or from low-rusoluionn quality to high-resolution one. Moreover, many implementations rely on neural architectures tailored specific data types, such as U-Nets [52], which are well-suited to grid-based data but struggle with abstract or unstructured modalities. These limitations highlight the need for general framework that minimizes reliance on modality-seecific design and remains theoretically grounded across a broad range of cross-modal tasks.
attempt to bridge the gap between diffusion models and general modality translation. However, most existing works adopt this formulation primarily for computational reasons- reasons-either to reduce the cost of the diffusion process itself [50, 751. or to improve sampling efficiency through direct endpoint mappings (24] and cannot be applied to general MT purposes. In order to take step towards a general effective framework, we suggest a new latent extension of DDBMs that facilitates generality for MT tasks along with effective performance.
Latent Diffusion Bridges such [24 36 29
25 70
LDDBM extends DDBMs to a shared latent space and learns bridge between embeddings of disparate modalities, enabling translation without requiring the two domains to share dimensionality. Concretely (see Fig. 2), we: (i) encode source and target examples into common latent space using simple modality-specific encoders; (ū) apply a latent diffusion bridge implemented with a Transformer denoiser that uses cross-attention to condition on the source latent while predicting the target latent; and (iii) decode the predicted latent back to the target modality. To promote semantic consistency, we utilize paired training data with a contrastive alignment loss (inspired by CLIP (8)) that pulls corresponding pairs together and pushes unrelated pairs apart. To ensure the bridge performs end-to-end translation, and not just local denoising, we add a predictive loss that compares the final decoded output to the ground-truth target. Finally, we reduce architectural bias with a domain -agnosic denoiser and use a simple yet effective iterative training scheme that alternates between alignment and denoising steps, improving stability and performance.
This approach offers several key advantages. First, it supports arbitrary modality pairs without relying on fixed priors, shared dimensionality constraints, or hand-crafted architectures-asid from the encoder and decoder used project data into the latent space. Second, the contrastive loss enhances semantic coherence across domains. and our bridge architecture achieves strong performance on a range of tasks, including multi view to 3D shape generation [55), super-resolution [38]. and scene generation from multi-view cameras [2].. Additionally, we provide simple, flexible, and theoreti- cally grounded framework for MT tasks and demonstrate its state-of-the-art performance. Finally, we conduct a thorough ablation study to evaluate each component of our framework, identifying the key elements that contribute to its improved performance.
2 Related Work
Diffusion models [57 19 and score-based generative models 22 have emerged as powerful approaches for wide range of modalities generation 4 42, 67
69
5 14
12), often surpassing
the performance of GANs [5], which previously defined the ttate-of A key strength of diffusion models lies in their inherently conditional architecture, which enables injecting of auxiliary signals such as diffusion time steps and text embeddings. This architectural flexibility has positioned diffusion models a natural fit for general Modality Translation (MT) tasks. Advancements in sampling techniques [99 58 27), particularly innovations like classifier-fre guidance [20 [) have further enhanced the generative capabilities of these models in diverse MT scenarios. While these developments are broadly applicable and do not rely on domain-specifit inductive biases, several works have leveraged the conditional structure of diffusion models with tailored data-specific adaptations to excel in particular MT tasks [21 46 38 66). Since our objective is general general-purposse modality translation, our method is best compared to general diffusion- approaches that maintain
broad applicability across domains 27 35
Distribution Translation. While diffusion models are among the most effective tools for general modality translation (MT) tasks, Denoising Diffusion Bridge Models (DDBMs) have shown strong performance for translating between modalities within the same data domain 75). This framework extends the standard denoising diffusion paradigm [19 from noise-to-dai generation to enabling data-to-data translation between paired distributions. Other principled approaches, such as Optimal Transport (OT) M| and Schrödinger bridges [59
32 54 offer alternative ways to model transitions
between arbitrary distributions. Diffusion Bridge Implicit Models (DBIMs) 73 generalize DDBMs using Mar-ovnnn bridges over discretized timesteps, reducing resource demands while preserving marginal distributions and training objectives. Similarly, Consistency Diffusion Bridge Models

---

improve efficiency by learning the consistency function of the probability- flow ODE (PF-ODE), enabling direct prediction at arbitrary time points. Recent work [63] further mitigates PF- ODE singularity issues at the reverse-time initialization via posterior sampling, improving continuity and reducing discretization error. Despite this progress, DDBMs and their variants remain limited to shared-modalin settings and are not directly applicable to general MT tasks. Other frameworks,
(CDBMs) 16
such as CLIP [48], diffusion-bar models [99 (88, and multimodal models [53 3 [3 76.. leverage
shared embeddings or conditional generation to bridge modalities. However, end-to-end modality- to-modality translation remains underexplored and lacks unified methodologies. While unpaired
translation (77] and weakly paired setups [37] have received substantial attention, our work, as well
as the models above, operates within the paired translation paradigm [23]
Finally, we note several concurrent efforts that share conceptual similarities but differ substantially in scope and formulation. CrossFlow 33 and FlowTok [17] both employ flow-based formulations to map directly between text and image modalities. While effective within that specific domain, they rely on task- specific design choices (e.g., text-image tokenization, contrastive language supervision) and thus do not generalize to arbitrary modality pairs. In contrast, our work pursues general-purposs latent bridging frumework that supports heterogeneous modalities, including 3D and audio, and is not tied to any particular data type or dimensionality, Moreover, the underlying formulations differ: the above methods are based on Flow Matching, whereas our approach extends the Denoising Diffusion Bridge Model (DDBM) framework, leading to distinct training objectives and sampling dynamics. Another concurrent work, DPBridge [24), adopts a latent diffusion bridge for dense prediction but remains limited to image-to-image translation, leveraging a pre-trained Stable Diffusion backbone. Unlike DPBridge, our method introduces contrastive and predictive objectives for alignment, a modallty--agostii architecture trained from scratch, and comprehensive studies on training strategies, enabling translation across diverse modality pairs beyond images.
3 Background
Diffusion Models 57 have emerged as leading generative models, achieving state-of-the-arl per- formance across various tasks [19] 30). Intuitively, these models operate by mapping a latent representation, sampled from a simple prior distribution (c.g., Gaussian), through an iterative process to meaningful outputs such as images audio. However, their confinement to transformations that originate from a predefined simple distribution poses a fundamental limitation, as it restricts the ability of diffusion models to translate between arbitrary and potentially complex data distributions flexibly. Researchers have proposed several approaches to address the challenge of translating be- tween arbitrary distributions. This work focuses DDBMs (75), which extend diffusion models to connect endpoints from two arbitrary image distributions of the same dimensionality.
Denoising Diffusion Bridge Models extend diffusion models enable transformations between two arbitrary distributions by conditioning the diffusion process on fixed, known endpoints using the Doob's h--transform 100 Given two distributions p(2r)) p(æT) and for given joint distribution p( Tr). where the subscripts denote the time associated with a state, the Doob's transform allows the diffusion process be initialized a specific state zo at time 0 and terminate at a desired state ar at time T. Its associated forward stochastic differential equation reads,
d dx.- [x,, t) dt g(t) 1,, NT, T) + g(t) d
(1)
where the functions/ f, f, g can be calculated analytically, similar to adding noise with particular scheduler in standard diffusion models. DDBM aims to learn the reverse process, which is given by
d t) 9 (t) (Vz, log q( (T)) h(z,, t, ær, TT)] dt + g d d
(2)
where uh is the Wiener process in reverse time and V., log g(z, | IT) is known as the score function. Let so (2r, IT, 1) denote a neural network approximation of the score function. To train the above model, we consider the following loss objective.
Ex,, .t w(t) [50 2r,t)
Vz, log logg(zr 2o, [0,,,)) x2))
(3)
In practice, we utilize the variance exploding (VE) scheme in this work from 75 where V,,, log g(z, Fo, (T) (xT (x) of) and x, (r
|
Finally, a process can be formulated to approximately sample from p(zo (T) by reversing the diffusion bridge. Using a training set of paired samples drawn from p(zo, IT), it is possible to
3

---

learn how to map states from an arbitrary distribution p(ar) to another arbitrary distribution p(zo). However, DDBMs rely on the explicit assumption that both p(æp) and p(ær) reside in the same space IR for some d. Consequently, this framework, and similar approaches, are not applicable to our multi-modal setup, where paired data do not share the same dimensionality or underlying manifold. To overcome this challenge of translating between distributions of different dimensionality, we connect representations of the two endpoints achieved through encoders and decoders that can be potentially pre-trained, which bring the endpoints to the same dimensionality.
Method
In this section, first formulate the problem of task-agnostic translation. Then, we gradually present our approach to solving task-agnostic translation, depicted in Figure beginning with introducing the framework and objectives and then moving to improved and adapted objectives and model architecture. Finally, we discuss the training procedure to achieve optimal translation performance.
4.1 Problem Formulation
We study the general task of translation between two modalities. Although researchers have ex- tensively explored this task across various domains [66, existing methods typically tailor their solutions to specific tasks and depend heavily on architectural biases or strong supervision. In contrast, we propose a task-agnostic framework capable of translating between any two modalities represented by shared multimodal distributions p(z) and p(v), where joint distribution p(ar, y) exists. Given target modality distribution p(z), where x R$ and source modality distribution p(y), where y E R' with k * s, along with a dataset of paired samples (z. y) ~ Pasu( (x. y), the goal of multimodal translation is to learn model .M that approximates the conditional distribution p(x y) or p(y z), thereby enabling translation from one modality to the other.
4.2 Multimodal Translation Framework
While dimensional mismatches between modalities can sometimes be solved via simple heuristics, for example, padding a smaller image to match the size of a larger one, these solutions generally under-perform, and many tasks, such as translating 2D 3D data, are inherently more complex. To tackle this, we introduce a latent structure that bridges the modalities, providing a theoretical and practical framework for addressing this gap. Although related ideas have been explored in the context of computational reductions (75) and specific tasks 33 177. to the best of our knowledge, they have not been investigated in the setting of general multimodal translation. We model the translation process via a probabilistic latent bridge to realize this idea. We aim to estimate the conditional distribution p(r y): however, direct modeling of this distribution is often intractable. To address this, we introduce intermediate latent variables and associated distributions: p(zo (x) and p(2r y). where 20, 2T E Rd reside in a shared latent space. In settings where reconstruction of x is required, we additionally define a decoder distribution g(x z0). The core objective is thus re-framed as modeling the conditional p(z y) through the composition: p(xly) = p(2r (B) p(2o(2r) 9( (x=(2)). Here, p(2r y) and q(1' zg) serve as the encoder E, and decoder D2 and are implemented as modality-speciff neural networks (e.g., MLPs or CNNs). During training, the encoding of x through the encoder E. (p(20 z)) also required to compute the bridge loss, but this is not needed during inference. The bridge p(20 2r) is modeled using a DDBM module, although our framework is agnostic to this choice and can extended to other choices such as those proposed in [ 19 27 35
The choice of encoder and decoder is the only mrodllity-specing component in our framework. In our experiments, we employed either very simple neural architectures or pre-trained models.
4.3 Basic Latent Bridge Loss
Our multimodal translation objective comprises two loss components: autoencoder loss and diffusion bridge loss. The autoencoder reconstruction loss penalizes poor reconstruction of inputs. Let E, and D2 denote the encoder and decoder, respectively, of an autoencoder AE, for the modality, i.c., AE, = D2 o E, (z), then the autvencoder loss is given by
LAE_ = d(a,, AF, (x)))))
where d is a distance metric such as lg or LPIPS (7]. A similar objective can be formulated for the y modality. To train the latent diffusion bridge, we adopt the score matching bridge loss from the DDBM formulation, applied on the latent variables.
Lonige = E, E2..o,,r..
[1) 1s0( ZT, - V.. log ((gl(=0 27 27))²]
(5)

---

Basic
Ours
L.
Basic
Ours
Figure I: Left:: Super-resolution examples obtained with the basic approach vs. ours. Right: Latent space structure of the basic formulation vs. our latent space.
where 2; is a noisy version of 2n == (x) at timestep w(() is weighting function over time, and 2T == (v). Combined together, we obtain the following training objective
L
Laridge + LAE. + LAE
This objective can be optimized in two phases (reconstruction followed by bridge training) or jointly in an end-to-end fashion. Next, we show that this basic setup, while effective, insufficient from several different perspectives.
4.4 Our Bridge Approach
Our baseline loss formulation consists of three components: standard score matching loss applied to the diffusion bridge and two separate reconstruction losses-ont one for the source distribution and one for the target, each responsible for mapping data to its respective latent space and reconstructing it. This decoupled setup cleanly separates the modeling of the marginal distributions (via the autoencoders) from the bridging process, allowing each component to specialize in its subtask. However, bridging between arbitrary distributions in latent space presents two key challenges. First, because the bridging is performed entirely in the latent domain, fine-graing fine-grained details, such as high- high-frequency visual information, may not accurately preserved reconstructed. Second, the autoencoders trained independently and without explicit coupling, which can result in misaligned latent spaces and significant divergences between the encoded source and target distributions.
In Fig. [ we qualitatively illustrate the limitations of the basic formulation through two analyses: super-resolution (left) and t-SNE embedding [60] plots of encoded ShapeNet representations [64] (right), On the left, the high-resolution reconstructions generated using the loss defined in Eq. (6) (labeled "Basic") exhibit noticeable artifacts and lack high-frequency details, such the yellow hair ornament and fine facial features. On the right, the t-SNE plots show latent representations colored by semantic category. Circles denote encodings of multi-view images, while triangles represent encodings of 3D shapes. The center plot corresponds to the basic method, while the rightmost plot shows results from our approach. Although intra-modality clustering is relatively coherent both cases, the basic method fails to align semantically similar samples across modalities, revealing significant gap in cross-todality consistency within the latent space. To address these issues, we introduce predictive loss that constrains the entire encode-briee -bwddb-dddddoooo decode pipeline, encouraging accurate domain translation. In addition, we incorporate contrastive loss attract semantically similar examples and repel dissimilar ones, as further detailed below.
predictive loss. To improve the fidelity of the learned diffusion bridge, we introduce predictive loss that enforces alignment between the input data and its reconstruction after being mapped through bridge. Specifically, we encode the signal, apply the bridge transformation, decode the result, and compare it to the original input in pixel space. Formally.
Lpod = d (D, B (z).
where B denotes the bridge model. This loss encourages the model to preserve semantic content across the full mncdde-bridge- -eddde pipeline, serving a powerful regularizer for the bridge mapping. Importantly, as we empirically showcase Sec. 5 replacing the two separate autoencoding losses previously used with this single predictive term for the source and target domains improves
5

---

Translation
2: Takeen
Mux
Biocks
T.Tosess
Memary
DOOM Naaure
Discad
Medsiity [Denakedd- [Beroladd--caaatttssee
Ny DTPIT
[MASI
Decoge
Morslyy
Tolkrn
Modility 201
00000
Time Enoeddingg
Thie Step
Eribed
Figure 2: End-to-end training pipeline for our framework. During inference, 24 is replaced with 2T and Heun sampling is applied autoregressively to obtain For more details, see Fig.
generative performance. As a result, we reduce computational overhead, shorten training time, provide one-way supervision, and improve the overall robustness and stability of the model.
A contrastive loss. Misaligned latent representations across independent distributions typically addressed using alignment objectives. Common approaches include cosine similarity [48) and contrastive estimation [43] Cosine similarity encourages similar samples align by minimizing the angle between their embeddings, while contrastive estimation explicitly pulls together positive pairs and pushes apart negative pairs. In our setting, where we bridge source and target distributions, contrastive learning offers natural fit. Leveraging the paired structure in the data, we treat (zo, 2r) as a positive pair and all other examples in the batch negative examples. This setup allows us to effectively encourage alignment between related samples while preserving separation between unrelated ones. Formally, the loss defined as
$120, ZT
LinfoNCE log
6(z0, 2T) + Σ5 o(zo: =T)
where 2/ are negative samples from the batch, and the function p(u, v) = (/rluu///lll)) mea- sures the similarity between examples u and with = 0.5 being temperature parameter
Overall objective. We incorporate the bridge, predictive and contrastive losses into a single unified objective forming approach to training bridge between arbitrary distributions
C
Lbridge Spred + LinloNCE
4.5 Multimodal Translation Architecture
In what follows, we motivate our neural architectural design by considering existing approaches and suggesting specific modifications to our bridging task. U-Nets and iitention-based U-Nets [52 19 are widely used in diffusion models. Recently, Transformer-baut based architectures such DiT have demonstrated superior performance using a decoder-only design. However, in many previous works such as 334 architectures over decoder-only architectures in the context of natural language translation tasks. Inspired by the effectiveness the original encoder-decoder transformer architecture [61], we design denoiser that is better suited for the distribution bridging task (see Fig. 2 and App. A.2.2
31, the authors showease superiority of encoder-decoder transformer
Given the inputs x and y. we first use modality-peeiffi encoders generate latent vectors 2g and 27, respectively, which are token sequences of the same embedding dimension []] 48 The details of the encoders used in our experiments are provided App. A.2.1] In our design, we apply a censsformer-nncodee [6]] to the tokens 2r, which outputs an embedding we term as memory. Unlike the original UNet implementation of DDBM, which concatenates this information to input of the noised latent, our model consumes this memory using the cross-attention layers inside the wansformer-decodes This allows for more expressive conditioning. The transformer-decoder processes input of the tokens concatenated with sequence of learnable output tokens that are

---

Figure 3: Dultiview-to-3D--Saap translation example with LDDBM.
heavily inspired by context summarization tokens from previous transformer works such as ViT ||| and CLIP [48]. These learnable output tokens are obtained by adding the positional encoding to a learnable [MASK] token. We want to highlight that the number of tokens in and 2g are the same. To spatially align the output tokens with the 2 tokens, we add the same positional encodings to both 2₁ and =o to explicitly enforce spatial correspondence. The outputs of both the self- and cross-attentions and the feedforward networks in each transformer-decode block are modulated using the timestep embedding of 45 At the output of the transformer-decodee output tokens in further loss computation. A modality decoder then processes the output estimate &o to obtain 2. The MASK (or context- sulllct-summmrizationn token is special token used in many transformer architectures that attends to, and attended by, all tokens, including under causal masking. It learns single embedding summarizing the entire context window. This design parallels the [CLS] token in Vision Transformers (ViTs) for classification, where a prepended summary token serves as input to a downstream classifier. We empirically justify our design choices by comparing the effectiveness of our method to previous architectures in Tab.
we only use the
At inference time, we compute the memory embedding once for the whole Heun sampling process and use for all subsequent denoising steps. We also replace 2t with 2y tokens for the first denoising step and then sample autoregressively, which is standard protocol to sample from a diffusion process.
Training Procedure. We experiment with several training regimes for our framework: (1) two- step procedure, where we first train the autoencoders and then the diffusion bridge; (2) fully end-to- end optimization; and (3) an iterative scheme that alternates between optimizing reconstruction and bridge alignment. The motivation for exploring the iterative regime stems from the inherent conflict between the bridge and the encoder networks. The bridge assumes that both marginal distributions remain fixed during training; however, since the encoders the structure of the latent distribution, effectively 'breaking' the previously learned bridge. This conffict of objectives resembles adversarial setups, which inspired to incorporate practices from adversarial training [8 44 Among the three strategies, the iterative approach provides the best trade-off between training stability and final performance (see Sec. A.3.4).
trainable, they continuously reshape
Experiments
In this section, we present comprehensive experiments that evaluate seneral-purpont framework for modality translation across a diverse set of tasks and domains. We benchmark our approach on four tasks and perform extensive ablation studies to assess the contribution of architectural components, loss functions, and training strategies. Due to space constraints, we include in App. A.3. the Hultiview-to-Scenee Sceene-Ocuupancy Generation task, in App. A.3.2 the DIV2K Super-Resolution task, and present a complexity analysis in App. A.3.3 Additional details regarding our experimental setup and implementation can be found in App. A.1 and App. A.2
5.1 Sollivveewtto3D--Shee
Generation
We begin with the task of generating 3D shapes from 2D multi-view images. In this task, general multimodal model given multiple images of the same object from random viewpoints and their
Table 1: Quantitative comparison on the Multiview-to-DD-Shapp Generation task.
PIx2Vox-A
EDM
3D-EDM
DIT
SiT
LDDBM
I-NNA IOU 1
.532 .013
.575
.009
.548
0044
.563 .007
.508 + .005
0.697
.611 4 .066
.602
.003
.613 4 001
.604 +: .003
.664 + .002

---

corresponding 3D shape during training. The goal of the model in inference time is to generate new 3D samples given multi-view samples that follow the distribution of 3D shapes.
Evaluation Setup. We evaluate our proposed approach using the ShapeNet dataset 64 following the protocol established in [55]. In this setup, each object is rendered into four 2D views captured from uniformly spaced azimuth angles around the object. These multi-view images form the input distribution p(y) for our multimodal generative framework, and the 3D occupancy will serve as the output distribution p(x). For evaluation, we adopt generative and reconstruction-base assess the quality of the models comprehensively, Note that our focus is on the generative performance of the models. For generative evaluation, we utilize the I-NNA as robust metric for assessing the similarity between the distributions of generated and real samples [74 66]. To evaluate reconstruction quality, we employ IoU. The 1-NNA assesses how well the model captures the generative distribution (diversity and realism), while IoU quantifies fidelity to specific target shape in reconstruction tasks. We compare our proposed method with several strong baselines capable of conducting general MT. Specifically, we include comparisons with the EDM and EDM-3D models
metries to
[27]. representing leading diffusion-bases -asodd approaches using 2D and 3D U-Net architectures [52]. Additionally, we evaluate diffusion models, including the Diffusion Transformer (DiT) 45 and the Scalable Interpolant Transformer (SiT) [35], both of which leverage transformer backbones to achieve strong performance on high-resolution generative tasks. For fair comparison, we implement the same encoders and decoders as ours. Finally, while direct comparisons to task-specific SOTA may not be fully fair, our framework is domain- -eeaiin-peeificccc inductive biases, we added a representative task speciapee SOTA, Vox2 65 to the Multiview-to-3] Shape Generation table (highlighted in gray) to illustrate trade-offs.
against recent Transformer-basee
whereas those methods embed strong
Results. In Tab. we present the quantitative results for the task of 3D shape generation from multi- view images, using both generative and reconstruction- based metrics. LDDBB achieves the lowest 1-NNA score (.508), significantly outperforming all baselines, indicating that our model generates 3D shapes more distributionally similar to the real data. Additionally, our model achieves the highest IoU score (.664), demonstrating superior fidelity in capturing fine-grained 3D geometry compared to all competing methods. Overall, LDDBM consistently outperforms strong baselines across both evaluation axes, confirming its strong generative capabilities and its ability to generate 3D content from multi-view images faithfully. Further, we show qualitative results in Fig. and in App. A.3 and comparison in A.3.8 showcasing the input multi-view images alongside its corresponding 3D shape.
5.2 Zero-Shot Low-to-High Resolution Generation
To further explore our model's ability to generalize across different modality generations, evaluate the zero-shot super-resolution capabilities of our model since images lie in different modality sizes. This task aims to generate high-resolution images from low-resolution inputs without task-specific fine-tuning. In this setup, the general multimodal generative model is trained on paired low-resolution and high-resolution images. During inference, the model receives only low-resolution image and must produce a realistic high-resolution output that conforms to the distribution of real images.
Evaluation Setup. We follow the experimental protocol of (38] The model trained on the Flickr- Faces-HQ (FFHQ) dataset (28) and evaluated on samples from the CelebA-HE dataset (26). The task is defined from a resolution of 16 x 16 to 128 x 128. To evaluate the perceptual and pixel-level quality of generated images, we follow the experimental protocol of (38), which uses the metrics: Peak Signal-to- -Nisee Ratio (PSNR), Structural Similarity Index (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS). These metries provide comprehensive evaluation: PSNR and SSIM capture low-level similarity, while LPIPS captures high-level perceptual realism, which is particularly relevant for generative tasks. We compare our method with similar baseline as the previous section and a task-specific state-of-the-art diffusion-basee baseline DiWa 38
Results. Fig. 4 (bottom) shows the quantitative results. Our model yields a PSNR of 25.6, surpass- ing other methods and indicating high reconstruction fidelity. Similarly, our model yields the highest SSIM of 0.68, demonstrating its superior ability to preserve structural information. Importantly, our model achieves the lowest LPIPS score of 0.32, highlighting its strength in generating perceptually realistic images, as judged by deep feature similarity. The consistent improvement across pixel-wise, structural, and perceptual metrics demonstrates the robustness and versatility of our approach in zero-shot generative tasks. Additionally, we provide qualitative results in Fig. [4] (top) and App. and comparisons in App. A.3.8, showcasing high- high-quality samples produced by our model.
8

---

EDM
DIWA
DIT
SIT
LDDBM
PSNR 1 SSIM
23.1 0.7
23.3
22.2 + 111
21.5 0.4
25.6 + 0.4
0.58 .05
0.65
0.52
+ .07
0.57 + .02
0.68 .03
IPIPS
0.41 0.39
0.49
.0
0.51 + .0
0.32
Figure 4: Top: Example low-to-high resolution outputs by LDDBM. Bottom: Zero-shot quantitative comparison measures. (Standard deviations could not be extracted for DiWa.)
5.3 Voice-to-Face and Face-to- Voice Translation
To further diversify our benchmarking and strengthen the robustness of our results, we introduce two additional multimodal translation tasks: Image-to-Audio and Audio-to-Image following the benchmark proposed by [40]. In these tasks, multimodal translation model receives one modality (either face or voice) as input and generates a meaningful representation in the other modality, which is subsequently evaluated through downstream classification accuracy. We compare
method against two baselines: (i) the SiT variant, which achieved the overall second- besd-bbetttt performance on other multimodal benchmarks, and (ii) the specialized state-of-the method from [40), specifically tailored for cross- moosll biometric matching between faces and voices. Implementation and evaluation details follow the same protocol in [40]. and full experimental details, including encoder and decoder architectures, are provided in the appendix.
Tab. presents the results, where higher values indicate better accuracy. LDDBM consistently outperforms the general-purpose purpose SiT model, demonstrating stronger multimodal translation capability, Although performance gap remains compared to the task-specific approach of 44),, this is expected given its domain specialization. Overall, the results highlight the generality of our framework, its superiority among modality-agnostie baselines, and its adaptability to diverse modalities.
Table 2: Cross-modal biometric translation accuracy face-voice matching tasks. Higher values indicate better performance.
Table 3: Edges + Bags (image-to-image). FID (lower is better) and inference time (seconds, lower is better). Measured with batch size 256.
Method
Face + Voice
Voice + Face f
Method / Metric
FID +
Time (s) 4
LDDBM
71.2
75.1
SiT
65.7
68.3
LDDBM DDBM
4.17
7.8
2.93
16.9
40
79.5
81.0
5.4 Image-to-Image Translation
While our benchmark was designed to target translation between dnfferenttdimensiinn modalities, it
also important assess tasks with aligned modality dimensions to further evaluate robustness and generalizability. To this end, we consider the edges + bags task from the DDBM benchmark [75]. which directly targets image-to-image translation. We evaluate two aspects: sample quality (FID J) and inference performance (wall-clock time in seconds J). Results are reported in Table Overall, we observe a clear trade-off: our method competitive in terms of quality while providing more than 2x faster inference than DDBM.

---

Table 4: Step-by-step ablation of our Transformer architecture. Each row introduces new component, illustrating cumulative improvements and the contribution of each design choice. The full results, with standard deviations, are reported in App. A.3
ShapeNet
nuScene
CelebA-HQ
Component
IoUt
I-NNA
IoU f
1-NNA
PSNR f
SSIM
LPIPS
(1) U-Net
.635
.588
.216
.818
23.2
0.57
0.42
(2) DIT
.613
.548
.208
.825
22.2
0.52
0.49
(3) Encoder-Decoder
.651
.518
.217
.821
23.4
0.53
0.38
(4) Spatial Embedding
.658
.522
.224
.812
22.9
0.56
0.41
(5) [MASK] (Ours)
.664
.508
.233
.807
25.6
0.68
0.32
Table 5: Loss ablation: Comparison of reconstruction and contrastive losses.
CREc
LIRED
Case LINFONCE
LIRED LINFONCE
SHAPENET
1-NNA
.625 .003
.22 + .002
.578 + .004
.508 + .005
IOU
.609 = .007
.643 .005
.627 + .007
.664 + .002
CELERA-HQ
PSNR 1
20.5 0.4
23.7 + 0.3
21.4 0.1
25.6 + 0.4
SSIM
0.49 + -03
0.64 + .02
0.51
0.68 + .03
LPIPS
0.62 + .04
0.41 + .02
0.63 + .03
0.32 + .01
5.5 Ablation Studies
Architectural ablation. We conduct a thorough ablation study of the architectural components in general-purpos Transformer. Additional ablations of different training procedures can be found in App. App.[A.3.4] As shown in T we begin with U-Net backbone (1), which leverages spatial inductive biases but lacks generality across modalities. Switching to DiT (2) removes these biases, revealing a performance gap. To improve conditioning, we adopt an encoder-decoder design (DiT is decoder only architecture) (3) inspired by sequence-to-sequenee translation models. We introduce constant spatial embeddings (4) to align tokens between input and target modalities. Finally, we incorporate a learnable [MASK] token (5) to enhance expressiveness and achieve
performance. We observe minor performance improvements in specific datasets certain metrics, where spatial embedding is added (4), suggesting that regular positional embedding (3) and spatial embedding (4) can be used interchangeably based on preference or prior biases.
Losses ablation.
We additionally assess the impact of various loss configurations. We compare:
1. Reconstruction loss: Standard reconstruction free LAL. + LAE, vs. consistent recon- struction through the multi-modal bridge Lped-
2. Contrastive loss: With or without additional contrastive estimation ListaNCE-
The results summarized in Table 5 indicate that the complete configuration combining both Cpee and LinfoNCE: achieves the highest performance, In addition, Fig. 10 qualitatively illustrates the differences between the Li + LiefNNE and Lpred + LIMONNEE models. It evident that the outputs of the Lme LintoNCE model suffer from facial artifacts, distorted backgrounds, and irregular distributions, while the Cped + LIfNNCE model produces images that are perceptually cleaner and more reliable.
Conclusion
In this work, we introduced general framework for modality translation based on latent-variable extensions of Denoising Diffusion Bridge Models. By operating in shared latent space and leveraging contrastive alignment and predictive objectives, our method enables flexible and accurate translation between arbitrary modalities without relying restrictive assumptions such as aligned dimensions or mopelity-specific architectures. Through comprehensive experiments across diverse tasks locks-includdng multi-view 3D shape generation, image super-resolution, and multi-view scene synthesis- we demonstrated strong performance and robust generalization, establishing new state- of-the-art for general modality translation. Looking forward, extending our framework to handle unpaired modality translation and scaling it to sequential or high-dimensional data such as video and volumetric representations offers promising directions for further expanding its applicability.
10

---

Acknowledgments
This research was partially supported by the Lynn and William Frankel Center of the Computer Science Department, Ben-Gurion University of the Negev, an ISF grant 668/21, an ISF equipment grant, and by the Israeli Council for Higher Education (CHE) via the Data Science Research Center, Ben-Gurion University of the Negev, Israel.

---

References
[1] M. S. Albergo, M. Goldstein, N. M. Boffi, R. Ranganath, and E. Vanden- Mijeen- Stochastic interpolants with data-dependent couplings. In Forty-first International Conference on Machine Learning, ICML., 2024.
[2] K. Armanious, C. Jiang, M. Fischer, T. Küstner, T. Hepp, K. Nikolaou, S. Gatidis, and B. Yang. Medgan: Medical image translation using gans. Computerized medical imaging and graphics, 79:101684, 2020.
[3] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli. wav2vee 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:12449 124600 2020.
[4] N. Berman, E. Kosman, D. D. Castro, and O. Azencot. Reviving life on the edge: Joint score-based graph generation of rich edge attributes. Trans. Mach. Learn. Res., 2024.
[5] N. Berman, I. Naiman, M. Eliasof, H. Zisling, and O. Azencot. One-step offline distillation of diffusion- based models via Koopman modeling. In Advances in Neural Information Processing Systems (NeurlPS) 39, 2025.
[6] F. Bhat, I. Alhashim, and P. Wonka. Adabins: Depth estimation using adaptive bins. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4009-4018, 2021.
[7] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621-11631, 2020.
[8] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[9] P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-888 8794008794422 2021.
[10] J. L. Doob and J. Doob. Classical potential theory and its probabilistic counterpart, volume 262. Springer, 1984,
[11] A. Dosovitskiy, I.. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlshy. An image is worth 16x words: Transformers for image recognition at scale, 2021.
[12] G. Fadlon, I. Arbiv, N. Berman, and O. Azencot. A diffusion model for regular time series generation from irregular data with completion and masking. In Advances in Neural Information Processing Systems (NeurlPS) 39, 2025.
[13] R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V. Alwala, A. Joulin, and I. Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15180-15190, 2023.
[14] T. Gonen, Pemper. I. Naiman, N. Berman, and O. Azencot. Time series generation under data scarcity: unified generative modeling approach. Advances in Neural Information Procexsing Systems (NeurIPS) 39, 2025.
[15] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[16] G. He, K. Zheng, J. Chen, F. Bao, and Zhu. Consistency diffuusion bridge models. In Advances in Neural Information Processing Systems 38, 2024.
[17] J. He, Q Yu, Q Liu, and L.-C. Chen. Flowtok: Flowing seamlessly across text and image tokens. arXiv preprint arxiv:2003. 10772, 2025.
[18] M. Heusel, H. Ramsauer. T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. In Advances Neural Information Processing Systems, 2017.
[19] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances neural information processing systems, 33:6840-6851, 2020.
[20] J. Ho and T. Salimans. Classifier-free difffusion guidance. arXiv preprint arXiv:2207.12592 2022.
[21] J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet. Video diffusion models, Advances in Neural Information Processing Systems, 35:8633 8646, 2022.
12

---

[22] A. Hyvärinen and P. Dayan. Estimation of non-normalized statistical models by score matching, Journal of Machine Learning Research, 6(4). 2005.
[23] P. Isola, J.-Y. Zhu, T. Zhou, and A. Efros. Image-to-ing to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125-11344 1114,114,,000 2017.
[24] H. Ji, T. Lin, and H. Li. Dphridge: Latent diffusion bridge for dense prediction. arXiv preprint 2225:2412.200066 2024.
[25] Y. Jiao, L. Kang, Lin, J. Liu, and H. Zuo. Latent schr (V o) dinger bridge diffusion model generative learning. arXiv preprint arXiv:2404. 13309, 2024.
[26] T. Karras, T. Aila, S. Laine, and J. Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv: 10196, 2017.
[27] T. Karras, M. Aittala, T. Aila, and S. Laine. Elucidating the design space of diffusion-basch generative models, Advances in neural information processing systems, 35:26565-266 26577, 2022.
[28] T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401-4410, 2019.
[29] Y. Kim, K. Lee. M. Park, B. Na, and L-C. Moon. Diffusion bridge autoencoders for unsupervised representation learning. arXiv preprint arX11 2024.
[30] Z. Kong, W. Ping, J. Huang, K. Zhao, and B. Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In 9th International Conference on Learning Representations, ICLR, 2021.
[31] B. Liu. Comparative analysis encoder-only, decoder-only, and encoder- decoder language models. In Proceedings of the International Conference on Data Science and Engineering Volume ICDSE, pages 524-530. INSTICC, SciTePress, 2024.
[32] G. Liu, A. Vahdat, D. Huang, E. A. Thcodorou, W. Nic, and A. Anandkumar. P2s: Image-to-imase schrödinger bridge. In International Conference on Machine Learning, ICML, volume 202 of Proceedings of Machine Learning Research, pages 22042 22062. PMLR, 2023.
[33] Q. Liu, X. Yin, Yuille, A. Brown, and M. Singh. Flowing from words to pixels: A framework for cross-modality evolution. arXiv preprint 2tt::24//21221333 2024.
[34] A. P. M., S. R. M. and O. Christopher. Machine translation with large language models: Decoder only vs. encoder-decoder, 2024.
[35] N. Ma, M. Goldstein, M. Albergo, N. M. Boffi, Vanden Vanden-Eijnden, and Xie. Sit: Exploring flow and diffusion based generative models with scalable interpolant transformers. In European Conference on Computer Vision, pages 23 40. Springer, 2024.
[36] M. Mancusi, Y. Halychanskyi, K. W. Cheuk, F. Moliner, C.-H. Lai, S. Uhlich, J. Koo, M. A. Martínez- Ramírez, W.-H. Liao, G. Fabbro, et al. Latent diffusion bridges for unsupervised musical audio timbre transfer. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2025.
[37] X. Mao, Q. Li, and H. Xie. Aligngan: Learning to align cross noss-domain images with conditional generative adversarial networks. arXiv preprint 170.:::177..14440 2017.
[38] B. Moser, S. Frolov, Raue, S. Palacio, and A. Dengel. Waving goodbye to low-res: A diffusion- diffusiionwwvelett approach for image super-resolution. In 2024 International Joint Conference on Neural Networks (JJNN), pages 1-8. IEEF, 2024.
[39] Z. Muhammad Zayyanu Ph.D. Revolutionising translation technology: A comparative study of variant transformer models bert, gpt, and 15. Computer Science and Engineering An International Journal, 14:15-27, 2024.
[40] A. Nagrani, S. Albanie, and A. Zisserman. Seeing voices and hearing faces: Cross-mode biometric matching. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1427-8436. 2018
13

---

[41] I. Naiman, N. Berman, and O. Azencot. Sample and predict your latent: modality-free sequential disentanglement via contrastive estimation. In International Conference Machine Leurning, pages 25694-25717. PMLR, 2023.
[42] Naiman, Berman, Pemper, I. Arbiv, G. Fadlon, and O. Azencot. Utilizing image transforms and diffusion models for generative modeling of short and long time series. Advances in Neural Information Processing Systems, 11221199-127777 2024.
[43] A. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 807. 03748, 2018.
[44] X. Ouyang and G. Agam. Accelerated wgan update strategy with loss change rate balancing. arXiv preprint arXiv:2008. 12463, 2020.
[45] W. Peebles and S. Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195-4205, 2023.
[46] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In The Eleventh International Conference on Learning Representations, ICLR, 2023.
[47] M. Qorib, G. Moon, and H. Ng. Are decoder-only language models hetter than encoder-only language models in understanding word meaning? In L.-W. Ku, A. Martins, and Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 16339-16347, Bangkok, Thailand, Aug. 2024. Association for Computational Linguistics.
[48] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PmLR, 2021.
[49] A. Ramesh, M. Pavlov, G. Goh, Gray, C. Voss, A. Radford, M. Chen, and Sutskever. Zero-shot text-to-image generation. In Proceedings of the 38th International Conference on Machine Learning, ICML, volume 139 of Proceedings of Machine Learning Research, pages 8821-8831. PMLR, 2021.
[50] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models, 2021.
[51] R. Rombach, Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[52] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer, 2015.
[53] S. Schneider, A. Baevski, R. Collobert, and M. Auli, wav2vec: Unsupervised pre-training for speech recog- nition. In 20th Anmual Conference of the International Speech Communication Association, Interspeech, pages 3465-34699 ISCA, 2019.
[54] Y. Shi, V. De Bortoli, A. Campbell, and A. Doucet. Diffusion schrödinger bridge matching. Advances in Neural Information Processing Systems, 36:62183 62223, 2023.
[55] Z. Shi, Z. Meng. Xing, Ma. and R. Wattenhofer. 3d-retr: End-to-end single and multi-view 3d reconstruction with transformers. arXiv preprint arXiv:2110.08800 2021.
[56] U. Singer, A. Polyak, T. Hayes, X. Yin, J. An, S. Zhang, Q. Hu, H. Yang. O. Ashual, O. Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.144. 14792, 2022.
[57] J. Sohl-Dickstein, F. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265. PMLR, 2015.
[58] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. In 9th International Conference on Learning Representations, JCLR, 2021.
[59] A. Stromme. Sampling from a schrödinger bridge. In International Conference on Artificial Intelligence and Statistics, pages 0058-4067 PMLR. 2023.
14

---

[60] L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
[61] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, I.. Jones, A. N. Gomez, F.. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[62] X. Wang, Z. Zhu, W. Xu, Y. Zhang. Y. Wei, X. Chi, Y. Ye, D. Du, Lu, and X. Wang- Openoccupancy: A large scale benchmark for surrounding semantic occupancy perception. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17850- 17859, 2023.
[63] Y. Wang, Jin, L. Zhang, Q. Li, Z. Chen, and D. Wu. An ordinary differential equation sampler with stochastic start diffrusion bridge models. arXiv preprint arXiv:2412.1999,, 2024.
[64] Z. Wu, Song, A. Khosla, F. Yu, L.. Zhang, X. Tang, and J. Xiao. 3d shapenets: deep representation for volumetric shapes. Proceedings of the IEEE conference on computer vision and pattern recognition. pages 1912-1920, 2015.
[65] Н. Xie, Yao, Sun, Zhou, and Zhang, Pix2vox: Context-aware reconstruction from single and multi-view images. In Proceedings of the IEEE/CVE international conference on computer vision, pages 2019.
[66] G. Yang. X. Huang, Z. Hao, M.-Y. Liu, S. Belongie, and B. Hariharan. Pointfoww 3d point cloud generation with continuous normalizing flows. In Proceedings of the IEEE/CVF international conference computer vision, pages 1541-4550, 2019.
[67] L. Yang, Z. Zhang, Y. Song, S. Hong, R. Xu, Y. Zhao, W. Zhang, B. Cui, and M.-H. Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 56(4):1-39. 2023.
[68] Q You, Jin. Z. Wang, C. Fang, and J. Luo. Image captioning with semantic attention. In Proceedings of IEEE conference on computer vision and pattern recognition, pages 4651-465555 1455,, 2016.
[69] C. Zhang, C. Zhang. S. Zheng. M. Zhang, M. Qamar, S.-H. Bae, and S. Kweon. A survey audio diffu- sion models: Text to speech synthesis and enhancement in generative ai. arXiv preprint arXiv:2303. 13336 2023.
[70] H. Zhang. An attempt to generate new bridge types from latent space of denoising diffusion implicit model. arXiv preprint arXiv:2402.07122 2024.
[71] R. Zhang, Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and patter recognition, pages 586-595 595, 2018.
[72] L. Zhen, P. Hu, X. Wang, and D. Peng. Deep supervised cross-modal retrieval. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pages 10394-10403, 2019.
[73] K. Zheng, G. He, J. Chen, F. Bao, and J. Zhu. Diffusion bridge implicit models. arXiv:2405. 15885, 2024,
arXiv preprint
[74] L. Zhou, Y. Du, and J. Wu. 3d shape generation and completion through point-voxel difftusion. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3826-5835, 2021.
[75] L. Zhou, A. Lou. S. Khanna, and S. Ermon. Denoising diffuusion bridge models. In The Twelfth International Conference on Learning Representations, ICLR, 2024.
[76] B. Zhu, B. Lin, M. Ning, Y. Yan, J. Cui, H. Wang. Y. Pang, W. Jiang. J. Zhang, Z. Li, C. Zhang, Z. Li, W. Liu, and L. Yuan. Languagebind: Extending based semantic alignment. In The Twelfth International Conference on Learning Representations, ICLR, 2024.
pretraining to n- n-modality by language-
[77] J.-Y. Zhu, T. Park, P. Isola, and A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223-2232, 2017.
15

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We support the claim with empirical results in the last section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the computational tradeoff and potential future improvement.
16

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We attach code example for the review process and publish the code upon accaptence.
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Code example is attached and code github will be released upon acceptance. Datasets are public.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.cc/public/eee.c www.deddeee..ouuummmceceed ssi ssionyolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
18

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:
[Yes]
Justification: Appendix contains all details needed Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: all results reported with standard deviation on repeated experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: complexity analysis is provided along with training details on the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
19

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httggs//berriis.cc/bbbbbb/Bbiii
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
20

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets are public
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
21

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22

---

A Appendix supplemental material
A.1
Additional Experimental Details
A.1.1 Datasets
Multi-view to 3D generation task ShapeNet 64 is a comprehensive dataset comprising tens of thousands of unique 3D models across 55 common object categories. In alignment with the protocol established in [55], we utilize a subset of this dataset, focusing on 13 specific categories, encompassing around 43,783 models. Each 3D model undergoes preprocessing with Binvox at resolution of 32 x 32 x 32. Subsequently, 4 images are rendered per model from random viewpoints, each with resolution of 224 x 224 pixels. We split the dataset by randomly assigning 70% of the objects to the training set, 10% to the validation set, and the remaining 20% to the test set, ensuring balanced distribution across all categories.
Zero-Shot Low-to-High Resolution Generation task FFHQ and CelebA-HQ: The FFHQ dataset comprises 70,000 high-quality images of human faces at resolution of 1024x1024 pixels. for our task, the images where resized to 16 x 16, 128 x 128 pair with Bilinear interpolation. These images exhibit considerable variation in age, ethnicity, and background, and include a diverse range of accessories such as eyeglasses, sunglasses, and hats. This dataset was used to train our model.
The CelebA-HQ dataset is a high-quality version of the original CelebA dataset, consisting of 30,000 images at resolution of 1024x1024 pixels, for our task, the images where resized to 16 16, 128 128 pair with Bilinear interpolation.. It created by selecting and processing images from CelebA to enhance their quality and resolution. This dataset was used to evaluate the model
hwwtview-to-scee--eeeupppp Generation nuScenes--Occupancy The dataset serves as 3D occupancy prediction benchmark derived from the nuScenes autonomous driving dataset. contains approximately 25,000 pairs of multi-view camera images and corresponding 3D occupancy representations of the ego vehicle's surroundings. The occupancy information is inferred from LiDAR data and discretized into a 512 x 512 x 32 voxel grid. Each voxel grid is paired with a set of nigh-resolution multi-view images captured from cameras mounted on the vehicle, covering full 360 field of view. To reduce computational demands, we downsample the original high-resolution images 256 256, as processing the full-resolution inputs would require minimum of 80 GB of memory per sample, which is impractical given resources limitations.
A.1.2 Metrics
Chamfer Distance is used to calculate distance metric for the 1-NNA metric. Therefore, we do not directly apply this metric. This distance is used to compare the similarity between two point clouds, typically between a generated 3D shape and reference. It is defined as:
CD(P, Q) = P
1
Σ PEP
min lp - TEO
Σ qEQ
min l
-p12
O
PEP
where P and Q are two point clouds. Lower Chamfer Distance indicates higher geometric similarity.
1-Nearest Neighbor Accuracy (1-NNA) is robust metric for assessing the similarity between the distributions of generated and real samples [66] 74). The I-NNA metric is grounded a two-sample statistical test and has been proven provide more reliable measure of distributional similarity than commonly used heuristics which may be sensitive to mode collapse or fail to capture full-shape distributions.
Formally, let Steal and Sgen denote the sets of real and generated samples, respectively. A I-NN classifier is trained over the combined dataset S = Speal U Syen with binary labels indicating whether each sample is real or generated. For each sample x E S, its nearest neighbor NN(x) is found among the rest of the set S {2), and leave-one-out classification is performed. The 1-NNA score is defined as:
1-NNA-
Σ xES
[label label(NN(
S
23

---

where [[[:] is the indicator function. Distances are computed using Chamfer Distance to reflect geometric dissimilarity between point clouds.
An ideal generative model produces samples indistinguishable from real data, which would result in I-NNN score of 50%. Scores significantly above 50% indicate that the classifier can distinguish between real and generated samples, thus revealing discrepancies between the distributions. Therefore, lower better for this metric, and values closer to 50% reflect stronger generative performance.
Since our data voxelized, we first convert each object into a point cloud representation in order to apply the metric. We then the standard evaluation protocol interface compute the final results.
Intersection over Union (IoU) aims to evaluate reconstruction quality, and its a standard metric in 3D shape prediction. For voxelized 3D shapes, IoU is computed between the predicted occupancy grid 1/r and the ground truth grid V as:
!VA IoU = IV VVI
A higher IoU indicates better overlap with the true shape, reflecting accurate reconstructions. This metric emphasizes pixel-wise consistency and most relevant when the goal is to reproduce specific ground-truth instances.
Peak Signal-to-Noise Ratio (PSNR): This metric measures the pixel-wise similarity between the generated image å and the ground truth image x, computed as:
MAX2
PSNR(=, 2) = 10 1og
N -x;
where M AX is the maximum pixel value (e.g., 255) and is the number of pixels. Higher PSNR indicates lower mean squared error and better reconstruction fidelity.
Structural Similarity Index (SSIM): SSIM evaluates the structural similarity between two images, capturing luminance, contrast, and texture similarity. is defined as:
(2p= + C2) + +22 SSIM(I, f) ( F CI)(o? + +o C2)
where Jr, are means and variances, and C₁ C2 are constants for numerical stability. Higher SSIM values (closer to 1) indicate better structural fidelity.
Learned Perceptual Image Patch Similarity (LPIPS): LPIPS 71 measures perceptual similarity using deep neural network features. compares deep activations between x and
LPIPS .)) = Σ H,W,
I o (ff
ff1l
h
where fi are the features from layer wi is learned weight vector, and H1, Wi are spatial dimensions. Lower LPIPS implies higher perceptual similarity.
24

---

A.2 Implementation Details
A.2.1 Modality Encoder and Decoder Architectures
The encoders and decoders are similar between different methods, providing a more fair comparison setup between different general multi-modal generative frameworks. Therein, describe the specific of each encoder or decoder that we use in the different tasks.
Multiview-to-3D--e -Sh-hSppe Generation this setup, except for the 3D-EDM method, all frameworks use the same encoder decoder scheme.
1. 3D Convolutional Encoder-
To encode 3D object inputs, we adopt a 3D convolutional encoder. The encoder takes as input tensor of shape (B, C, D. H. W), where B is the batch size, C the number of channels, and И W D the 3D-voxel resolution of the object. The network comprises a sequence of three 3D convolutional layers with kernel size 4, stride 2, and padding progressively reducing the spatial dimensions. Specifically, the first layer maps the input to 64 channels, followed by second and third layer expanding the channels 128 and 256, respectively. Each convolution followed by a SiLU (Sigmoid Linear Unit) activation function and optional dropout for regularization. The final output is flattened or reshaped into a latent tensor, depends on the model requirements. This latent representation serves as the encoded feature map for downstream generative modeling tasks.
2. 3D Convolutional Decoder This module mirrors the architecture of the 3D encoder, but uses transposed convolutional layers to progressively upsample the latent representation back to the original volumetric shape.
3. Multi-View 2D Convolutional Encoder To process multi view image inputs, we utilize a 2D convolutional encoder that independently encodes each view using a shared backbone. The encoder receives input tensors of shape (B, V, C, H, W), where is the batch size, V is the number of views, C' the number of image channels (e.g., RGB), and H x W the resolution of each view. Each view is flattened across the batch and view dimensions and passed through a shared 2D convolutional stack consisting of six layers. Each layer applies a convolution with kernel size 4, stride 2, and appropriate padding, followed by a SiLU (Sigmoid Linear Unit) activation and optional dropout for regularization. The convolutional stack progressively increases the channel dimension from 32 to 256 while reducing the spatial resolution, effectively extracting high-level features from each view.
Following the convolutional encoding, the features from all views are concatenated and passed through fully connected projection layer to form unified latent representation. Specifically, the concatenated features are mapped to a latent space of configured size using a linear layer.
4. Identity Encoder/Decoder for 3D Data This is a special case used in the 3D-EDM model, where the diffusion process operates directly in the input space x. As a result, no encoding decoding modules are required, and the model functions similarly to standard diffusion models without latent-space adaptation.
Zero-Shot Low-to-High Resolution Generation In this setup, for all methods, including ours, except for 33). whose results were taken from the benchmark, we used the same encoder-decoder architecture. Specifically, we employed the pre-trained autoencoder from (50) with the configuration = 32, KL regularization, and latent dimension d 64. All models were initialized with the pre-trained weights before training.
To handle varying image resolutions (as the autoencoder was originally trained on 256 x 256 images), we prepend each model with a stack of 2D convolutional upsampling layers to resize inputs to 256 x 256, and append symmetric stack of downsampling layers to revert to the original resolution. The latent representations used for generative modeling of distribution translation.
Multiview-to- Ser-rieew-t-SSeee---e -cccenncy Generation In this setup, we adopt a 3D convolutional encoder- decoder architecture for encoding and decoding the occupancy scene, following similar design to that used in multi-view to 3D shape generation. For encoding the multi-view images, we utilize the encoder from the pre-trained autoencoder proposed in [50].
25

---

A.2.2 Our Transformer Architecture
Fig. shows the structure and order of operations performed by the Distribution Translation DiT (DTDIT). The DTDiT Encoder block (Fig. 5b) consists of self-attention and feed-forware opera- tions, each preceded by layernorm, exactly like the transformer-encodee shown 61). In the DTDIT Decoder block (Fig- (Fig.Ba). we extend the original DiT Block [45) by adding time-embedding modulated cross-attention operation, in between the original self-attention and feed-forwar opera- tions. Consequently, the MLP learns 9 vectors (a,, B, 74 (a4s B Y
Vi {1, 3}) instead of the original 6
vi E 2})
Cl
FeedForward Network
B,
Scale, Shift
LayorNorm
FeedForward Network
Cl2
Scale
nead Cross Attention
B₂
LayerNorm
Scale, Shift
LayerNorm
Memory
Olg
Scale
Multi-head Self-Attention
Multi-head Celf-Attention
Y
Scale,
LayerNorm
Input Tokens
Time Embedding
DTDiT Encoder Block
Structure of DTDIT Decoder block
(b) Structure of DTDIT Encoder block
Figure 5: Detailed network architecture of the Distribution Translation DiT (DTDiT)
A.2.3 Baselines Implementation
Forall baselines, we employed similar encoding and decoding scheme to ensure a fair comparison for general modality translation (MT) evaluation. To enable general MT across all baselines [35] 45 27 we used the encoding of y, denoted ær, as the conditional input to the diffusion model, and the encoding of denoted 2g. the target signal to be generated. The diffusion process was then trained to generate 2o conditioned on žT using the standard diffusion loss, along with an additional reconstruction loss to map the predicted latent back to its original domain. We evaluated both predictive and reconstruction. based losses for each method, reporting the better-performing one.
During inference, we used the original sampler of each baseline, but standardized the number of sampling steps to 40 for consistency. For each method, we retained the original architectural hyperparameters and only adjusted the latent space dimensionality to match our setup except for EDM-3D (on the ShapeNet dataset), which operates directly in the same dimension as All methods used the same latent dimensionality our framework.
26

---

A.2.4 Hyperparameters
We adopted the Variance Exploding (VE) configuration for sampling, following the hyperparameter settings provided in the official codebase of [76]. The hyperparameters we tuned are summarized in 6 Our search focused on variations in batch size, learning rate, and latent space dimensionality. All loss terms were used without weighting adjustments.
Training was performed on a single NVIDIA A100 GPU for 1,000,000 iterations, which corresponds approximately 3-5 days of runtime depending on the dataset. We used the RAdam optimizer.
Table 6: Summary of Hyperparameters
MILIT-VEEW-T-3---HHAE TEBO-SROKAAUPRAAABSAAUUTTO
MULTT-VIEW-TO-SCEN
BATCH SIZE
256
32
LEARNING RATE
0.0001
0.0003
0.0001
LATENT
16384
24576
4096
SAMPLING STEPS
40
40
40
RHO
7
SAMPLER
HEUN
HEUN
HEUN
CHURN
0
GUIDANCE
0.5
0.5
0.5
CHANNELS
EMBED
256
256
NUM HEADS
8
DEPTH (Np, Ng)
6
27

---

A.3
Additional Experiments and Details
A.3.1
Multrieew-oo--eene-Dccuuaaayy Generation
Table 7: I-NNA and IoU metrics for multi-view to 3D occupancy on the nuScene dataset.
EDM
3D-EDM
DIT
SIT
LDDBM
1-NNA +
.22 + .009
832 + .012
825 + .007
.854 + .005
.807 + .008
IOU 1
213 - 007
.00 + .013
208 + 0011
.198 + .007
.233 + .005
We conclude our evaluation by considering a more complex task: generating 3D occupancy scenes from multi-view images using the nuScenes-Occupancy framework [2] for occupancy prediction. This subset of nuScene dataset offers multi-camera views along with voxelized 3D occupancy map. For computational feasibility, downsample both input images and occupancy volumes. While this experiment not designed to push state-of-the the-art boundaries, it showcases the flexibility and applicabiliit of our framework in more realistic autonomous driving scenarios. We adopt the metric introduced in the previous Sec. [_1] For all these methods, and for a fair comparison, we implement the same encoders and decoders ours for embedding x; We report further details in App. A.2
Ap2
Results. We report the results in Table 7 Our method achieves the best performance across both evaluation metrics for the task. Specifically, it obtains the lowest 1 1-Nearest Neighbor Accuracy (1- NNA) of 0.807, indicating that the distribution of generated 3D shapes closest to that the real data in comparison to other methods. This reflects the generative quality in terms of realism and diversity, Nevertheless, these results indicate that all models still have a significant gap toward producing realistic generations. In addition, model achieves the highest Intersection over Union (IoU) score of 0.233, significantly outperforming all baselines. This demonstrates superior reconstruction accuracy in capturing the underlying 3D structure. Taken together, these results highlight again the robustness and effectiveness of our approach.
A.3.2 Generalization to Natural Images: DIV2K Super-Resolution
To assess the generalization ability of our framework beyond face-centric data, we incorporated the DIV2K dataset, large and diverse collection of high-quality RGB natural images, into the Resolutino (SR) benchmark. We trained and evaluated our model following the same setup as
Results. Qualitatively, as showen in Fig. 9 our model produces sharp, artifact-free images across a wide variety of natural scenes, demonstrating strong generalization beyond facial domains. Quantita- tively, our approach achieves a PSNR of 28.9, outperforming the DiWa baseline (38], which attains a PSNR of 28.2 under identical settings. These findings reinforce the flexibility of latent bridging framework and its ability to generalize to domains with diverse textures and structures.
A.3.3 Complexity Analysis
Table presents comparison of model complexity and inference efficiency for different approaches evaluated on the ShapeNet dataset. We report the number trainable parameters (in total) as well the sampling time (in seconds) for single batch of size 256. Our model demonstrates clear advantage in terms both model size and runtime efficiency. Specifically, requires only 288M parameters, significantly fewer than both EDM (498M) and DiT (685M), representing reduction of over 40% compared to EDM. This leaner architecture implies lower memory footprint and easier deployment. Moreover, model achieves sampling time of 12.46 seconds, which is times faster than the DaT-b model (79.36 seconds) while maintaining strong generative quality. It is also comparable to the convolutional EDM model in speed, yet substantially more compact and versatile. These results underscore the scalability and efficiency of our framework for high-resolution 3D generation tasks.
than 7
Sampling complexity analysis. We evaluated sampling quality on SHAPENET while varying the number of diffusion steps from 40 down to 10 (Table 9). Increasing the step count beyond 40 did not
28

---

Figure 6: Original High Resolution
Figure 7: Low Resolution
Figure 8: Our Super Resolution
Figure 9: DIV2K super-resolution examples.
Table Complexity analysis on the ShapeNet dataset. Parameters are in millions. Sampling time is measured in seconds per batch of size 256.
EDM
DIT
LDDBM
PARAMETRES SAMPLING TIME J
498.6 6
685.5 79.36
288.7
11.17
12.46
29

---

yield measurable gains. Notably, reducing from 40 to 30 steps slightly improves quality, whereas using fewer than 30 steps leads an apparent degradation. as expected.
Model / steps
40
30
20
10
Our 1-NNA +
508 + .005
504 + .066
522 - .008
631 + .018
loU 1
664 = .002
663 + 004
642 = .004
533 + .014
Table 9: Sampling-step ablation on SHAPENET. Performance as a function of the number of
sampling steps. Values are mean + std.
Evaluation of our model with different loss functions. The first row shows the low- resolution input, the second row displays the output using the regular reconstruction loss (Lrec) LinfONCE, and the last row presents the output using the end-to-end loss (Spred LintoNCE..
Figure 10:
A.3.4 Training Strategies Ablation
We compare three training paradigms for multi-modal generation: Two-Step training, End-to-Fine End training of the whole system, and our proposed Iterative procedure inspired by practices in adversarial training 18 (4).. As shown in Table [0] the Iterative strategy consistently yields the best results across datasets and metrics.
Runtime setup and observations. Unless stated otherwise, all runs use identical hardware and hyprraarameters single NVIDIA A100 GPU, For both End-to-End and Iterative strategies,
30

---

Table 10: Comparison of training strategies: Two-Step, Snd-to-End, and Iterative.
TWO-STEP
END-TO-END
ITERATIVE
SHAPENET
I-NNA +
.522 #.006
.517 #.003
.508 + .005
IOU 1
.637 #.033
.642 :.005
.664 = .002
CELERA-HQ
PSNR
23.3 0.6
23.4 +0033
25.6 0.4
SSIM f
0.58 000
0.57 2.05
0.68 + .03
LPIPS 4
0.40 .00
0.39 #.01
0.32 + .01
we train for total of 1,000,000 iterations. The Two-Step strategy comprises autoencoder pre- pre-training phase followed by 1,000,000 iterations for the diffusion bridge. In practice, the Iterative strategy is not slower; it is slightly faster in wall-clock time per epoch than the End-to- End because each alternating phase updates fewer parameters and therefore requires fewer gradient computations per iteration. Overall runtime for Two-Step is comparable to Iterative.
A.3.5 Degregraion of sampling steps
we conducted experiments on the ShapeNet dataset using different sampling step counts, starting from 40 and reducing down 10. We include the results the table below. We also experimented with more than 40 steps, but this didn't t lead to noticeable improvements. Interestingly. reducing the number of steps 30 appears be beneficial; however, below 30 steps, the sample quality begins to degrade significantly, as expected.
Model / steps 40 30 20 10 Our I-NNA IoU Table: Results across differrntt step counts. Values mean std.
A.3.6 Encoder Quality Ablation: Impact Latent Bridging
The bridge model assumes that source and target modalities share sufficiently expressive and consistent latent space. To probe this assumption, we ablate the emcoder/decoder quality within the Resor-eRsoot benchmark
16x 128 128). We compare three variants: (1) a pre-trained autoencoder (AE) from Latent Diffusion Models [5T], (2) the same AE architecture trained from scratch (no pre- teeinrannin))... and (3) lightweight 5-layer convolutional encoder/decooerr ("Simple CNN"), All other components and training budgets are held fixed.
Findings. Table 11 shows that (1) and (2) yield comparable PSNR, indicating limited benefit from pre-training in this setup. In contrast, (3) substantially underperforms, suggesting that architectural capacity and representation quality critical for robust latent bridging under losses (bridge, predictive, contrastive).
Method Metric
PSNR
(1) Pre-trained AE
25.6 = 0.4
(2) Vanilla AE
25.3 + 0.5
Simple CNN
22.9 + 0.9
Table 11: Encoder quality ablation on Super- (FFHQ-xCelebAA Values are mean + std over three runs.
A.3.7 Standard Deviation
For all quantitative experiments, we run the evaluation protocol three times and report the mean and standard deviation of the results. The full results are provided below.
A.3.8
Qualitative Comparison on SR Task
We present a qualitative comparison of various methods for the super-resolution (SR) task in Fig. The figure shows set of low-resolution input images and their corresponding high-resolution outputs
31

---

Table 12: Complete Results for Tab
SeapeNet
muSeene
CelebA-HQ SSIM
Component
IoU
I-NNA
1oU 1
1-NNA
PSNR
LPIPS 4
(1) U-Net
.33 .002
.18 1 .008
.216 1.002
.818
.007
23.2
1.3 0.3
0.57 1.
0.42 1: .04
(2) DTT
613 1006
.388 1 .012
.218 1 004
.825
005
22.2 022
0.52
0.49 113
(3) + Encoder- Fncoder-Decoder
.651 .002
.518
1: .044
.217 + 002
.821 4 .006
23.4
0.4
0.53 4: 00
0.38 + .01
Spatial Embedddin
22.9
(5) Learaalle Token (Ours)
.61 + .022
.508 + 000
233 + 005
.807 + .008
25.6 + 0.4
0.68 + 00
0.32 = .01
generated by four different models: SiT, DīT, EDM, and our proposed method. Each row corresponds to different model, with the first row displaying the low-resolution inputs and the subsequent rows showing the results generated by the different methods. We can see that the images generated by our method appear more realistic, with fewer artifacts and distortions.
A.3.9 Qualitative Comparison on multi-view to 3D shape task
We present a qualitative comparison various methods for the 3D shape generation task from multi- view images Fig. 12 The figure shows set of two multi view input images and their corresponding 3D shape outputs generated by four different models: SiT, DìT, EDM, and our proposed method. Each row corresponds subsequent rows showing the results generated by the different methods. We can see that the images generated by our method appear more realistic, with fewer artifacts and distortions.
different model, with the first displaying the multi-view inputs and the
A.3.10 Additional Qualitative Results
We provide additional qualitative results for the super-resolution task in Fig. 13 and for the multi- view-to-3D shape generation, including the ground truth occupancy 3D shape, in Fig. [4] It is important to note that while the generated shapes may not retain the exact same structure, they should exhibit similar features.
32

---

Figure 11: comparison of all methods on the super-resolution task. The first row displays the low-resolution input images, and the following rows show the high-resolution outputs generated by SiT (second row), DiT (third row), EDM (fourth row), and LDDBM (fifth row).
33

---

Figure 12: A comparison of all methods on the multi-view to 3D shape task. The first row displays the multi-view input images, and the following rows show the 3D shape outputs generated by SiT (second row), DiT (third row), EDM (fourth row), and LDDBM (fifth row).
34

---

Figure 13: Additional Examples for super resolution task of our model.
35

---

Figure 14: Additional Examples for multi-view to 3D task of our model. Left: the multi-views.
Middle: the predicted 3D shape. Right: The ground truth 3D shape.
36


Reviews:
[{"review_id": "0uhyAsPA0y", "reviewer": "Reviewer_Egf9", "summary": "This paper introduces a novel approach to multi-modal domain translation, specifically addressing a key limitation of existing bridge models. While current methods, such as those relying on optimal transport for tasks like image-to-image or image-to-3D translation, are often restricted by the defined marginal spaces of their start and end distributions, this work offers a new solution to model the distribution translation using latent bridge by estimating a conditional distribution conditioned on both the modalities by factorizing the conditional distribution and introducing a shared latent space. \nIn addition they define a modality agnostic transformer based encoder -decoder training framework eliminating the need to additional conditioning and any implicit biases added due to the architecture.", "questions": "1. The paper demonstrates strong capabilities in multi-modal translation. To further showcase the versatility and robustness of the proposed modality agnostic transformer, suggest exploring its application to outpainting and/or image editing tasks in image-to-image translation? This might provide some insights on spatial relationships and compositional representations? \n\n2. Sampling complexity of DDBMs are generally higher to achieve accurate interpolations can the authors elaborate on how the complexity is addressed ? \n\n3.  While it is impressive that the approach does not require any conditioning, It would be good to see how does the objective affect the training complexity.  Does the process of learning the 'bridge' between two distributions, and the modality-agnostic denoiser in a shared latent space, introduce a higher computational cost (e.g., longer training times, increased GPU memory usage) per iteration or per epoch compared to standard multi-modal EDMs. In additional how does the training convergence compared to standard diffusion pipelines.", "limitations": "The approach is interesting and very relavent, However I have requested for clarifications on complexity and ability to model spatial relationships.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is well motivated and the structured and directly addresses the limitations of bridge models and modality specific encoder decoder architectures. \n\n\n2. The authors introduce a sampling approach for training diffusion-based bridge models, specifically by learning a joint latent space shared across multiple modalities. This method offers a latent space that is trained to be more semantic. Current approaches to learn a shared latent space in multi-model models involve learning a bottleneck layer or mixture of expert architectures which require large number of pairs samples to learn from and the validation and practical deployment of these specific diffusion based approaches have been challenging. \n\n3. This work proposes a general purpose transformer based encoder-decoder architecture that is trained for 3 tasks super-resolution, 3D shape generation and occupancy tasks. The idea of universal auto-encoder decoder architecture is novel and to the best of my knowledge not been explored. \n\n4.  This work qualitatively illustrates the limitations of baseline objective demonstrating good intuitions on semantic alignment . In addition to comparing the proposed architecture Distribution Translation DiT against UNET and naive DiT architecture\n\nWeakness: \n\n1.  Although the results look very promising, the paper does not sufficiently address or analyze the inference time complexity of their bridge model approach compared to other multi-modal shared latent space methodologies that define a separate encoder decoder models.   Secondly Sampling complexity of DDBMs are generally higher to achieve accurate interpolations, would be curious to see how is the sample quality affected \n\n2.  DDPMs inherently support compositional representations especially would be beneficial when performing multi-modal generations. The paper did not address ability to preserve and transfer semantic relationships, compositional representations between modalities and retain usefull properties for image tasks.\n\n3. In general the training framework involves iteratively learning the auto-encoder/latent spaces jointly with the bridge essentially learning the marginal distributions making it a harder optimization problem.", "was_revised": true, "final_justification": "I have looked at the authors' response to questions and I retain my score. \n1. Training stability:  More empirical ablations are required to understand the training stability over different encoder-decoder architectures as mentioned by the authors\n2. The impact of inference vs latency tradeoff is clear. \n\nI will retain the score so far, overall direction is promising and the paper is well motivated. \nI have reduced my confidence to 3, I am not an expert/unfamilar in the litterature work the more I read the authors rebuttals.", "rebuttal": "We thank the reviewer for the constructive feedback and for recognizing the importance of our motivation, the novelty of our approach, and the extensiveness of our evaluation. Below, we address the concerns and questions raised. We'd be happy to provide further clarification on any additional points, if needed.\n\n> Although the results look very promising, the paper does not sufficiently address or analyze the inference time complexity of their bridge model approach compared to other multi-modal shared latent space methodologies that define a separate encoder decoder models. Secondly Sampling complexity of DDBMs are generally higher to achieve accurate interpolations, would be curious to see how is the sample quality affected\n\nThank you for raising these two points, which help clarify an important aspect of our work.\n\nRegarding the first comment, we've added a **Complexity Analysis** in Section A.3.2 of the appendix, where we compare and contrast the inference-time complexity of our approach with other methods. We also include a memory analysis. The results demonstrate that our model offers a clear advantage in terms of both model size and runtime efficiency. To make this section more accessible to readers, we've added a reference to it in the main text. We appreciate you pointing this out.\n\nTo address the second point, we conducted experiments on the ShapeNet dataset using different sampling step counts, starting from 40 and reducing down to 10. We include the results in the table below. We also experimented with more than 40 steps, but this didn't lead to noticeable improvements. Interestingly, reducing the number of steps to 30 appears to be beneficial; however, below 30 steps, the sample quality begins to degrade significantly, as expected.\n\n| **Model / steps** | **40** | **30** | **20** | **10** |\n| :---------------- | :------------ | :------------ | :------------ | :------------ |\n| Our 1-NNA &darr;        | $508 \\pm .005$ | $504 \\pm .006$ | $522 \\pm .008$ | $631 \\pm .018$ |\n| IoU     &uarr;          | $664 \\pm .002$ | $663 \\pm .004$ | $642 \\pm .004$ | $533 \\pm .014$ |\n\n*Table: Results across different step counts. Values are mean $\\pm$ std.*\n\nThank you again for highlighting this issue. We will include these findings in the final revision.\n\n\n> DDPMs inherently support compositional representations especially would be beneficial when performing multi-modal generations. The paper did not address ability to preserve and transfer semantic relationships, compositional representations between modalities and retain usefull properties for image tasks.\n\nThank you for raising this important point and helping us improve the quality of our paper. It's worth emphasizing that we intentionally designed our benchmark to focus on modality translation between modalities with different dimensionalities. However, we fully agree that including benchmarking tasks with aligned modality dimensions can further enhance the robustness and generalizability of our method. To address this, we leveraged the **edges-to-bags task from the DDBM benchmark**, which serves as an appropriate evaluation for your suggestion. We tested our framework on this task, analyzing two key aspects: sample quality (FID) and inference performance (runtime in seconds). The results are presented in the table below and will be included in the final revision of the paper.\n\nBriefly, the findings highlight a clear trade-off: while our method remains competitive in terms of quality, it offers more than 2x faster inference speed compared to DDBM.\n\n| **Method / Metric** | **FID** | **Inference Time (sec)** |\n| :------------------ | :------ | :----------------------- |\n| Ours         &darr;       | 4.17    | 7.8                      |\n| DDBM        &darr;        | 2.93    | 16.9                     |\n\n*Table: Comparison of FID and inference time (in seconds).*\n\nInference time is measured on a batch size of 256. We used [1] autoencoder as the encoder-decoder backbone for our method.\n\n[1] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models\n\n> In general the training framework involves iteratively learning the auto-encoder/latent spaces jointly with the bridge essentially learning the marginal distributions making it a harder optimization problem.\n\nWe agree that jointly learning both the encoder/decoder (which define the latent space marginals) and the diffusion bridge (which assumes fixed marginals during training) introduces a nontrivial optimization challenge. To address this, we introduced and evaluated the iterative training strategy (see Appendix A.3.4 and Table 7) that alternates between optimizing the autoencoders and the bridge module. This approach reduces interference between components and improves the final performance. For example, our iterative setup achieves a 1-NNA of 0.508 and IoU of 0.664 on ShapeNet, outperforming two-step and fully end-to-end alternatives. Its important to note, that while the iterative setup is highly stable, we do not observe major stability issues in other setups as well.\n\nWhile our experiments focus on jointly trained encoders, we note that the use of pre-trained modality-specific foundation models—such as CLIP or DINO for vision, and wav2vec or Whisper for audio—could further mitigate this training burden. These models may stabilize the latent marginals early in training and offer performance gains, particularly in low-data regimes. We consider this a promising direction for future work.\n\n> The paper demonstrates strong capabilities in multi-modal translation. To further showcase the versatility and robustness of the proposed modality agnostic transformer, suggest exploring its application to outpainting and/or image editing tasks in image-to-image translation? This might provide some insights on spatial relationships and compositional representations?\n\nPlease refer to the discussion in the 2nd response.\n\n> Sampling complexity of DDBMs are generally higher to achieve accurate interpolations can the authors elaborate on how the complexity is addressed ?\n\nThe DDBM sampler employs a sampling strategy similar to that used in the Elucidated Diffusion Model (EDM). We observe that even complex bridging tasks, such as the one considered in our work, can be accomplished in no more than **40 sampling steps**—comparable to the number used by EDM for datasets like FFHQ or AFHQv2, which typically require around 40 steps depending on the implementation. Furthermore, the experiment discussed in the first comment provides additional insight, suggesting that further optimization of the number of sampling steps is possible. We'd be happy to provide any further clarification if needed.\n\n> While it is impressive that the approach does not require any conditioning, It would be good to see how does the objective affect the training complexity. Does the process of learning the 'bridge' between two distributions, and the modality-agnostic denoiser in a shared latent space, introduce a higher computational cost (e.g., longer training times, increased GPU memory usage) per iteration or per epoch compared to standard multi-modal EDMs. In additional how does the training convergence compared to standard diffusion pipelines.\n\nRegarding the first point, please refer to our first response; we'd be happy to address any more aspects. As for the second point, we observe that the convergence behavior is very similar. The training time of our model closely matches that of the EDM variants, including both the U-Net and DiT implementations."}, {"review_id": "wVHU7FBSuA", "reviewer": "Reviewer_TLPh", "summary": "This paper introduces a general-purpose framework for Modality Translation (MT) using a novel extension of Denoising Diffusion Bridge Models (DDBMs) into a shared latent space. It proposes a domain-agnostic encoder-decoder architecture and augment it with predictive and contrastive losses to improve semantic consistency and translation accuracy across modalities of differing dimensionality. The proposed method is evaluated on multiple tasks including multi-view to 3D shape generation, image super-resolution, and multi-view scene synthesis, and reaches leading performance over baselines.", "questions": "- Is the bridge model $B$ in the predictive loss $L_{pred}$ also being trained under the bridge loss $L_{bridge}$, or is it frozen when $L_{pred}$ is backpropogating?\n\n- In terms of transitions between different modalities using diffusion models, how does the proposed method relate to and differ from CrossFlow[1] or FlowTok[2], which both directly map between images and texts?\n\n[1] Flowing from Words to Pixels: A Noise-Free Framework for Cross-Modality Evolution\n[2] FlowTok: Flowing Seamlessly Across Text and Image Tokens", "limitations": "The paper only very briefly mentions the computational limitations of processing higher-resolution images.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The proposed method novelly designs an end-to-end predictive loss to train the autoencoder, and proposes to iteratively train the reconstruction and bridging models, which are different from current mainstream two-stage visual generative models.\n\nWeaknesses:\n\n- [Major] The paper targets to address generic transition tasks between different modalities, and the proposed method, including the training scheme and model architecture, is claimed to be designed for this purpose. However, the benchmark tasks are relatively simple, and especially, don't involve much different modalities but are mostly limited to pixel data like images/masks and voxels, which are still very similar. There is no clue that distinct modalities like language, audio, action, or other unusual sensor data formats are involved. Therefore the capability of the proposed framework is doubted.\n\n- The paper writing could be improved. Specifically, there are too many generally descriptive words (e.g. what a component is designed for) of the proposed method, but without actually illustrating what it really is (e..g how a component is designed), since the abstract, introduction and until Ln 206. Concrete definitions of the proposed method should be summarized and emphasized even before the method section, with various granularity.\n\n- The detailed model architecture related to Sec. 4.5 needs to be presented in the main paper. Many descriptive words could be more concise and concentrated, and the analyses (Ln191-203) and even the background (Sec. 3) could be moved to the Appendix if necessary.", "was_revised": true, "final_justification": "Thanks the authors for the informative feedback. I think for the additional experiments on non-visual modalities, I'd further suggest that 1) text would be a more popular and convincing modality to evaluate, and 2) other baseline bridge models need to be compared to demonstrate the effect of the proposed modules, besides the provided ordinary models -- especially [1] is dated back to 2018.\n\nOverall I think there are still potential improvements for this paper. But I wouldn't object to the acceptance to it, given the feedback from other reviewers.", "rebuttal": "We thank the reviewer for the constructive feedback and for acknowledging the novelty of our framework. Below, we respond to the questions and concerns raised. We are happy to provide any additional clarification if needed.\n\n> [Major] The paper targets to address generic transition tasks between different modalities, and the proposed method, including the training scheme and model architecture, is claimed to be designed for this purpose. However, the benchmark tasks are relatively simple, and especially, don't involve much different modalities but are mostly limited to pixel data like images/masks and voxels, which are still very similar. There is no clue that distinct modalities like language, audio, action, or other unusual sensor data formats are involved. Therefore the capability of the proposed framework is doubted.\n\nTo directly address this concern, which was also raised by another reviewer, diversify our benchmarking, and strengthen the robustness of our method results, we introduce two new tasks to our model: *Image-to-Audio* and *Audio-to-Image*, following the benchmark proposed in \\[1]. In these tasks, a multimodal translation model receives one modality as input and generates a meaningful representation, which is then used for downstream classification.\n\nWe evaluate our method against the SiT variant, which showed the overall second-best performance on other benchmarks, as well as against the current state-of-the-art, which is a task-specific method tailored for audio and image representations. The results are shown in the table below. For implementation and evaluation details, we refer the reader to \\[1], and we have also added full experimental details, including encoder and decoder architectures, in the appendix of the revised version.\n\nAs shown in the table, where higher values indicate better classification accuracy, our method outperforms SiT. However, a performance gap remains when compared to the specialized method in \\[1], which is expected given its very task-specific nature. Overall, the results below demonstrate the generality of our framework, its superiority among general multimodal translation models and its possible adaptation to more diversified domains.\n\n| **Method** | **Face to Voice** &uarr; | **Voice to Face**, &uarr; |\n| :--------- | :---------------- | :---------------- |\n| Our method | 71.2              | 75.1              |\n| SiT        | 65.7              | 68.3              |\n| \\[1]       | 79.5              | 81.0              |\n\n*Table: Performance comparison on cross-modal face-voice tasks.*\n\n\\[1] Seeing Voices and Hearing Faces: Cross-modal biometric matching - A. Nagrani, S. Albanie, A. Zisserman\n\n\n> The paper writing could be improved. Specifically, there are too many generally descriptive words (e.g. what a component is designed for) of the proposed method, but without actually illustrating what it really is (e.g. how a component is designed), since the abstract, introduction and until Ln 206. Concrete definitions of the proposed method should be summarized and emphasized even before the method section, with various gra\n    \nWhile we provide formal definitions of our method starting in Section 4 and Appendix A.2 (e.g., latent bridge in Eq. 4, full objective in Eq. 10), we agree that an earlier, more concrete summary would improve clarity. We have changed the paper structure to include a brief structured overview of the method (summarizing key components and pointing to Figure 2) earlier in the paper to better guide the reader through the technical core and directly explain components. We are more than happy to include any further clarification and restructuring if needed.\n\n> The detailed model architecture related to Sec. 4.5 needs to be presented in the main paper. Many descriptive words could be more concise and concentrated, and the analyses (Ln191-203) and even the background (Sec. 3) could be moved to the Appendix if necessary.\n\nThank you for pointing out this point, which will help improve the clarity of our paper. Initially, due to space constraints, we omitted the architecture figure related to Section 4.5. However, in light of your comment—and given that the revised submission allows for an additional page—we have restructured the paper to include this figure, which we agree significantly enhances clarity.\n\n\n> Is the bridge model in the predictive loss also being trained under the bridge loss, or is it frozen when is backpropagating?*\n\nIt depends on the training strategy. In both the Two-Step and End-to-End strategies, the model is always actively trained throughout the training process (With a small exception in the frist step of the Two-Step where only the encoders/decoers are trained). In the Iterative strategy, the model remains frozen if and only if the encoders are being trained, which occurs in an alternating fashion.\n\n> In terms of transitions between different modalities using diffusion models, how does the proposed method relate to and differ from CrossFlow \\[1] or FlowTok \\[2], which both directly map between images and texts?*\n\nThank you for raising this point and giving us the opportunity to clarify how the proposed method relates to and differs from our work. We have added the following discussion to the related work in the final revision to better differentiate our model:\n\nThe first fundamental difference between our method and the two referenced works lies in their task specificity. Both of the mentioned methods focus exclusively on a particular latent transition: from text to image. They incorporate task-specific components into the architecture and loss design to facilitate this transition. In contrast, our work aims to develop a general-purpose solution for latent bridging. We investigate several general aspects, including training strategies, which we study in depth.\n\nAnother major distinction is in the bridge architecture. While both prior works rely on off-the-shelf DiT architectures, our approach introduces a custom-designed architecture specifically tailored to the more general latent bridging setting. We demonstrate both its efficiency and effectiveness in this context.\n\nFinally, from a theoretical standpoint, our method builds on Denoising Diffusion Bridge Models (DDBMs), a fundamentally different framework from the Flow Matching approach used in the referenced works. These frameworks differ in both their loss formulations and sampling procedures. Although Flow Matching is theoretically valid for bridging any two distributions, similar to DDBMs, the specific implementations in the referenced works are designed for Gaussian-to-data transitions. Indeed, both methods incorporate a Gaussian loss term on one side of the bridge, which is not required in our model.\n\nInterestingly, recent theoretical work such as \\[3] proposes extensions to Flow Matching that could enable latent bridging without requiring one side to be Gaussian. This presents a compelling direction for future research. However, the current versions of those methods are constrained by this Gaussian constraint, whereas our DDBM-based approach operates without such a limitation.\n\n\\[3] Stochastic interpolants with data-dependent couplings - Michael S Albergo\\*, Mark Goldstein\\*, Nicholas M Boffi, Rajesh Ranganath, Eric Vanden-Eijnden"}, {"review_id": "e7TCcAeC6F", "reviewer": "Reviewer_8Cc5", "summary": "This work extends Denoising Diffusion Bridge Models (DDBMs) for general modality translation (MT) via a shared latent space leveraging modality-specific encoder/decoders. It utilizes a combination of bridge loss, predictive loss, and contrastive loss for accurate and high-fidelity domain translation. The proposed approach is evaluated on multi-view to 3D shape generation, image super-resolution, and multi-view scene synthesis tasks, showcasing improved performances.", "questions": "- The iterative training protocol significantly improves performance, however, it would be useful to provide more specific details. Is the training time for the iterative procedure significantly longer compared to two-step or end-to-end? \n- It would be great if the authors could provide more intuition for the MASK token and some analysis for why its inclusion drastically improves performance on some tasks (+2.7 points on CelebA-HQ PSNR).\n- In the iterative process, are all three losses applied for both reconstruction and bridge alignment steps? \n- It would be great if the specific DDBM noising method is provided in the background section or appendix for readers who are unfamiliar.\n\nPost rebuttal note:\nThe rebuttal has clarified most of my concerns and I will increase my rating to reflect this.", "limitations": "Yes - computational tradeoffs as well as the limitation of training on paired data are mentioned.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The key contribution of a modality-agnostic translation method is novel, design choices such as the encoder-decoder architecture for the bridge model are well-motivated.\n- Comprehensive ablations are provided for architectural and loss components as well as training procedures for all benchmarks.\n- The proposed approach is effective across all benchmarks. \n\nWeaknesses:\n- The proposed approach is evaluated primarily on vision-based modality translation tasks, it would be interesting to see how well this approach works outside of vision (e.g., text to image). \n- The bridge model requires that the source and domain share a robust latent space. Ablations with lower quality encoder/decoders for each modality would elucidate the effectiveness of the proposed losses. \n- The image super-resolution experiments focus exclusively on faces. It would be interesting to evaluate how well the proposed pipeline generalizes to other domains such as natural images.", "was_revised": true, "final_justification": "The rebuttal has clarified most of my concerns and I will increase my rating to reflect this.", "rebuttal": "We thank the reviewer for the thoughtful feedback and for recognizing the novelty of our framework design and the extensiveness of our experiments. Below, we address the questions and concerns raised. We are happy to provide any additional clarification if needed.\n\n> The proposed approach is evaluated primarily on vision-based modality translation tasks, it would be interesting to see how well this approach works outside of vision (e.g., text to image).\n\nTo directly address this concern, which was also raised by another reviewer, diversify our benchmarking, and strengthen the robustness of our method results, we introduce two new tasks to our model: *Image-to-Audio* and *Audio-to-Image*, following the benchmark proposed in \\[1]. In these tasks, a multimodal translation model receives one modality as input and generates a meaningful representation, which is then used for downstream classification.\n\nWe evaluate our method against the SiT variant, which showed the overall second-best performance on other benchmarks, as well as against the current state-of-the-art, which is a task-specific method tailored for audio and image representations. The results are shown in the table below. For implementation and evaluation details, we refer the reader to \\[1], and we have also added full experimental details, including encoder and decoder architectures, in the appendix of the revised version.\n\nAs shown in the table—where higher values indicate better classification accuracy—our method outperforms SiT. However, a performance gap remains when compared to the specialized method in \\[1], which is expected given its very task-specific nature. Overall, the results below demonstrate the generality of our framework, its superiority among general multimodal translation models and its possible adaptation to more diversified domains.\n\n| **Method** | **Face to Voice** &uarr; | **Voice to Face**  &uarr;|\n| :--------- | :---------------- | :---------------- |\n| Our method | 71.2              | 75.1              |\n| SiT        | 65.7              | 68.3              |\n| \\[1]       | 79.5              | 81.0              |\n\n*Table: Performance comparison on cross-modal face-voice tasks.*\n\n\\[1] Seeing Voices and Hearing Faces: Cross-modal biometric matching - A. Nagrani, S. Albanie, A. Zisserman\n\n> The bridge model requires that the source and domain share a robust latent space. Ablations with lower quality encoder/decoders for each modality would elucidate the effectiveness of the proposed losses.\n\nThank you for raising this ablation perspective, which helps improve the understanding of our method. To address this point, we conducted an experiment within the Super-Resolution benchmark evaluating the impact of encoder architecture. Specifically, we examined three variants:\n    (1) a pre-trained autoencoder \\[1],\n    (2) the same encoder architecture without pre-training, and\n    (3) a simpler encoder consisting of a 5-layer convolutional neural network.\n    We report the results in the table below and will include this experiment in the final revision. Briefly, we observe that both (1) and (2) achieve comparable PSNR scores, suggesting that pre-training provides only marginal benefit in this case. However, variant (3) shows a noticeable drop in performance, indicating that architectural capacity and representation quality play a crucial role in the final outcome. This experiment underscores the importance of encoder quality in our framework.\n\n| **Method / Metric** | **PSNR** &uarr; |\n| :------------------ | :------------- |\n| (1) – Pre-trained AE | $25.6 \\pm 0.4$ |\n| (2) – Vanilla AE    | $25.3 \\pm 0.5$ |\n| (3) – Simple CNN    | $22.9 \\pm 0.9$ |\n\n*Table: Comparison of PSNR across different encoder architectures. Values are mean $\\pm$ std.*\n\n\\[1] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models\n\n> The image super-resolution experiments focus exclusively on faces. It would be interesting to evaluate how well the proposed pipeline generalizes to other domains such as natural images.\n\nWe incorporated the **DIV2K dataset**—a large, diverse collection of high-quality RGB images, into our framework and extracted visual results. It is encouraging to observe that our model successfully performs the super-resolution task, generating high-quality outputs on this dataset. For quantitative evaluation, our method outperforms DiWa, achieving a PSNR score of 28.9 compared to DiWa’s 28.2. These results will be included in the final revision of the paper.\n\n> The iterative training protocol significantly improves performance, however, it would be useful to provide more specific details. Is the training time for the iterative procedure significantly longer compared to two-step or end-to-end?\n\nThank you for raising this important point. For both the End-to-End and Iterative strategies, we used the same total number of training iterations (1,000,000). In practice, we observed that the Iterative strategy is faster in terms of wall-clock time, as it involves fewer gradient computations per iteration. Regarding the Two-Step strategy, we first trained the autoencoders for 100,000 iterations and then trained the diffusion bridge for an additional 1,000,000 iterations. The overall runtime of this variant is comparable to that of the Iterative approach. We will include this discussion and the corresponding runtime comparisons in the revised version of the paper. Thank you again for bringing this to our attention and helping us improve the clarity of our work.\n\n> It would be great if the authors could provide more intuition for the MASK token and some analysis for why its inclusion drastically improves performance on some tasks (+2.7 points on CelebA-HQ PSNR).\n\nThe MASK token, also known as the context summarization token, is a token added to a lot of common transformer-based architectures that participates in all the attention mechanisms, even through the causal ones. Intuitively, as the name suggests, this token learns to effectively combine all the information from the tokens in the context window and generates a *summary* embedding token. This token is commonly used when your application needs to use a summary of all the tokens in the context window, such as when using a ViT for classification applications, they pad with a \\[CLS] token that generates a summary embedding that can be fed into any downstream classifier model. In our method, we use this token to summarize the semantics stored in one of the modalities to effectively transfer them into the other modality using our technique. We have incorporated the above discussion in the final revision to help clarify the motivation.\n\n> In the iterative process, are all three losses applied for both reconstruction and bridge alignment steps?\n    \nThat's a great question. During training, we compute all three losses, but only propagate gradients through the architectural components that are currently being trained. Specifically, when training the encoders and decoders, the bridge parameters remain frozen; conversely, when training the bridge, the encoder and decoder parameters are not updated.\n\n> It would be great if the specific DDBM noising method is provided in the background section or appendix for readers who are unfamiliar.*\n\nWe will be happy to make this information easier to find in the final revision and elaborate more on the topic in the appendix. The specific DDBM noising and denoising processes are described in Section 3, including the forward SDE in Eq. (1), the reverse-time formulation in Eq. (2), and the corresponding training loss in Eq. (3). We also clarify that we adopt the variance exploding (VE) noise schedule from \\[68] (lines 118–120). However, we will try to include more information that will help the reader to gain a better understanding of the DDBM framework."}, {"review_id": "oJMeEVA1nk", "reviewer": "Reviewer_BFU8", "summary": "This paper proposes a novel architecture for modality translation, enabling information transfer across heterogeneous modalities. \nThe key idea is to extend Denoising Diffusion Bridge Models (DDBM) to handle general modality pairs with differing dimensions.\nTo achieve this, the authors introduce an encoder–decoder framework that projects diverse modalities into a shared latent space. Additionally, a contrastive loss is employed to enforce semantic consistency across domains.\nExperiments are conducted on image-to-3D generation and image super-resolution using relatively simple datasets, demonstrating reasonable performance in both tasks.", "questions": "1. ***Originality***: This work extends DDBM into latent space using an encoder-decoder, which has been implemented in DPBridge. What differentiates the proposed method from this highly relevant prior work, DPBridge [21]?\n2. ***Significance***: The second contribution is the contrastive loss that enhances semantic coherence across domains. The paper only shows visual results in Figure 1 using t-SNE. However, the t-SNE is highly sensitive to the hyperparameters. Do they use the same parameter for both basic and ours results? The paper would be stronger to provide ablations on the contrastive loss in Table 2. \n3. ***Clarity***: In figure 1, what's the meaning of the different colors? The denosied output $\\hat{z}_t$ still has noise. How to train the encoder-decoder and diffusion denoiser together in eq.8?\n4. ***Results***: Similar to the DPBridge, the paper would be stronger to compare with the state-of-the-art methods on each task. For example, for image-to-3D, 3DiM [r1] is also a based model trained from scratch. \n\n[r1] Watson, D., Chan, W., Martin-Brualla, R., Ho, J., Tagliasacchi, A., & Norouzi, M. (2022). Novel view synthesis with diffusion models. arXiv preprint arXiv:2210.04628.", "limitations": "Limitations and potential negative societal impact are not discussed.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### ***Strengths***\n\n* A good attempt at an interesting problem\n   * The task of learning a shared latent space for modality translation tasks is very valuable given the heavily different dimensions and distributions in diverse modalities.\n   * This paper makes a good initial attempt to tackle this problem in simple cases (simple synthetic ShapeNet dataset)\n\n* Good writing\n   * The paper is well written, with clear motivations and illustrative visualizations.\n\n### ***Weaknesses***\n\n* (Major) Significance is not well demonstrated\n   * The key motivation of this paper is to extend DDBM to handle general modality pairs with differing dimensions. To achieve this, it is natural to extend it into latent space using an encoder-decoder architecture. However, a prior work DPBridge [21] has implemented a similar idea and shows the best performance on multiple tasks, even compared to these state-of-the-art methods in each domain. Hence, what differentiates the proposed method from this DPBridge [21]?\n   * The second contribution is to implement contrastive loss to enhance semantic coherence across domains. However, such a key contribution is only validated in Figure 1. Why is such coherence distribution important? The significance is not well demonstrated in the paper. It would be stronger to ablate it in Table 2.\n\n* (Major) Oversold technical novelty\n   * In Line 236, the authors claimed \"Inspired by ... we design a denoiser that is better ...\". However, the results of DiT on all tasks are worse than UNet in Table 2. What's the benefit of the new architecture?\n  * In Line 244, \"unlike the original UNet..., our model consumes this memory using the cross-attention...\". Such an operation has been employed in DPBridge [21].\n  * In Line 210, \"Unlike cycle-consistency losses..., our formulation does not require convertibility or backward mapping...\". The cycle-consistency losses are designed for unpaired data. However, the proposed architecture is trained with paired data. \n \n* (Minor) Experiments are too simple\n    * To explore the new architecture, it is reasonable to deal experiments with Toy datasets. However, it is still important to report the state-of-the-art performance in each domain. For example, for multiview-to-3D shape generation, 3DiM[r1] and Shap-e[r2] are also trained from scratch. The paper would be stronger to compare one of these state-of-the-art approaches for each task.\n    * For visual results in the main paper and supp., the fair comparison to prior works is missing.\n\n[r1] Watson, D., Chan, W., Martin-Brualla, R., Ho, J., Tagliasacchi, A., & Norouzi, M. (2022). Novel view synthesis with diffusion models. arXiv preprint arXiv:2210.04628.\n\n[r2] Jun, H., & Nichol, A. (2023). Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305.02463.", "was_revised": true, "final_justification": "Thanks for the detailed rebuttals. All my concerns have been addressed,  especially the motivation of comparing with DPbridge and the results in other additional tables. Hence, I would like to increase my score and vote to accept this paper.", "rebuttal": "We sincerely thank the reviewer for their thoughtful feedback and for acknowledging the significance of our problem, our initial efforts, and the clear writing of the paper. We have addressed each question and concern raised below and are happy to provide any additional clarification if needed. We apologize for abbreviating some of the reviewer comment references due to space constraints in the rebuttal.\n\n> The key motivation of this paper is to extend DDBM to handle general modality pairs with differing dimensions. To achieve this, it is natural to extend it into latent space using an encoder-decoder architecture. However, a prior work DPBridge \\[21] has implemented a similar idea and shows the best performance on multiple tasks, even compared to these state-of-the-art methods in each domain. Hence, what differentiates the proposed method from this DPBridge \\[21]?\n\nThank you for the opportunity to clarify the distinctions between our framework and DPBridge and improve our work's clarity. We will incorporate the below discussion in the new revision's related work.\n\nIt is important to note that DPBridge is not a *general* multimodal translation framework under our problem definition in Section 4.1. It was specifically designed for image-to-image translation and is based on the Stable Diffusion (SD) model, which is restricted to the image domain. As such, it cannot be applied to the broader range of tasks we address, including 3D voxel prediction and our newly introduced audio-to-image and image-to-audio translation tasks (see our first response to Reviewer 8Cc5 for more details). Consequently, a direct comparison is not feasible. Beyond this, our method introduces several fundamental differences. First, we propose a composition of loss functions, combining InfoNCE (contrastive), predictive, and standard diffusion bridge losses, and we conduct a thorough ablation study to evaluate their individual and joint effects (see Table 6 in the appendix). DPBridge, in contrast, does not explore alignment-based losses. Second, DPBridge employs a pre-trained Stable Diffusion architecture as its backbone, which limits its generality. In contrast, we design and validate a modality-agnostic architecture specifically tailored to support diverse modality pairs without sacrificing performance. Additionally, while DPBridge relies solely on fine-tuning SD (with all other components frozen), our framework trains all components from scratch, and we are the first to explore various training strategies for building a general-purpose multimodal bridge model, extending the scope of our work beyond architectural and loss design. Finally, it is worth noting that SD v2.1, which DPBridge relies on, was pre-trained on hundreds of millions of images. As shown in their ablation study (Table 3 in their paper), their method performance drops significantly without the use of this foundational model, falling well below state-of-the-art levels. \n\n>  The second contribution is to implement contrastive loss to enhance semantic coherence across domains...\n\nThank you for raising this important point and helping us improve the paper quality. We do include such an extensive evaluation in our work, as shown in Table 6. However, we appreciate you bringing to our attention that it was not clearly referenced from the main paper. To address this, we have revised the final submission to move this table into the main paper, taking advantage of the additional space now permitted and in accordance with your suggestion. In this evaluation, we present a quantitative comparison of the four loss combinations discussed in the paper. The results show that incorporating the InfoNCE contrastive loss into the training protocol significantly enhances model performance.\n\n> In Line 236, the authors claimed \"Inspired by ... we design a denoiser that is better ...\". However, the results of DiT on all tasks are worse than UNet in Table 2. What's the benefit of the new architecture?\n\nWe apologize for the lack of clarity in our original explanation and thank you for helping us improve the presentation of our contribution. We would like to emphasize that the denoiser architecture used in our framework is **not** a DiT. Instead, we propose an improved architecture tailored for general modality translation tasks, drawing inspiration from natural language translation. Specifically, we employ an encoder-decoder Transformer architecture as our denoising network—unlike DiT, which is based on a decoder-only Transformer design. In addition to this architectural shift, we integrate several components specifically designed to enhance performance in the multimodal setting such as spatial embedding and mask token. As demonstrated in our ablation study (Table 2), this architecture significantly outperforms the commonly used U-Net backbone. Moreover, Table 2 also shows that our denoiser surpasses a standard DiT in modality translation tasks. We have revised Section 4.5 of the paper to include this clarification, in light of this valuable feedback.\n\n> In Line 244, \"unlike the original UNet..., our model consumes this memory using the cross-attention...\". Such an operation has been employed in DPBridge \\[21].\n\nIn the DPBridge framework, the authors do not use the cross-attention feature of the UNet, which is reserved for text-prompt conditioning in stable diffusion. As the authors of DPBridge discuss in their paper, \"As DPBridge is an image-conditioned generative framework, we employ null-text embedding for all cross-attention layers to eliminate the impact of text prompts,\" suggesting that the application of any cross-attention layer in their framework has been nullified. The target modality information is concatenated with the source modality at the input of the UNet, along the channels dimension. Contrastingly, since our model uses a transformer encoder and decoder, we use cross-attention to condition our modality translation.\n\n> In Line 210, \"Unlike cycle-consistency losses..., our formulation does not require convertibility or backward mapping...\". The cycle-consistency losses are designed for unpaired data. However, the proposed architecture is trained with paired data.\n\nWe would like to clarify that our intention was simply to convey that our method does not require a full cycle starting from the diffusion bridge and returning back. This was mentioned as a minor comment to help provide additional intuition for the reader, and was not intended as a comparison to unpaired data methods. We understand how this phrasing may have caused confusion. To address the reviewer’s concern and avoid any further misunderstanding, we will remove this line in the revised version of the paper.\n\n>  To explore the new architecture, it is reasonable to ...\n\nWe appreciate the opportunity to clarify our position on this point. While we agree that comparisons to state-of-the-art methods can be informative, we would like to emphasize that such comparisons are not entirely fair to our framework. Our model is designed to be general and capable of handling a wide range of multimodal translation tasks. In contrast, state-of-the-art methods in the benchmarks we evaluate on are highly specialized, incorporating strong inductive biases tailored to the specific data domain through architectural choices, loss functions, and training procedures.\n\nNonetheless, we agree that including such comparisons can help convey the broader context and performance trade-offs. Therefore, in response to your suggestion, we have added the results of a task-specific state-of-the-art method \\[1] to the quantitative evaluation table for the Multiview-to-3D Shape Generation task (highlighted in gray). Additionally, for the Zero-Shot Low-to-High Resolution Generation task, we already include a comparison with DiWa. Finally, we have also added comparisons to relevant baselines for the new image-to-audio and audio-to-image tasks introduced in this work. Thank you again for raising this point and helping us improve the completeness of our benchmarking.\n\n\\[1] Dan Wang, et al - Multi-view 3D Reconstruction with Transformer\n\n> For visual results in the main paper and supp., the fair comparison to prior works is missing.\n\nWe include a visual comparison in Figures 7 and 8 of the appendix. However, we acknowledge that there was no clear reference to this experiment in the main paper. To improve the clarity and reading flow, we have added a reference to this experiment in the revised version of the main text. Thank you for pointing this out.\n\n> This work extends DDBM ...\n\nPlease see our first comment.\n\n> The second contribution is the contrastive loss ...\n\nThank you for this question. Yes, we use the exactly same hyperparameters for both the basic and ours. Regarding the second point, we include ablations in the appendix; please see our second comment for full explanation.\n\n> In figure 1, what's the meaning of the different colors? The denoised output still has noise. How to train the encoder-decoder and diffusion denoiser together in eq.8?\n\nEach color represents a category in the dataset—for example, a car or an airplane. We compute both the reconstruction loss and the predictive loss using the reconstructions of the original latent representation and the denoised latent representation. This setup enables standard backpropagation-based training using a single optimizer, as all components are implemented within a unified PyTorch module.\n\n> Results: Similar to the DPBridge, the paper would be ...\n\nAs previously addressed, we welcomed the suggestion and have added comparisons with state-of-the-art methods. Unlike DPBridge, which is specifically designed for the tasks it addresses and leverages a powerful pre-trained foundational model for fine-tuning, our framework is intended to be much more general, and thus a direct comparison may be unfair. As discussed earlier, this introduces a natural trade-off between performance and generality."}]

Author rebuttal:

Meta-review: The paper proposes denoising diffusion bridge models for enabling multimodal translation. The key innovation is to use a shared latent space using a modality agnostic encoder-decoder. A combination of bridge loss, predictive loss, and multimodal contrastive loss are used to train the diffusion bridge. Experiments are provided on two  tasks: 3D shape generation from multi-view images and multi-view face generation.

The paper received overall positive reviews with the reviewers appreciating the importance of the problem setup and the technical exposition. There were important concerns raised on multiple fronts:
1) Relatively simpler benchmark tasks used in the experiments (TLPh, 8Cc5, Egf9) -- to which authors provided additional results during the rebuttal on the task of audio-to-image as well as on edges-to-bags (with aligned modalties) showing compelling results.
2) Lack of analysis on compute complexity, ablations on losses (Egf9, BFU8) -- authors point to the numerical results in the Appendix showing better run-time efficiency and model size
3) Concerns on similarities to prior works such as CrossFlow, FlowTok, DPBridge, etc. were pointed out (TLPh) -- authors argued that prior architectures are specific to certain domains, while the proposed model is modality agnostic.

AC had an independent reading of the paper and agrees with the reviewers that the paper makes an interesting contribution to the multimodal translation domain. Authors provided a strong rebuttal with additional experiments and numerical results addressing a majority of the reviewers' concerns. 

Some concerns remained after the discussion, such as the need for additional ablation studies (Egf9), missing studies on non-visual domains (TLPh), and missing benchmarking studies to stronger baselines (TLPh). AC also thinks the audio-to-image results provided during the rebuttal would need visual scrutiny for quality, which is not possible.

Taking into consideration the overall reviewer sentiment and the  technical exposition, AC recommends accepting the paper. Authors should address all the remaining concerns in the camera-ready paper, including revising the paper for clarity and incorporation of the reviewers suggestions.
Decision: Accept (poster)