Title: Towards General Modality Translation with Contrastive and Predictive Latent Diffusion Bridge
Venue: NeurIPS 2025
Accepted: yes
Keywords: Modality Translation

Diffusion Models

General Cross-Domain Generation

Latent Space
Abstract: Recent advances in generative modeling have positioned diffusion models as state-of-the-art tools for sampling from complex data distributions. While these models have shown remarkable success across single-modality domains such as images and audio, extending their capabilities to *Modality Translation (MT)*, translating information across different sensory modalities, remains an open challenge. Existing approaches often rely on restrictive assumptions, including shared dimensionality, Gaussian source priors, and modality-specific architectures, which limit their generality and theoretical grounding. In this work, we propose the Latent Denoising Diffusion Bridge Model (LDDBM), a general-purpose framework for modality translation based on a latent-variable extension of Denoising Diffusion Bridge Models. By operating in a shared latent space, our method learns a bridge between arbitrary modalities without requiring aligned dimensions. We introduce a contrastive alignment loss to enforce semantic consistency between paired samples and design a domain-agnostic encoder-decoder architecture tailored for noise prediction in latent space. Additionally, we propose a predictive loss to guide training toward accurate cross-domain translation and explore several training strategies to improve stability. Our approach supports arbitrary modality pairs and performs strongly on diverse MT tasks, including multi-view to 3D shape generation, image super-resolution, and multi-view scene synthesis. Comprehensive experiments and ablations validate the effectiveness of our framework, establishing a new strong baseline in general modality translation. For more information, see our project page:  https://sites.google.com/view/lddbm/home.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel architecture for modality translation, enabling information transfer across heterogeneous modalities. \nThe key idea is to extend Denoising Diffusion Bridge Models (DDBM) to handle general modality pairs with differing dimensions.\nTo achieve this, the authors introduce an encoder–decoder framework that projects diverse modalities into a shared latent space. Additionally, a contrastive loss is employed to enforce semantic consistency across domains.\nExperiments are conducted on image-to-3D generation and image super-resolution using relatively simple datasets, demonstrating reasonable performance in both tasks.\n\nSTRENGTHS AND WEAKNESSES:\n### ***Strengths***\n\n* A good attempt at an interesting problem\n   * The task of learning a shared latent space for modality translation tasks is very valuable given the heavily different dimensions and distributions in diverse modalities.\n   * This paper makes a good initial attempt to tackle this problem in simple cases (simple synthetic ShapeNet dataset)\n\n* Good writing\n   * The paper is well written, with clear motivations and illustrative visualizations.\n\n### ***Weaknesses***\n\n* (Major) Significance is not well demonstrated\n   * The key motivation of this paper is to extend DDBM to handle general modality pairs with differing dimensions. To achieve this, it is natural to extend it into latent space using an encoder-decoder architecture. However, a prior work DPBridge [21] has implemented a similar idea and shows the best performance on multiple tasks, even compared to these state-of-the-art methods in each domain. Hence, what differentiates the proposed method from this DPBridge [21]?\n   * The second contribution is to implement contrastive loss to enhance semantic coherence across domains. However, such a key contribution is only validated in Figure 1. Why is such coherence distribution important? The significance is not well demonstrated in the paper. It would be stronger to ablate it in Table 2.\n\n* (Major) Oversold technical novelty\n   * In Line 236, the authors claimed \"Inspired by ... we design a denoiser that is better ...\". However, the results of DiT on all tasks are worse than UNet in Table 2. What's the benefit of the new architecture?\n  * In Line 244, \"unlike the original UNet..., our model consumes this memory using the cross-attention...\". Such an operation has been employed in DPBridge [21].\n  * In Line 210, \"Unlike cycle-consistency losses..., our formulation does not require convertibility or backward mapping...\". The cycle-consistency losses are designed for unpaired data. However, the proposed architecture is trained with paired data. \n \n* (Minor) Experiments are too simple\n    * To explore the new architecture, it is reasonable to deal experiments with Toy datasets. However, it is still important to report the state-of-the-art performance in each domain. For example, for multiview-to-3D shape generation, 3DiM[r1] and Shap-e[r2] are also trained from scratch. The paper would be stronger to compare one of these state-of-the-art approaches for each task.\n    * For visual results in the main paper and supp., the fair comparison to prior works is missing.\n\n[r1] Watson, D., Chan, W., Martin-Brualla, R., Ho, J., Tagliasacchi, A., & Norouzi, M. (2022). Novel view synthesis with diffusion models. arXiv preprint arXiv:2210.04628.\n\n[r2] Jun, H., & Nichol, A. (2023). Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305.02463.\n\nQUESTIONS:\n1. ***Originality***: This work extends DDBM into latent space using an encoder-decoder, which has been implemented in DPBridge. What differentiates the proposed method from this highly relevant prior work, DPBridge [21]?\n2. ***Significance***: The second contribution is the contrastive loss that enhances semantic coherence across domains. The paper only shows visual results in Figure 1 using t-SNE. However, the t-SNE is highly sensitive to the hyperparameters. Do they use the same parameter for both basic and ours results? The paper would be stronger to provide ablations on the contrastive loss in Table 2. \n3. ***Clarity***: In figure 1, what's the meaning of the different colors? The denosied output $\\hat{z}_t$ still has noise. How to train the encoder-decoder and diffusion denoiser together in eq.8?\n4. ***Results***: Similar to the DPBridge, the paper would be stronger to compare with the state-of-the-art methods on each task. For example, for image-to-3D, 3DiM [r1] is also a based model trained from scratch. \n\n[r1] Watson, D., Chan, W., Martin-Brualla, R., Ho, J., Tagliasacchi, A., & Norouzi, M. (2022). Novel view synthesis with diffusion models. arXiv preprint arXiv:2210.04628.\n\nLIMITATIONS:\nLimitations and potential negative societal impact are not discussed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work extends Denoising Diffusion Bridge Models (DDBMs) for general modality translation (MT) via a shared latent space leveraging modality-specific encoder/decoders. It utilizes a combination of bridge loss, predictive loss, and contrastive loss for accurate and high-fidelity domain translation. The proposed approach is evaluated on multi-view to 3D shape generation, image super-resolution, and multi-view scene synthesis tasks, showcasing improved performances.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The key contribution of a modality-agnostic translation method is novel, design choices such as the encoder-decoder architecture for the bridge model are well-motivated.\n- Comprehensive ablations are provided for architectural and loss components as well as training procedures for all benchmarks.\n- The proposed approach is effective across all benchmarks. \n\nWeaknesses:\n- The proposed approach is evaluated primarily on vision-based modality translation tasks, it would be interesting to see how well this approach works outside of vision (e.g., text to image). \n- The bridge model requires that the source and domain share a robust latent space. Ablations with lower quality encoder/decoders for each modality would elucidate the effectiveness of the proposed losses. \n- The image super-resolution experiments focus exclusively on faces. It would be interesting to evaluate how well the proposed pipeline generalizes to other domains such as natural images.\n\nQUESTIONS:\n- The iterative training protocol significantly improves performance, however, it would be useful to provide more specific details. Is the training time for the iterative procedure significantly longer compared to two-step or end-to-end? \n- It would be great if the authors could provide more intuition for the MASK token and some analysis for why its inclusion drastically improves performance on some tasks (+2.7 points on CelebA-HQ PSNR).\n- In the iterative process, are all three losses applied for both reconstruction and bridge alignment steps? \n- It would be great if the specific DDBM noising method is provided in the background section or appendix for readers who are unfamiliar.\n\nPost rebuttal note:\nThe rebuttal has clarified most of my concerns and I will increase my rating to reflect this.\n\nLIMITATIONS:\nYes - computational tradeoffs as well as the limitation of training on paired data are mentioned.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a general-purpose framework for Modality Translation (MT) using a novel extension of Denoising Diffusion Bridge Models (DDBMs) into a shared latent space. It proposes a domain-agnostic encoder-decoder architecture and augment it with predictive and contrastive losses to improve semantic consistency and translation accuracy across modalities of differing dimensionality. The proposed method is evaluated on multiple tasks including multi-view to 3D shape generation, image super-resolution, and multi-view scene synthesis, and reaches leading performance over baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The proposed method novelly designs an end-to-end predictive loss to train the autoencoder, and proposes to iteratively train the reconstruction and bridging models, which are different from current mainstream two-stage visual generative models.\n\nWeaknesses:\n\n- [Major] The paper targets to address generic transition tasks between different modalities, and the proposed method, including the training scheme and model architecture, is claimed to be designed for this purpose. However, the benchmark tasks are relatively simple, and especially, don't involve much different modalities but are mostly limited to pixel data like images/masks and voxels, which are still very similar. There is no clue that distinct modalities like language, audio, action, or other unusual sensor data formats are involved. Therefore the capability of the proposed framework is doubted.\n\n- The paper writing could be improved. Specifically, there are too many generally descriptive words (e.g. what a component is designed for) of the proposed method, but without actually illustrating what it really is (e..g how a component is designed), since the abstract, introduction and until Ln 206. Concrete definitions of the proposed method should be summarized and emphasized even before the method section, with various granularity.\n\n- The detailed model architecture related to Sec. 4.5 needs to be presented in the main paper. Many descriptive words could be more concise and concentrated, and the analyses (Ln191-203) and even the background (Sec. 3) could be moved to the Appendix if necessary.\n\nQUESTIONS:\n- Is the bridge model $B$ in the predictive loss $L_{pred}$ also being trained under the bridge loss $L_{bridge}$, or is it frozen when $L_{pred}$ is backpropogating?\n\n- In terms of transitions between different modalities using diffusion models, how does the proposed method relate to and differ from CrossFlow[1] or FlowTok[2], which both directly map between images and texts?\n\n[1] Flowing from Words to Pixels: A Noise-Free Framework for Cross-Modality Evolution\n[2] FlowTok: Flowing Seamlessly Across Text and Image Tokens\n\nLIMITATIONS:\nThe paper only very briefly mentions the computational limitations of processing higher-resolution images.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel approach to multi-modal domain translation, specifically addressing a key limitation of existing bridge models. While current methods, such as those relying on optimal transport for tasks like image-to-image or image-to-3D translation, are often restricted by the defined marginal spaces of their start and end distributions, this work offers a new solution to model the distribution translation using latent bridge by estimating a conditional distribution conditioned on both the modalities by factorizing the conditional distribution and introducing a shared latent space. \nIn addition they define a modality agnostic transformer based encoder -decoder training framework eliminating the need to additional conditioning and any implicit biases added due to the architecture.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper is well motivated and the structured and directly addresses the limitations of bridge models and modality specific encoder decoder architectures. \n\n\n2. The authors introduce a sampling approach for training diffusion-based bridge models, specifically by learning a joint latent space shared across multiple modalities. This method offers a latent space that is trained to be more semantic. Current approaches to learn a shared latent space in multi-model models involve learning a bottleneck layer or mixture of expert architectures which require large number of pairs samples to learn from and the validation and practical deployment of these specific diffusion based approaches have been challenging. \n\n3. This work proposes a general purpose transformer based encoder-decoder architecture that is trained for 3 tasks super-resolution, 3D shape generation and occupancy tasks. The idea of universal auto-encoder decoder architecture is novel and to the best of my knowledge not been explored. \n\n4.  This work qualitatively illustrates the limitations of baseline objective demonstrating good intuitions on semantic alignment . In addition to comparing the proposed architecture Distribution Translation DiT against UNET and naive DiT architecture\n\nWeakness: \n\n1.  Although the results look very promising, the paper does not sufficiently address or analyze the inference time complexity of their bridge model approach compared to other multi-modal shared latent space methodologies that define a separate encoder decoder models.   Secondly Sampling complexity of DDBMs are generally higher to achieve accurate interpolations, would be curious to see how is the sample quality affected \n\n2.  DDPMs inherently support compositional representations especially would be beneficial when performing multi-modal generations. The paper did not address ability to preserve and transfer semantic relationships, compositional representations between modalities and retain usefull properties for image tasks.\n\n3. In general the training framework involves iteratively learning the auto-encoder/latent spaces jointly with the bridge essentially learning the marginal distributions making it a harder optimization problem.\n\nQUESTIONS:\n1. The paper demonstrates strong capabilities in multi-modal translation. To further showcase the versatility and robustness of the proposed modality agnostic transformer, suggest exploring its application to outpainting and/or image editing tasks in image-to-image translation? This might provide some insights on spatial relationships and compositional representations? \n\n2. Sampling complexity of DDBMs are generally higher to achieve accurate interpolations can the authors elaborate on how the complexity is addressed ? \n\n3.  While it is impressive that the approach does not require any conditioning, It would be good to see how does the objective affect the training complexity.  Does the process of learning the 'bridge' between two distributions, and the modality-agnostic denoiser in a shared latent space, introduce a higher computational cost (e.g., longer training times, increased GPU memory usage) per iteration or per epoch compared to standard multi-modal EDMs. In additional how does the training convergence compared to standard diffusion pipelines.\n\nLIMITATIONS:\nThe approach is interesting and very relavent, However I have requested for clarifications on complexity and ability to model spatial relationships.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-07"}