Improving Distribution Matching via Score-Based Priors and Structural Regularization

Authors: Ziyu Gong, Jim Lim, David I. Inouye
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: distribution matching, score-based models, representation learning

Abstract:
Distribution matching (DM) can be applied to multiple tasks including fair classification, domain adaptation and domain translation.
However, traditional variational DM methods such as VAE-based methods unnecessarily bias the latent distributions towards simple priors or fail to preserve semantic structure leading to suboptimal latent representations.
To address these limitations, we propose novel VAE-based DM approach which incorporates a flexible score-based prior and a semantic structure preserving regularization.
For score-based priors, the key challenge is that computing the likelihood is expensive.
Yet, our key insight is that computing the likelihood is unnecessary for updating the encoder and thus we prove that the necessary gradients can be computed using only one score function evaluation.
Additionally, we adapted the structure preserving regularization inspired by the Gromov-Wasserstein distance, which explicitly encourages the retention of geometric structure in the latent space, even when the latent space has fewer dimensions than the observed space. 
Our framework further allows the integration of semantically meaningful structure from pretrained or foundation models into the latent space, ensuring that the representations preserve semantic structure that is informative and relevant to downstream tasks.
We empirically demonstrate that our DM approach leads to better latent representations compared to similar methods for fair classification, domain adaptation, and domain translation tasks.

Decision: reject
Reviewer scores: 3, 3, 3, 6, 5

Reviews and discussion:
<thread>
creation date:
1729606400440

thread_type:
official_review

<thread_object>
creator: reviewer_7j8h

<content>
summary:
This paper considered the distribution matching approach through the scoring matching framework. By overcoming the limitations on both adversarial training and variational inference, the proposed scoring matching method incorporated Gromow-Wasserstein (GW). Experiences on fairness, domain adaptation and domain translation have been done.

strengths:
1. This paper naturally extended the previous work Gong (2024) by improving certain limitations in variational inference. 
2. The paper is seemingly technically sound. 
3. The experiments are conducted in diverse domains and applications, thereby a clear improvement.

weaknesses:
Unfortunately, I think this paper clearly falls short for acceptance. 

My most important concerns lies in a significant lack of **clear, concrete and convincing** supporting evidence within the paper. The overall feeling is full of self-claim but very limited justification and support. Most arguments are simply presented without clear and concrete support. The following are notable concerns.

- I could not understand which parts constitute the most significant contributions. If we check the claimed contribution in the paper. 

> Introduction of Score-Based Priors for Flexible Representation: We propose score based priors, enhancing flexibility and preserving complex data structures without requiring sampling or likelihood estimation. 

The paper claimed many benefits of proposed methods. However, this reviewer did not understand why they are beneficial or what’s the point in the context of distribution matching. For example, *flexibility and preserving complex data structures without requiring sampling or likelihood estimation*. I could not understand why likelihood estimation is a bad thing. In complex data structures, how complex is it? From the paper, the actual contribution in data structure is about the cluster assumption. Does cluster information is equivalent to complex data structure? I believe there is a significant gap here. How to justify the flexibility, can you give concrete evidence such as improving the efficiency?

From the analysis and paper, I did not clearly identify the concrete and convincing evidences for the claim. 

> Structure-Preserving Constraints Inspired by Gromov-Wasserstein Distance: We introduce a Gromov-Wasserstein-based constraint to preserve geometric relationships, ensuring robust, task-relevant latent representations across transformations.

Again there are several questions regarding this claim. How GW distance explicitly preserves the geometric relationships, why not other relevant techniques such as hierarchical generative models could not? There is a lack of clear rationale and throughout comparison without others. 

This contribution is also claimed by robust, task-relevant latent representations across transformations. However, this reviewer could find very limited **concrete** supporting evidence about robust, task-relevant latent representations. 

> Empirical Validation: Our experiments demonstrate improved downstream task performance in fairness learning, domain adaptation, and domain translation using score-based priors and structural preservation.

I found this to be the most concerning point. Indeed, what does it mean by improved downstream tasks in Digit dataset or tabular Adult dataset? Do we need downstream tasks in these datasets? Why do we need to learn a representation in these datasets? The representation learning, back to the original sense, aims to learn meaningful information from very high-dimensional and complex datasets. I could not understand how the proposed experiments are associated with real-world representation learning in this sense. 

*Using score-based priors and structural preservation.* Again, I could not think of this as an advantage, this paper simply compared some old baselines and the most relevant paper (Gong, 24). But the fact is that there is a rich literature in generative models that incorporate score-based or structural level information. There is a clear lack of comparison. 


- There are many awkward claims within the paper. I will list several in the introduction.  

> (List 33-35) Unfortunately, simply collecting more data or building bigger models is unlikely to solve these problems, as they require imposing additional constraints on the learning process.

What is the supporting evidence by saying *more data or bigger models is unlikely to ….*. There are many papers that indeed support more diverse datasets or bigger models could solve this issue, such as scaling law papers. This essentially reveals the larger, more data is indeed better. 

*they require imposing additional constraints on the learning process*, again what is the supporting evidence to say additional constraints? Can you provide concrete evidence by justifying it?


> (Line 43) One of the most used distribution matching methods is adversarial models,

This claim is not necessarily true. Indeed, most generative models can be broadly viewed as distribution matching between data distribution and generative distribution such as VAE (though KL divergence), score matching (via Fisher divergence).
> (Line 51-53) While this simplifies the optimization process and
ensures tractability during generative tasks, it biases the latent space, often leading to a loss of critical structure in the data during transformation.

This is not always true. For example, the high-dimensional gaussian distribution is indeed expressive. Considering the experimental datasets such as Adult and Digits, I think this is sufficient. 


> (Line 65) For instance, domain adaptation tasks require not only distribution alignment but also the preservation of clusters or other semantic features.

My question is that if we have a perfect distribution alignment, that should imply the cluster or other semantic feature will also be well-aligned, right?

> (Line 75) Unlike Gaussian priors, score-based models do not bias the latent representations towards a fixed distribution, enabling
the model to capture richer and more nuanced patterns. 

General form is richer but also enhances the risk of overfitting, right?

> (Line 85) Our framework can also integrates semantic information from pretrained models, such as CLIP(Radford et al., 2021),

Unfortunately there is no concrete supporting evidence for this. 


> (Line 11-12) Distribution matching (DM) can be applied to multiple tasks including fair classification,


Can you differentiate fair classification and fair representation learning **in your paper**? What are the key differences in your context? What are the exact downstreaming applications in your paper if we consider fair representation learning?

questions:
See the weakness section. Overall, In the rebuttal or future revision I would strongly suggest authors 

- For each statement or argument, please provide concrete evidence such as citation from well-established literature, your own experimental results or throughout analysis. 
- Precisely and accurately express all the notations and terminologies.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730589459368

thread_type:
official_review

<thread_object>
creator: reviewer_r3fp

<content>
summary:
This manuscript introduces a novel VAE structure that leverages score-based priors instead of the Gaussian ones, which might overcome some limitations of conventional VAEs by allowing the model to learn more complex patterns in the latent space without needing explicit likelihood calculations. Moreover, the authors incorporate a structure-preserving regularization based on the Gromov-Wasserstein distance, which maintains the geometric relationships among data. The authors demonstrate the effectiveness of the proposed methods on various tasks, including fair classification, domain adaptation, and domain translation. By combining theoretical insights with practical applications, this work could take a step forward in distribution matching.

######################################### Post Rebuttal ######################################### 

None author response found. I will keep my score.

######################################### Post Rebuttal #########################################

strengths:
1. The manuscript is well-written and clearly motivated, with the step-by-step derivations and sufficient literature to understand each proposed component.

2. Using score-based priors instead of the Gaussian priors in variational inference methods like VAEs is interesting and promising, which might help the model identify more complex hidden patterns in the input data. Moreover, using the score function to evaluate the encoder gradients might associate how the model learns its parameters with changes in the data’s probability density. 

3. Using the Gromov-Wasserstein distance as the regularization term to make sure that the latent space maintains the semantic structure of the data is interesting.

weaknesses:
1. Although using score-based priors is interesting, however, score-based priors are inherently more computationally intensive than simple Gaussian priors, potentially leading to longer training times. I would suggest an ablation study to compare the runtime complexity (e.g., training time per epoch, total training time until convergence, and inference time) of the proposed score-based priors with those of VAEs using the Gaussian priors. The evaluation metric could be the time in seconds, along with an indicator specifying the type of GPU platform used for these experiments.

2. From Equation (13) to Equation (14), it seems like noisy versions of the latent samples are introduced to develop a denoising score matching objective for the proposed method. If that is the case, based on my understanding, it seems the method still uses a Gaussian prior in variational inference but with more constrained regularization. I suggest the authors provide more details on how to implement Equation (14) for a specific task (e.g., domain adaptation or fair classification).

questions:
1. Could the authors elaborate more on why other optimal transport methods can only compare points from spaces with the same number of dimensions?

2. Equation (14) shows the final derivation of the proposed method, which, to me, appears similar to denoising score matching in the latent space. Could the authors explain how the proposed method differs from denoising score matching?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730691814466

thread_type:
official_review

<thread_object>
creator: reviewer_woto

<content>
summary:
The paper proposes a VAE-based distribution matching approach using a score-based prior. The authors introduce the Score Function Substitution (SFS) trick that facilitates efficient VAE training.
Additionally, they combine the VAE objective with a GW regularizer to ensure that the latent space retains the structural/semantic properties of the data space. The authors validate their approach across several applications, including fair classification, domain adaptation and domain translation.

strengths:
**S1 |** Score Function Substitution (SFS) is a simple and interesting modification of the LSGM approach.

**S2 |** The experiments in Sections 5.1 and 5.2 support the proposed method.

weaknesses:
**W1 |** It would be beneficial to provide more clarification and motivation regarding the VAUB formulation and the associated challenges. I can see that VAUB is simply a domain/class-conditioned VAE with a shared learnable prior. If so, proposing score-based priors for this formulation is a marginal contribution compared to LSGM[1].

**W2 |** The Score Function Substitution (SFS) appears very similar, if not identical, to score distillation sampling (SDS) [2]. This idea has been widely explored for text-to-3D generation [2,3] and diffusion distillation [4,5,6,7]. The proposed method can be interpreted as a direct SDS application to LSGM[1]. It is important to discuss these works and their connection to the proposed method thoroughly. 

**W3 |** The use of the GW metric for VAE does not seem novel either [9,10]. The connection to this line of work also needs to be carefully discussed. For example, GWAE[9] shares similar ideas and motivation. 

**W4 |** The experimental setups lack important details, making it difficult to understand how the proposed method is exactly applied to various tasks. For example, what are the model inputs and targets for the source and target domains across all tasks?

**W5 |** I do not think that domain adaptation and translation between USPS and MNIST are appropriate tasks, as the domains appear too similar. Also, why were CLIP embeddings chosen for MNIST? Could the authors consider exploring other tasks for this problem, such as those suggested in [8]?

**W6 |** I can hardly agree that the domain translation results are informative. While it is evident that the method preserves the original content better than VAUB, it struggles with effective translation in most cases.

**W7 |** If I understand correctly, the usefulness of the GW regularizer is demonstrated only for the domain adaptation task. Could the authors investigate the effect of this regularizer on other tasks when it is applicable? Could the authors provide insights into which cases the regularizer is more effective? 

**W8 |** Given that the SFS modification approximates the objective in LSGM [1], it would be interesting to compare these two approaches directly.

**Minor**

* $d$ is undefined in Section 2. 

----
[1] Vahdat et al. Score-based Generative Modeling in Latent Space, 2021

[2] Poole et al. DreamFusion: Text-to-3D using 2D Diffusion, 2022

[3] Wang et al. ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation, 2023

[4] Yin et al. One-step Diffusion with Distribution Matching Distillation 2023

[5] Yin et al. Improved Distribution Matching Distillation for Fast Image Synthesis, 2024

[6] Zhou et al. Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation, 2024

[7] Salimans et al., Multistep Distillation of Diffusion Models via Moment Matching, 2024

[8] The Variational Fair Autoencoder, 2015

[9] Nakagawa et al. Gromov-wasserstein autoencoders, 2023

[10] Xu et al. Learning Autoencoders with Relational Regularization, 2020

questions:
Please address the questions and concerns in Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730712049158

thread_type:
official_review

<thread_object>
creator: reviewer_hyic

<content>
summary:
This paper proposes a distribution matching approach using score-based priors and leverages a distance-preserving distortion loss inspired by Gromov-Wasserstein. The authors present a Score Function Substitution (SFS) trick for efficient training of their score-based prior and evaluate their method on domain adaptation, fairness, and domain translation tasks.

While the paper presents an interesting approach to distribution matching with some promising empirical results, the lack of proper acknowledgment of prior work on Gromov-Wasserstein losses in autoencoders significantly diminishes the claimed novelty. The experimental evaluation would benefit from clearer ablation studies, additional quantitative results for domain translation, and comparisons to relevant competing methods with flexible priors. These additions would help better position the work's contributions relative to the existing literature. Thus, currently the paper does not reach the bar for acceptance.

strengths:
- Novel technical contribution with the Score Function Substitution (SFS) trick
- Empirical evaluation across multiple tasks
- Demonstrates improvements over standard Gaussian prior baselines

weaknesses:
- Missing discussion & contextualization of related work:
	- Employing a Gromov-Wasserstein-inspired loss in an Autoencoder setting has previously been proposed in [1,2,3]. The authors propose to use a distance-preserving distortion loss. This has first been proposed in [1], and then extended in [2,3]. Thus, the method and contribution part of the paper needs to be significantly adapted as the proposed term is not a novel loss but an incorporation of previous existing methods.
- Experimental section:
	- It is unclear whether, in Sections 5.1 and 5.2, the structure-preserving loss was used for all experiments.
	- Comparison to other competing methods. The authors compare their score-based prior to a Gaussian prior. However, competing methods have also proposed to learn a flexible prior, e.g. [1,4,5]. A comparison to other methods leveraging a trainable prior would significantly strengthen the experimental section of the paper.
	- For section 5.4, no quantitative results are reported. Adding quantitative results would validate this section of the experiments as sole qualitative results are hard to judge.

- Minor:
	- Notation: After Eq. 1 it is unclear what d is and what the exact problem setup is here. This is only explained after Eq. 9. I think it would be beneficial to move this to the beginning of Section 2, including a more detailed introduction of the problem statement.
	- State-of-the-art Unpaired Domain Translation methods are not discussed in the related work section [6,7,8,9].


[1] Nao Nakagawa, Ren Togo, Takahiro Ogawa, Miki Haseyama. "Gromov-Wasserstein Autoencoders". ICLR 2023. 

[2] Athina Sotiropoulou, David Alvarez-Melis. "Strongly Isomorphic Neural Optimal Transport Across Incomparable Spaces". GRaM @ ICML 2024.

[3] Uscidda et al. "Disentangled Representation Learning through Geometry Preservation with the Gromov-Monge Gap". SPGIM @ ICML 2024.

[4] Jakub Tomczak and Max Welling. "Vae with a vampprior". AISTATS 2018.

[5] Bin Dai and David Wipf. "Diagnosing and enhancing vae models". ICLR 2019.

[6] Torbunov et al. "UVCGAN v2: An Improved Cycle-Consistent GAN for Unpaired Image-to-Image Translation". arXiv:2303.16280, 2023.

[7] Tong et al. "Improving and generalizing flow-based generative models with minibatch optimal transport". TMLR, 2024.

[8] Eyring et al. "Unbalancedness in Neural Monge Maps Improves Unpaired Domain Translation". ICLR, 2024.

[9] Kim et al. "Unpaired Image-to-Image Translation via Neural Schrödinger Bridge". ICLR, 2024.

questions:
- In Sections 5.1 and 5.2, is the structure-preserving loss used for all experiments? What would be the effect of removing/applying it?
- How important is the structure-preserving loss for the domain translation experiments in Section 5.4?
- How does the work compare to existing trainable prior approaches?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720412972

thread_type:
official_review

<thread_object>
creator: reviewer_kbqr

<content>
summary:
The following paper proposes score-based priors for VAEs in combination with Gromov-Wasserstein distance regularization for the domain adaptation problem.

strengths:
The combination of score-based priors for VAEs with Gromov-Wasserstein distance seems to be a novel approach.

weaknesses:
The contribution of the paper is unclear, making it difficult to follow. The authors frequently shift between topics in introduction section. Initially stating that they are addressing the distribution matching problem. However, they then build their motivation around fairness, robustness, causality, and explainability concepts, before ultimately changing the narrative, and  focus on the domain adaptation problem. 

In addition, the evaluation provided is very poor, the central claims of the paper about "flexibility" are not supported. The evaluation is limited to domain adaptation, and the datasets used for comparison are simple. In addition, the paper proposes the use of structural regularization based on the Gromov-Wasserstein (GW) distance, but fails to provide any evaluation showing that this structural regularization is beneficial. Furthermore, the paper does not consider any datasets where the importance of structural regularization would be relevant. 

Finally, the presentation can be improved. More examples and comparison on the face translation task, improved fonts on MNIST figure, comparison on at least, classic domain adaptation tasks as MNIST->SVHN, MNITS-MNIST-M. GW abbreviation is defined a few times in each section.

questions:
1) Why are "flexible" representations important for trustworthiness, and what is your contribution? How do you demonstrate this flexibility? 

2) Why are score-based priors useful for domain adaptation? 

3) In the introduction, the authors criticize optimal transport methods with Euclidean cost functions. But the comparison with these methods in the domain adaptation task is missing, see papers (1,2,3).

4) Experimental settings in section 5.1 are unclear, why did the authors consider this dataset and not the more popular Celeb-A benchmark? 

5) Why is it important to use structural regularization for the MNIST to USPS dataset? it would be more valuable to consider a typical dataset where Gromov-Wassesrstein regularization is applied. It is important to show how the method performs compared to other domain adaptation methods based on GW (4,5).

**References:**
1) https://openaccess.thecvf.com/content_CVPR_2020/papers/Li_Enhanced_Transport_Distance_for_Unsupervised_Domain_Adaptation_CVPR_2020_paper.pdf
2) http://proceedings.mlr.press/v139/fatras21a/fatras21a.pdf
3) https://proceedings.neurips.cc/paper_files/paper/2020/file/9719a00ed0c5709d80dfef33795dcef3-Paper.pdf
4) https://arxiv.org/pdf/2205.10738
5) https://arxiv.org/pdf/2303.05978
</content>
</thread_object>

</thread>
<thread>
creation date:
1732882663116

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response 3

comment:
## Conclusion

In its initial form, our paper admittedly fell short in several critical aspects, as pointed out by the reviewers. These shortcomings included unclear motivations, insufficient comparisons, and a lack of theoretical depth, particularly in the context of our proposed SFS trick and our Gromov-Wasserstein Semantic Preserving (GW-SP) loss. However, due to the extensive feedback and insightful critiques from the reviewers, we have undertaken a significant revamp of the manuscript. This includes major changes to the storyline, substantial experimental additions, and the incorporation of rigorous theoretical analysis. Most notably, the discovery of the connection between the SFS trick and the LSGM loss—thanks to the reviewers’ comments—has provided deeper insights and theoretical grounding. We believe these improvements make our work a meaningful contribution to the field.

We sincerely thank the reviewers for their invaluable feedback, which has enabled us to significantly improve the quality and clarity of the paper. We kindly request the reviewers to reread our substantially revised manuscript, as we believe it now addresses the raised concerns and offers meaningful contributions to the research community. We are deeply grateful for the reviewers’ guidance throughout this process, which has allowed us to refine our work and strengthen its potential impact.

[1] Roeder, G., Wu, Y., and Duvenaud, D. (2017). Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inference. arXiv preprint, arXiv:1703.09194. Retrieved from https://arxiv.org/abs/1703.09194.

[2] Vahdat, A., Kreis, K., and Kautz, J. (2021). Score-based Generative Modeling in Latent Space. arXiv preprint, arXiv:2106.05931. Retrieved from https://arxiv.org/abs/2106.05931.

[3] Poole, B., Jain, A., Barron, J. T., and Mildenhall, B. (2022). DreamFusion: Text-to-3D using 2D Diffusion. arXiv preprint, arXiv:2209.14988. Retrieved from https://arxiv.org/abs/2209.14988.

[4] Nakagawa, N., Togo, R., Ogawa, T., and Haseyama, M. (2023). Gromov-Wasserstein Autoencoders. arXiv preprint, arXiv:2209.07007. Retrieved from https://arxiv.org/abs/2209.07007.

[5] Uscidda, T., Eyring, L., Roth, K., Theis, F., Akata, Z., and Cuturi, M. (2024). Disentangled Representation Learning with the Gromov-Monge Gap. arXiv preprint, arXiv:2407.07829. Retrieved from https://arxiv.org/abs/2407.07829.

[6] Tomczak, J. M., and Welling, M. (2018). VAE with a VampPrior. arXiv preprint, arXiv:1705.07120. Retrieved from https://arxiv.org/abs/1705.07120.

[7] Kladny, K.-R., von Kügelgen, J., Schölkopf, B., and Muehlebach, M. (2024). Deep Backtracking Counterfactuals for Causally Compliant Explanations. arXiv preprint, arXiv:2310.07665. Retrieved from https://arxiv.org/abs/2310.07665.

[8] Pan, Y., and Bareinboim, E. (2024). Counterfactual Image Editing. arXiv preprint, arXiv:2403.09683. Retrieved from https://arxiv.org/abs/2403.09683.

[9] Komanduri, A., Zhao, C., Chen, F., and Wu, X. (2024). Causal Diffusion Autoencoders: Toward Counterfactual Generation via Diffusion Probabilistic Models. arXiv preprint, arXiv:2404.17735. Retrieved from https://arxiv.org/abs/2404.17735.

[10] Sauer, A., and Geiger, A. (2021). Counterfactual Generative Networks. arXiv preprint, arXiv:2101.06046. Retrieved from https://arxiv.org/abs/2101.06046.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732882696650

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response 2

comment:
## Revisions and Improvements to Experiments

All replaced experiments have been moved to the appendix.

- We acknowledge the valid concerns raised by **Reviewer KBQr**, **Reviewer HyiC**, and **Reviewer 7J8h** regarding the lack of clear evidence supporting the benefits of our learnable, expressive prior. To address this, we conducted synthetic experiments comparing different learnable priors with conventional priors. The results of these experiments are presented in **Section 5.1**, with additional detailed information and results provided in **Appendix H.2** (pages 25–29).  
  **Note:** We would like to apologize for forgetting to highlight the text changes in **Section 5.1**. We want to assure the reviewers that the corresponding changes have been made, and we sincerely regret this oversight. Thank you for your understanding.

- As discussed above, and with the valuable suggestions from **Reviewer woTo** and **Reviewer R3FP** in mind, we have conducted stability experiments comparing LSGM and the SFS trick. Specifically, we measured stability when training on low noise levels, demonstrating that our model is significantly more stable under these conditions. This stability enables more precise modeling of the posterior distribution observed in their negative log-likelihood values, as detailed in **Section 3.2.1**.

- We would like to address the concerns regarding the weaknesses in our domain translation experiment, as highlighted by **Reviewer KBQr** and **Reviewer woTo**. Specifically, these concerns pertain to the effectiveness of the translation when evaluated qualitatively and the lack of supporting quantitative metrics. In response, and as suggested by **Reviewer KBQr**, we have conducted additional experiments on the CelebA dataset, focusing on image translation between black-haired females and blonde-haired females. To provide quantitative support, we utilized image retrieval as our evaluation metric. Qualitatively, we demonstrate that our approach not only preserves the semantic content of the input images but also successfully translates them to the target domain. These improvements are detailed in **Section 5.4**.

- **Reviewer KBQr**, **Reviewer HyiC**, **Reviewer woTo**, and **Reviewer 7J8h** raised important concerns regarding the GW metric and questioned the lack of comparisons between different GW metric configurations—specifically, the absence of GW regularization, GW in the pixel/Euclidean space, and GW in the semantic space. In response, we have reran all experiments with these three configurations to demonstrate the significant improvement in efficacy when using the semantic space.

  We argue that for high-dimensional, real-world image datasets, it is more appropriate to perform comparisons in the semantic space rather than the pixel space. To further clarify the drastic improvements observed in certain tasks (e.g., CelebA) and the more modest improvements in others (e.g., domain adaptation), we present AUROC scores in **Appendix F**. The AUROC scores measure how well a model can separate two classes (e.g., positive and negative) or distinguish between two distributions (e.g., source and target domains). These scores, computed on MNIST, USPS, and CelebA datasets in both the pixel and semantic spaces, provide additional evidence supporting our claim that the semantic space is more appropriate for high-dimensional images.

- We would also like to respectfully point the reviewers to the additional experiments in the appendix that may have been overlooked during the review process. 
  - In **Appendix D**, we showcase the effectiveness of our model in achieving strong domain adaptation results in a limited source label setting. Specifically, our model requires less than 1% of the original source labels to achieve accurate predictions on the target labels. This claim is further supported by UMAP visualizations of the domain adaptation classes, which clearly show well-separated clusters in the latent space. 
  - Furthermore, in **Appendix E**, we present results on domain translation in a multi-domain setting using rotated MNIST datasets with three different angles (0°, 30°, 60°). Our method demonstrates impressive performance in translating between these angles while preserving both stylistic and semantic information. Although this task may appear trivial, it is, in fact, a challenging problem to manipulate MNIST images without altering their semantics, a topic closely related to ongoing research in causal-based image generation \[6, 7, 8, 9, 10\].

  We hope these additional results provide a more comprehensive understanding of our model's capabilities and its potential impact in these areas.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732882804884

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response 1

comment:
First and foremost, we would like to extend our apologies to the reviewers for the delayed submission of our rebuttals. We recognize that our initial submission did not meet the high standards expected by the reviewers, and we take full responsibility for this shortcoming. Our intention was to dedicate as much time as possible to addressing these issues by refining our storyline, deepening our insights, and conducting additional experiments, as we felt our initial submission was inadequate.

We are deeply grateful to all the reviewers for their thoughtful and constructive critiques. Their insights have been instrumental in guiding us toward significant improvements in the quality and clarity of our work. We are especially thankful to the reviewers for pointing us toward relevant literature, enabling us to deepen our understanding of related works. We have carefully reviewed all the suggested papers and humbly acknowledge that we were previously unaware of the paper highlighted by **Reviewer HyiC** and **Reviewer woTo** related to current Gromov-Wasserstein-based methods. In light of this, we have made corresponding changes to the storyline of the paper. This guidance has been invaluable in broadening our perspective and situating our work within the existing body of research. We sincerely appreciate the reviewers’ time and effort in evaluating our submission and for their role in helping us grow through this process.

## Substantial Changes to the Storyline

- Thanks to **Reviewer KBQr** and **Reviewer 7J8h**, we have completely revamped the introduction and overall storyline by adding clearer motivations and decoupling our novelty from past works. We have also made the focus more precise, as our revisions have made the narrative more cohesive and focused.
- We are especially grateful to **Reviewer HyiC** and **Reviewer woTo** for bringing to our attention prior works that have also utilized GW metrics within their frameworks. In response, we have ensured that these works are appropriately credited in our methodology section. Furthermore, we have revised the storyline to better highlight our contributions.
- **Reviewer 7J8h** posed important and deeply critical questions that helped us recognize that some of our wording suggested a level of certainty that was not fully justified by the evidence provided. In response, we have carefully revised our claims, adopting a more measured tone and ensuring all claims are substantiated with appropriate evidence and references to related works.

## Significant Revisions to the Methodology Section

- We would like to extend our special thanks to **Reviewer HyiC** for providing the invaluable insight of connecting the Score Function Substitution (SFS) trick to Latent Score-Based Generative Models (LSGM) \[2\] in a manner resembling the Score Distillation Sampling (SDS) loss \[3\]. Initially, the derivation of the SFS trick appeared relatively straightforward, and its effectiveness was not immediately evident as suggested by **Reviewer R3FP**. However, this insight allowed us to rigorously prove that SFS is, in fact, a distilled version of the LSGM loss. Specifically, by applying the Sticking-the-Landing principle \[1\] to the LSGM objective, we demonstrate that a loss proportional to the SFS trick can be derived (the full proof can be seen in **Appendix E.1**).

  This discovery provides theoretical grounding for the stability of our loss function at lower noise levels—due to the removal of the Jacobian term in LSGM, which serves as a control variate—and highlights practical advantages, such as reduced memory usage. We further corroborate this statement by conducting a stability analysis between LSGM and SDS in **Section 3.2.1**, showcasing the empirical benefits of our approach.

  This finding reveals that the SFS trick is significantly more novel and impactful than we initially understood, and we believe it could pave the way for future advancements in latent score-based/diffusion generative models.

- As importantly highlighted by **Reviewer HyiC** and **Reviewer woTo**, prior works, such as GWAE \[4\], have also adopted similar GW metrics. In response, we have clarified in our methodology section that our work builds upon their GW metric and extends its application by exploring its use in the semantic space for high-dimensional image data. This adjustment ensures proper acknowledgment of prior contributions while emphasizing our novel extensions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733303582131

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Final Remark and Request for Consideration

comment:
We sincerely thank the reviewers for their detailed feedback and the time they have invested in evaluating our work. Their critiques have greatly enhanced the clarity, rigor, and overall quality of our paper, and we deeply respect their thoughtful assessments. However, we would like to humbly request reconsideration of our submission in light of the revisions and responses we have provided.

While some feedback suggested that the storyline of our paper changed significantly, we respectfully clarify that the core storyline remains the same. Our revisions primarily focused on refining and narrowing the narrative to make it more specific and coherent. For instance, the semantic-space Gromov-Wasserstein loss has always been a central component of our work, and our revisions primarily involved acknowledging prior works we were previously unaware of and presenting our loss as an adaptation. This adjustment ensures proper credit while maintaining the novelty and utility of our approach.

Similarly, the deeper insight we gained into the Score Function Substitution (SFS) trick has only strengthened our contribution without altering its essence. Moreover, we augmented our work with additional experiments to substantiate our claims and extensively cited related literature to further validate our methodology and contextualize our contributions.

Regarding concerns about state-of-the-art (SOTA) comparisons, we would like to respectfully emphasize that the primary goal of our work is not to establish new SOTA benchmarks. Instead, our focus is on introducing a very adaptable and general framework for distribution matching with a flexible prior, demonstrating its versatility across diverse downstream tasks. With promising initial performance, we believe this broader contribution provides a strong foundation for future advancements in the field, which could potentially lead to SOTA results in downstream tasks as new techniques and methodologies are developed on top of our framework.

Once again, we are deeply grateful for the reviewers’ dedication and constructive feedback, which have been invaluable throughout this process. We sincerely hope that the reviewers will reconsider our revised submission, as we believe it addresses their concerns and presents meaningful contributions to the research community. Thank you for your time, effort, and commitment to fostering impactful research.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents a distribution matching approach using score-based priors and Gromov-Wasserstein (GW) distance regularization. Reviewers acknowledge the technical soundness and potential contributions of the work. Still, they express major concerns about insufficient acknowledgment of prior work, limited experimental validation (ablation studies, comparisons with existing methods, and quantitative results), and inadequate justification for the benefits of score-based priors and structural preservation. The authors' rebuttal provides additional experiments and expanded comparisons, addressing some of the important concerns. However, reviewers find that the paper requires substantial modifications to integrate the experiments and clarifications presented during the rebuttal. As such, the AC recommends further improvements before the paper can be accepted.