Title: Domain-Indexing Variational Bayes: Interpretable Domain Index for Domain Adaptation
Authors: Zihao Xu
Guang-Yuan Hao
Hao He
Hao Wang
Venue: ICLR 2023

Abstract: Previous studies have shown that leveraging "domain index" can significantly boost domain adaptation performance (Wang et al., 2020; Xu et al., 2022). However, such domain indices are not always available. To address this challenge, we first provide a formal definition of domain index from the probabilistic perspective, and then propose an adversarial variational Bayesian framework that infers domain indices from multi-domain data, thereby providing additional insight on domain relations and improving domain adaptation performance. Our theoretical analysis shows that our adversarial variational Bayesian framework finds the optimal domain index at equilibrium. Empirical results on both synthetic and real data verify that our model can produce interpretable domain indices which enable us to achieve superior performance compared to state-of-the-art domain adaptation methods. Code is available at https://github.com/Wang-ML-Lab/VDI.


Review discussion:
Strengths:
- The paper is well-motivated and its contributions are presented clearly.
- The proposed method is technically sound and elegant, although some relevant details seem to be missing (see 'Weaknesses' below).
- The methodology is well grounded by the presented theoretical results.
- The experiments confirm the effectiveness of the method.
In particular, the visualizations of the learned global domain indexes in Figures 5 and 6 are quite interesting.
Weaknesses:
1. As is apparent from section 3.1, the paper considers the unsupervised multi-source, multi-target DA problem.
However, it is not entirely clear which terms in the loss (eq. (14)) are active for unlabeled target examples. I suppose it should be all the terms of the ELBO except for eq. (5), plus the domain discriminator loss, but this should be clarified.
From a theoretical perspective, I think this would correspond to the maximization of the ELBO for $p(x)$ plus domain discriminator loss.
2. In the proof of Lemma A.1, the authors claim that "since $z_1 \perp \beta$ and $z_2 \perp \beta$, then $(z_1, z_2) \perp \beta$", but this is not true in general. E.g. suppose $z_1$ and $z_2$ are Bernoulli r.v.'s and $\beta = z_1  \mathrm{xor}  z_2$. The authors should explain why the implication holds in their case or otherwise review the proof and reformulate the remainder of the theoretical analysis in case the Lemma cannot be proved.
3. Architecture and implementation details are missing from the experimental section. This should be provided as supplementary material. Releasing the source code would also be beneficial for reproducibility.
The paper is clear and very well written. To the best of my knowledge, this is the first paper proposing to learn continuous domain indexes for DA directly from data. As mentioned before, some details are missing for reproducibility.
NO.
Thank you for your constructive and encouraging comments as well as the insightful questions. We are glad that you find our paper ``"well-motivated"``, our contributions ``"presented clearly"``/``"solid"``, our methodology ``"well grounded by the presented theoretical results"``/``"elegant"``, and our experiments ``"interesting"``. Below we address your questions in detail one by one.
**Q1. As is apparent from section 3.1, the paper considers the unsupervised multi-source, multi-target DA problem.
However, it is not entirely clear which terms in the loss (eq. (14)) are active for unlabeled target examples. I suppose it should be all the terms of the ELBO except for eq. (5), plus the domain discriminator loss, but this should be clarified.
From a theoretical perspective, I think this would correspond to the maximization of the ELBO for p(x) plus domain discriminator loss.**
Yes, you are correct. We follow the typical domain adaptation setting to use labeled data $(x, y)$ from source domains and unlabeled data $x$ from target domains during training. For source-domain data, we use the full ELBO, i.e., Eq (4-7), plus the discriminator loss; for target-domain data, we use Eq (4), (6), and (7) plus the discriminator loss. We have revised the paper accordingly as suggested to make this clear.
**Q2. In the proof of Lemma A.1, the authors claim that " since z1⊥β and z2⊥β, then (z1,z2)⊥β", but this is not true in general. E.g. suppose z1 and z2 are Bernoulli r.v.'s and β=z1 xor z2. The authors should explain why the implication holds in their case or otherwise review the proof and reformulate the remainder of the theoretical analysis in case the Lemma cannot be proved.**
Thanks for mentioning this and we apologize for the confusion.
Indeed, for $(z_1, z_2)⊥\beta$ to hold, one needs an additional assumption $z_1 ⊥ z_2 | \beta$. We therefore revised Lemma A.1 and provided proof that does not need such an assumption.
Note that (and as you correctly pointed out) the revision on Lemma A.1 does not affect any conclusions or claims in the main paper. Lemma A.1 is only meant for a more detailed characterization of the (optimal) domain index defined in Definition A.2.
**Q3. Architecture and implementation details are missing from the experimental section. This should be provided as supplementary material. Releasing the source code would also be beneficial for reproducibility.**
Thanks for your suggestion. We have added the architecture and implementation details in the Appendix as suggested (Sec. E). We have finished cleaning up the source code and will release it after the paper is accepted to facilitate further research in the community.
- Strength
1. The paper is well-written and clearly presented.
2. The proposed method is well-motivated and is theoretically guaranteed.
3. The idea of automatically inferring the domain index is interesting.
- Weakness
1. Some of the statements should be further justified.
For example, why is leveraging the domain index better than domain identity?
Though previous work can somewhat support the statement, it would be better to briefly but explicitly justify it again here.
2. The domain index is formally defined in section 3.2.
However, the definitions and differences between the local index and the global index are not quite clear.
For example, the global index \beta is required to be independent of z, but this is not required for the local index, and why?
In addition, the definition does not show the global index is responsible for indicating different domains.
Moreover, why we explicitly need both the global index and local index is also not well justified.
3. The global domain index is inferred based on the local index. Is it necessary to introduce the additional local index? Is it possible to infer the global domain index directly? More analysis and experiments are encouraged for supporting this.
The paper is well-written and clearly presented. The problem addressed in this paper is novel and reproducible.
NO.
No
**Q2.2.
For example, the global index $\beta$ is required to be independent of z, but this is not required for the local index, and why?**
**Independence between $z$ and the Global Index $\beta$.** In typical domain adaptation methods, an encoder first takes $x$ as input to produce the representation $z$, and a predictor then takes $z$ as input to predict the label. To improve $z$'s *generalization across different domains*, it is common practice [1, 2, 3] to enforce independence between the domain index $\beta$ and representation $z$ such that *domain-specific information is removed* from $z$.
Therefore the assumption/constraint of independence between $\beta$ and $z$ is natural.
**Why Independence between $z$ and the Local Index $u$ is Not Required.** As mentioned in the response to the previous question **Necessity of EMD and MDS.** and our newly added Fig. 8 of Appendix D (where $u$'s in Domain 11 contain two clusters, with one cluster corresponding to positive data and the other corresponding to negative data), we need the local indices $u$ to capture the multi-cluster structure in the data, with each cluster corresponding to local indices for data with the same label $y$. Since $z$ contains label-specific information, if we enforce independence between $z$ and $u$, such label-specific information will be removed from $u$, making it impossible for $u$ to capture the label-specific multi-cluster structure in the data.
We have included the discussion above in the revised paper (Sec. 3.4 and Sec. K in the Appendix).
[1] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Franccois
Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. JMLR 2016.
[2] Hao Wang, Hao He, and Dina Katabi. Continuously indexed domain adaptation. ICML 2020.
[3] Zihao Xu, Guang-He Lee, Yuyang Wang, Hao Wang. Graph-relational domain adaptation. ICLR 2022.
**Q3. The global domain index is inferred based on the local index. Is it necessary to introduce the additional local index? Is it possible to infer the global domain index directly? More analysis and experiments are encouraged for supporting this.**
This is a good question and is highly related to our response to Q2.1.
Local domain indices are necessary for VDI to work. Please kindly refer to the paragraphs **Global versus Local Domain Indices**, **Necessity of Local Domain Indices**, and **Necessity of EMD and MDS** in Q2.1's response above.
**Q2.1. The domain index is formally defined in section 3.2.
However, the definitions and differences between the local index and the global index are not quite clear...In addition, the definition does not show the global index is responsible for indicating different domains.
Moreover, why we explicitly need both the global index and local index is also not well justified.**
This is a good question. Local domain indices are necessary for VDI to work. Below we briefly clarify the difference between global and local domain indices and then discuss the necessity of local indices.
**Global versus Local Domain Indices.** Local domain indices $u$ contain instance-level information, i.e., each data point has a different local domain index $u$;
in contrast, global domain indices contain domain-level information, i.e., all data points of the same domain share the same global domain index $\beta$.
**Necessity of Local Domain Indices.** (1) Essentially, local domain indices $u$ serve as extremely low-dimensional summaries of representations $z$;
for example, in CompCars experiments the numbers of dimensions for $u$ and $z$ are $4$ and $4096$, respectively. With such low-dimensional $u$, global domain indices can be efficiently inferred from $u$ in the $4$-dimensional space using Eq. (8-11).
In contrast, if one only uses global indices without $u$, one would need to compute the Earth Mover's distance (EMD) and multi-dimensional scaling (MDS) in Eq. (8-11) in the $4096$-dimensional space; this is much more computationally expensive, susceptible to overfitting, and dramatically slows down model training.
Therefore local domain indices are necessary in VDI. (2) Note that using a smaller number of dimensions for $z$ (e.g., $4$) is not a good choice either, since the predictor takes $z$ as input to predict the label, and using lower-dimensional $z$ harms prediction performance.
**Necessity of EMD and MDS.** It is also worth noting that EMD and MDS are also necessary when inferring global domain indices $\beta$ from local domain indices $u$. The main reason is that $u$'s distribution tends to be *multi-modal*. As an example, we plot the local indices from $3$ domains in *DG-15* in Fig. 8 of Appendix D. We can see that the $u$'s in Domain 11 contain two clusters (i.e. two modes), with one cluster corresponding to positive data and the other corresponding to negative data. This is also the case for both Domain 1 and 0, but the distance between the two clusters gets smaller and smaller.
Therefore, directly using the mean of $u$'s as the global domain index $\beta$ does not work because all three domains will have similar means, and consequently similar $\beta$'s. Using EMD and MDS fixes this issue since EMD can naturally compute the distance between two multi-modal distributions. Our preliminary experiments also confirm their necessity; removing EMD and MDS will significantly bring down VDI's performance.
**How the Global Index $\beta$ Indicates Different Domains from the Definition.** One key property that makes sure the global index $\beta$ can distinguish different domains is the second point of Definition 3.1, i.e., *information preservation of $x$*.
Specifically, maximizing the mutual information $I(x; u, \beta, z)$ ensures that $\beta$ contains as much information on $x$ as possible (under the constraint that $\beta$ and $z$ is independent); since different domains usually have different distributions over $x$, *information preservation of $x$* helps $\beta$ to distinguish (indicate) different domains. This is also empirically verified by results in Figure 3(d), Figure 5, and Figure 6, where VDI successfully inferred meaningful domain indices $\beta$ of different domains in various datasets.
Besides, we also provided an additional figure (Figure 7) of the Appendix to show that the learned global indices $\beta$ highly correlated with the ground-truth indices in the *Circle* dataset.
We have included the discussion above in the revision (Sec. 3.4 and Sec. K in the Appendix).
Thank you for your constructive and encouraging comments as well as the insightful questions. We are glad that you find our paper ``"well-written and clearly presented"``, our method ``"well-motivated and theoretically guaranteed"``, and the idea of automatically inferring the domain index ``"interesting"``/``"novel"``. Below we address your questions in detail one by one.
**Q1. Some of the statements should be further justified.
For example, why is leveraging the domain index better than domain identity?
Though previous work can somewhat support the statement, it would be better to briefly but explicitly justify it again here.**
This is a good suggestion.
**(1)** Compared to domain identities, domain indices can better capture the similarity and complex relations among different domains, thereby better guiding the adaptation process. This is also empirically verified in previous works such as [1, 2]. As a simple example, note that domain identities are one-hot vectors, the distance between any two domain identities is identical;
in contrast, domain indices are real-value vectors and therefore contain richer information.
**(2)** More specifically, since the encoder takes domain indices as additional input, domain indices tend to encourage data $x$ of similar domains to go through similar transformations to produce the encoding $z$;
consequently, the predicted decision boundaries of similar domains will also be similar.
**(3)** Theoretically, the target domain's error is upper-bounded by three terms, i.e., source error, domain gap, and optimal joint error of the source and target domains [3]. Encouraging similar transformations for similar domains can reduce the joint error term of the upper bound, thereby achieving better performance.
We have added the discussion above in the revised paper (in both the main paper and Sec. J of the Appendix) as suggested.
[1] Hao Wang, Hao He, and Dina Katabi. Continuously indexed domain adaptation. ICML 2020.
[2] Zihao Xu, Guang-He Lee, Yuyang Wang, Hao Wang, et al. Graph-relational domain adaptation. ICLR 2022.
[3] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman
Vaughan. A theory of learning from different domains. Machine Learning, 79(1):151–175, 2010.
*Strength
- The authors provide a formal definition of domain index from the probabilistic perspective.
- Their theoretical analysis justifies the proposed method.
*Weaknesses
- The assumption of the independence of beta from representation z is not justified well.
- The introduction of the hierarchical modeling of global and local domains is not well supported and motivated. Is there any problem when we only use the global one?
- The paragraph "Inferring Global Domain Indices" in Section 3.4 is hard to follow, and it is a different formulation from Eq.2, which is confusing. Is it impossible to use ELBO simply?
*Clarity
- The order of input variables is inconsistent, such as in Definition 3.1, which is confusing.
- The formats of Eqs.4-7 have problem.
- The description for "(4) Regularization Terms for All Latent Variables" is hard to follow. What kind of regularization is it?
- In the description of the data CompCars, there is no description on what part is source or target.
- It is unclear the proposed method is trained also on the target domain. If so, how do the authors get labels for the target domain?
- It is better to provide some intuition that why the proposed method performed so well in Table 1 since there is a large margin from baselines.
- In the experiment on W->E Level 1, why Source-Only performed the best?
- In figure 6, there is no desctiption for the numbers on points.
*Quality
- Please see the above comments.
*Novelty
- The proposed method seems to be novel.
*Reproducibility
- Code is not available, but the authors provide enough information in the main text.
NO.
**Q6. The description for "(4) Regularization Terms for All Latent Variables" is hard to follow. What kind of regularization is it?**
Eq. 7 includes two KL divergence terms and one entropy term, all of which serve as regularizers to prevent these approximate posteriors of latent variables $u$, $\beta$, and $z$ from overfitting.
For example, the first term implies that the approximate posterior distribution $q(\beta|u)$ (which VDI learns) should be close to the prior distribution $p(\beta)$ (which is set to a standard Gaussian distribution $N(0,I)$).
Therefore it is a regularization term for $q(\beta|u)$.
Similarly, the second term is also a KL divergence term and therefore ensures that $q_{\phi}(z|x,u, \beta)$ is close to $p_{\theta}(z|x,u, \beta)$.
To better see why KL divergence has a regularization effect, consider the KL divergence $D_{KL}(p||q)$ between two Gaussian distributions $N(\mu_p, \Sigma_p)$ and $N(\mu_q, \Sigma_q)$. We have $D_{KL}(p||q) = \frac{1}{2}[\log\frac{|\Sigma_q|}{|\Sigma_p|} - k + (\mu_p-\mu_q)^T\Sigma_q^{-1}(\mu_p-\mu_q) + tr(\Sigma_q^{-1}\Sigma_p)]$. We can see that minimizing $D_{KL}(p||q)$ tries to bring $\mu_p$ closer to $\mu_q$ and $\Sigma_p$ closer to $\Sigma_q$. Such KL divergence terms often appear in VAE-based models [1].
The last term $-E_{q_{\phi}(u|x)}[\log q_{\phi}(u|x)]$ is the entropy of the approximate posterior distribution $q_{\phi}(u|x)$ (which VDI learns). Note that a Gaussian distribution $N(\mu, \sigma^2)$'s entropy is $\frac{1}{2}ln(2\pi \sigma^2)+\frac{1}{2}$. We can see that maximizing the Gaussian distribution $q_{\phi}(u|x)$'s entropy as a regularization term helps prevent the Gaussian distribution from collapsing to zero variance (i.e., $\sigma^2=0$).
We have included additional clarification in Sec. 3.4 as suggested.
[1] Diederik P Kingma, Max Welling. Auto-encoding variational Bayes. ICLR 2014.
**Q7. In the description of the data CompCars, there is no description on what part is source or target.**
We apologize for missing this detail. We choose the domain with front view (the most common view) and years of manufacture (YOM) 2009 (the earliest YOM) as the source domain and all the others as target domains. We have also revised the paper accordingly (marked in blue).
**Q8. It is unclear the proposed method is trained also on the target domain. If so, how do the authors get labels for the target domain?**
Sorry for the confusion. We follow the typical domain adaptation setting to use labeled data $(x, y)$ from source domains and unlabeled data $x$ from target domains during training. For source-domain data, we use the full ELBO, i.e., Eq (4-7); for target-domain data, we use Eq (4), (6), and (7). We have revised the paper accordingly as suggested to make this clearer.
**Q9. It is better to provide some intuition that why the proposed method performed so well in Table 1 since there is a large margin from baselines.**
This is a good suggestion. Previous work has shown that including domain indices significantly boosts domain adaptation performance [1, 2]; this is because domain indices can better capture the similarity and complex relations among different domains, thereby better guiding the adaptation process.
However, domain indices are often unavailable in practice. Our VDI can infer reasonable domain indices (see Figure 3(d), Figure 5, Figure 6, and Figure 7 in the Appendix) from data in an unsupervised manner, and then leverage such inferred domain indices to significantly improve domain adaptation performance. We have added more discussion in Sec. 5.3 of the revised version as suggested.
[1] Hao Wang, Hao He, and Dina Katabi. Continuously indexed domain adaptation. ICML 2020.
[2] Zihao Xu, Guang-He Lee, Yuyang Wang, Hao Wang, et al. Graph-relational domain adaptation. ICLR 2022.
**Q10. In the experiment on W->E Level 1, why Source-Only performed the best?**
Good question. As shown in Figure 4, W->E Level 1 contains the target domains that are the closest to the source domains.
Therefore it is relatively easy to adapt from the six source domains on the west coast to these Level-1 target domains. Similar results could be observed in the N->S setting of Table 2, where Source-Only is the strongest baseline for N->S Level-1 domains.
**Q11. In figure 6, there is no description for the numbers on points.**
We are sorry for the confusion. The CompCars dataset consists of 30 domains and these numbers (from $0$ to $29$) on the points in Figure 6 indicate their domain identities (as discrete values). We have included more details in Figure 6 in the revision (marked in blue).
**Q3. The paragraph "Inferring Global Domain Indices" in Section 3.4 is hard to follow, and it is a different formulation from Eq.2, which is confusing. Is it impossible to use ELBO simply?**
We are sorry for the confusion. the paragraph "Inferring Global Domain Indices" is actually consistent with Eq. 2.
Specifically, Eq. 2 factorizes our variational distribution $q_{\phi}(u,z,\beta|x)$ into three terms, $q_{\phi}(u|x)$, $q_{\phi}(\beta|u)$ and $q_{\phi}(z|x, u,\beta)$.
While the terms $q_{\phi}(u|x)$ and $q_{\phi}(z|x, u,\beta)$ are directly parameterized by two neural networks, the remaining term $q_{\phi}(\beta|u)$ is more complicated; this is why we dedicate a separate paragraph, "Inferring Global Domain Indices" in Section 3.4, to describe the details on $q_{\phi}(\beta|u)$.
As mentioned in the paragraph, to aggregate local domain indices $u$ to obtain the global domain indices $\beta$, we need to
**(1)** group local domain indices $u_i$ in each domain $k$ (each domain contains multiple local domain indices),
**(2)** compute pairwise domain distance using the Earth Mover's distance (EMD), where the function $f_{EMD}$ takes as input two sets of $u$'s from two domains and returns their EMD,
**(3)** compute the raw global domain indices using multi-dimensional scaling (MDS), where MDS takes as input the $N$-by-$N$ pairwise domain distance matrix ($N$ is the number of domains) and outputs the raw global domain indices $\beta_k^r$ for each domain $k$, and
**(4)** use a neural network to take $\beta_k^r$ as input and output the final global domain indices $\beta_k$ for each domain.
We can see that $q_{\phi}(\beta|u)$ is much more complicated than directly using a neural network and therefore need a separate description in the paragraph. Our preliminary experiments also confirm the necessity of such complexity; removing EMD and MDS will bring down VDI's performance. Please also refer to the response on **Necessity of EMD and MDS** in the previous question on why such additional complexity is needed. We have included additional clarification in Sec. 3.3 and Sec. 3.4 in the revision as suggested.
**Q4. The order of input variables is inconsistent, such as in Definition 3.1, which is confusing.**
Thanks for mentioning this. We have adjusted the order of input variables such that it is consistent throughout the paper as suggested (for better readability, we did not mark the equations involved in blue).
**Q5. The formats of Eqs.4-7 have problems.**
In Eqs. 4-7, we intentionally group the terms with different meanings in different lines of the long equation (see Eq. 15 on Page 28 of https://www.jmlr.org/papers/volume3/blei03a/blei03a.pdf for a similar format). These four parts of the equation are then explained in the four bullet points below.
Note that Eq. 7 contains three terms with similar meanings; for clarity and convenience when explaining them in Item (4) of the paragraph below Eq. 7, we therefore follow the common practice to group them into the same line with a slightly smaller font size. We are happy to split Eq. 7 into two lines if the reviewer thinks it would be more readable.
Thank you for the constructive and encouraging comments as well as the insightful questions. We are glad that you find our proposed method ``"novel"``, our theoretical analysis ``"solid"``, that we ``"provide a formal definition of domain index from the probabilistic perspective"``, that ``"the proposed method performs much better than existing methods"``, and that our ``"theoretical analysis justifies the proposed method"``. Below we address your questions one by one.
**Q1. The assumption of the independence of $\beta$ from representation z is not justified well.**
In typical domain adaptation methods, an encoder first takes $x$ as input to produce the representation $z$, and a predictor then takes $z$ as input to predict the label. To improve $z$'s *generalization across different domains*, it is common practice [1, 2, 3] to enforce independence between the domain index $\beta$ and representation $z$ such that *domain-specific information is removed* from $z$.
Therefore the assumption/constraint of independence between $\beta$ and $z$ is natural. We have included the discussion above in the revised paper (Sec. J in the Appendix) as suggested.
[1] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Franccois
Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. JMLR 2016.
[2] Hao Wang, Hao He, and Dina Katabi. Continuously indexed domain adaptation. ICML 2020.
[3] Zihao Xu, Guang-He Lee, Yuyang Wang, Hao Wang. Graph-relational domain adaptation. ICLR 2022.
**Q2. The introduction of the hierarchical modeling of global and local domains is not well supported and motivated. Is there any problem when we only use the global one?**
This is a good question. Local domain indices are necessary for VDI to work. Below we briefly clarify the difference between global and local domain indices and then discuss the necessity of local indices.
**Global versus Local Domain Indices.** Local domain indices $u$ contain instance-level information, i.e., each data point has a different local domain index $u$;
in contrast, global domain indices contain domain-level information, i.e., all data points of the same domain share the same global domain index $\beta$.
**Necessity of Local Domain Indices.** (1) Essentially, local domain indices $u$ serve as extremely low-dimensional summaries of representations $z$;
for example, in CompCars experiments the numbers of dimensions for $u$ and $z$ are $4$ and $4096$, respectively. With such low-dimensional $u$, global domain indices can be efficiently inferred from $u$ in the $4$-dimensional space using Eq. (8-11).
In contrast, if one only uses global indices without $u$, one would need to compute the Earth Mover's distance (EMD) and multi-dimensional scaling (MDS) in Eq. (8-11) in the $4096$-dimensional space; this is much more computationally expensive, susceptible to overfitting, and dramatically slows down model training.
Therefore local domain indices are necessary in VDI. (2) Note that using a smaller number of dimensions for $z$ (e.g., $4$) is not a good choice either, since the predictor takes $z$ as input to predict the label, and using lower-dimensional $z$ harms prediction performance.
**Necessity of EMD and MDS.** It is also worth noting that EMD and MDS are also necessary when inferring global domain indices $\beta$ from local domain indices $u$. The main reason is that $u$'s distribution tends to be *multi-modal*. As an example, we plot the local indices from $3$ domains in *DG-15* in Fig. 8 of Appendix D. We can see that the $u$'s in Domain 11 contain two clusters (i.e. two modes), with one cluster corresponding to positive data and the other corresponding to negative data. This is also the case for both Domain 1 and 0, but the distance between the two clusters gets smaller and smaller.
Therefore, directly using the mean of $u$'s as the global domain index $\beta$ does not work because all three domains will have similar means, and consequently similar $\beta$'s. Using EMD and MDS fixes this issue since EMD can naturally compute the distance between two multi-modal distributions. Our preliminary experiments also confirm their necessity; removing EMD and MDS will significantly bring down VDI's performance.
We have included the discussion above in the revision (Sec. 3.4 and Sec. K of the Appendix).
Thank you for the effort to address my concerns. I upgraded my score as in the review (see "update" part).
Strengths.
1.	I think that the problem the authors try to solve is a new (domain adaptation learning with domain indices) and interesting (inference of unknown domain indices) problem.
2.	I think that the manuscript is well written and organized.
Although I did not check all the detailed derivations, the theoretical analysis well supports the proposed method.
Weaknesses
1.	The manuscript does not include the analysis of the computational complexities of the proposed method.
2.
Regarding the experiments. I think it would be better to include a table summarizing the data sets, e.g., the input dimension, the number of samples. I also think that it would be better if the previous work, Wang et al. (2020) Xu et al. (2022), had been included in the comparison as all the datasets used in the experiments already include the domain indices. With the same reason, I think that the experiments did not really explain well about what advantages of using the proposed method in practice would be.
Clarity & Quality
I think that the manuscript is well written. I have a minor comment. In Table 2, SENTRY is the best performer for the case “N(24)->S(24): Average of 8 Level-3 domains”.
Novelty & reproducibility
Please see “Strengths and Weaknesses “
NO.
Thank you for the constructive and encouraging comments. We are glad that you find our research problem ``"new and interesting"``,  our manuscript ``"well written and organized"``, and our theoretical analysis ``"well supports the proposed method"``. Below we address your questions one by one.
**Q1. The manuscript does not include the analysis of the computational complexities of the proposed method.**
The computational complexity of our model is similar to most deep learning models, and therefore our VDI is scalable. The overall complexity is $O(TMP+TN^3)$ where $T$, $M$, $P$, and $N$ are the number of epochs, the number of training data points, and the number of parameters in the neural networks, and the number of domains, respectively (note that $N$ is usually very small). Here $O(TMP)$ is the complexity for optimizing neural network parameters, and $O(TN^3)$ is the complexity for inferring global domain indices, i.e., Eq. (8-11).
Note that VDI's complexity is similar to most deep learning models because it is dominated by the first term $TMP$. To see this, observe that (1) A neural network usually has a large number of parameters $P$ (e.g., in the scale of millions) (2) $N^3$ is much smaller than $MP$ in practice (the number of domains $N$ is the scale of tens in practice).
Therefore overall, our VDI scales linearly with $T$, $M$, and $P$. Empirically we found that our VDI's training time is similar to that of other methods such as DANN, MDD, and SENTRY.
**Q2. (1) Regarding the experiments. I think it would be better to include a table summarizing the data sets, e.g., the input dimension, the number of samples.**
Thank you for your suggestion. Following your suggestion, we compile the table summarizing dataset statistics and settings as follows. We have also included the table in Table 4 of the Appendix as suggested.
| Dataset  | Number of Samples | Input Dim        | Toy/Real | Adaptation Task      |
|:--------:|:-----------------:|:----------------:|:--------:|:--------------------:|
| Circle   | 3,000             | 2                | Toy      | 2-Way Classification |
| DG-15    | 1,500             | 2                | Toy      | 2-Way Classification |
| DG-60    | 6,000             | 2                | Toy      | 2-Way Classification |
| TPT-48   | 6,912             | 6                | Real     | Regression           |
| CompCars | 18,735            | 224 $\times$ 224 | Real     | 4-Way Classification |
| | | |
**Q2. (2) I also think that it would be better if the previous work, Wang et al. (2020) Xu et al. (2022), had been included in the comparison as all the datasets used in the experiments already include the domain indices.**
This is a good question.
**(a)** To evaluate the effectiveness of our VDI, we need to compare VDI's inferred domain indices against domain adaptation datasets that *include ground-truth domain indices*; this makes the datasets of Wang et al. (2020) and Xu et al. (2022) our perfect choice.
**(b)** Wang et al. (2020) and Xu et al. (2022) are not applicable to our setting of inferring domain indices, and direct comparison with our VDI is unfair since Wang et al. (2020) / Xu et al. (2022) and our VDI focus on *different settings* (i.e., the former uses additional information).
Specifically, our VDI assumes that domain indices are *not available* and tries to infer these indices, while Wang et al. (2020) and Xu et al. (2022) assume that ground-truth domain indices are *readily available* and use them as additional input to improve performance.
**(c)** It is also worth noting that our VDI is actually complementary to Wang et al. (2020) and Xu et al. (2022) in the case of unknown domain indices, where one could infer domain indices from data and use them as input for Wang et al. (2020) and Xu et al. (2022). This would indeed be interesting future work.
**Q2. (3)  With the same reason, I think that the experiments did not really explain well about what advantages of using the proposed method in practice would be.**
Thanks for mentioning this. We should have emphasized that in practice, domain indices are usually *unknown* (e.g., datasets such as Office-Home and DomainNet); previous domain adaptation methods can only rely on *domain identity* (which contains much less information than domain indices) to align data from different domains and therefore tend to suffer in terms of accuracy.
In contrast, our VDI can infer domain indices from data and use these domain indices to improve domain adaptation performance, as shown in Table 2 and 3.
Besides, it is also worth noting that VDI's inferred domain indices can also provide practitioners with meaningful interpretations of the domain adaptation model (see Figure 3(d), Figure 5, and Figure 6).
**Q3. I have a minor comment. In Table 2, SENTRY is the best performer for the case “N(24)->S(24): Average of 8 Level-3 domains”.**
We apologize for the typo and have updated Table 2 accordingly.
We thank all reviewers for their valuable comments. We are glad that they found the problem we solve ``"new"``/``"interesting"``/``"novel"``/``"well-motivated"`` (eAJo, SAsR, YbH3), our method ``"interesting"``/``"novel"``/``"elegant"`` (eAJo, uSSJ, SAsR, YbH3) and ``"theoretically guaranteed"``/``"well grounded by theoretical results"`` (SAsR, YbH3), our theoretical analysis ``"solid"``/``"well supports the proposed method"`` (uSSJ, eAJo), our experiments ``"demonstrate that the proposed method performs much better than existing methods"``/``"confirm the effectiveness of the method"``/``"interesting"`` (uSSJ, YbH3), and our paper ``"well written"``/``"organized"``/``"presented clearly"`` (eAJo, SAsR, YbH3). Below we address the reviewers’ questions. We have also updated both the main paper and the Appendix (with the changed part marked in blue).

Meta-review:
role: PC
data: metareview:_summary,_strengths_and_weaknesses: Thanks for your submission to ICLR.  After the rebuttal and discussion phase, all four reviewers were in agreement that this paper is suitable for publication.  The noted weaknesses are mostly minor and many have already been fixed (clarity in the text, additional justifications required).  Strengths included the novelty of the approach, the writing of the paper, and the theoretical analysis to justify the method.
justification_for_why_not_higher_score: This paper could potentially be bumped up to an oral.  
justification_for_why_not_lower_score: With three overwhelmingly positive reviewers, this is clearly a strong paper.  Also, the topic is important and of relevance to many, so giving the paper at least a spotlight seems justified.

Decision: Accept: notable-top-25%