Bayesian Domain Adaptation with Gaussian Mixture Domain-Indexing

Authors: yanfang  ling
jiyong  li
lingbo  li
shangsong  liang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 16:34:08

Abstract: Recent methods are proposed to improve  performance of domain adaptation by inferring domain index under an adversarial variational bayesian framework, where domain index is unavailable. 
However, existing methods typically assume that the global domain indices are sampled from a vanilla gaussian prior, overlooking the inherent structures among different domains.
To address this challenge, we propose a Bayesian Domain Adaptation with Gaussian Mixture Domain-Indexing(GMDI) algorithm. 
GMDI employs a Gaussian Mixture Model for domain indices, with the number of component distributions in the ``domain-themes'' space adaptively determined by a Chinese Restaurant Process. 
By dynamically adjusting the mixtures at the domain indices level, GMDI significantly improves domain adaptation performance. 
Our theoretical analysis demonstrates that GMDI achieves a more stringent evidence lower bound, closer to the log-likelihood. 
For classification, GMDI outperforms all approaches, and surpasses the state-of-the-art method, VDI, by up to 3.4%, reaching 99.3%. 
For regression, GMDI reduces MSE by up to 21% (from 3.160 to 2.493), achieving the lowest errors among all methods.

Main paper: 
 introduction: 
 Machine learning models often suffer from performance degradation when applied to new domains that differ from their training domains, a phenomenon known as domain shift [21, 8, 28, 15] . Domain Adaptation (DA) [4, 44, 49, 42, 6, 1, 37, 32, 38, 13] seeks to mitigate this issue by producing domaininvariant features, thereby enhancing generalization from source to target domains [23, 19, 12, 39, 45] .Recent research has explored the use of domain identity and domain index to improve domaininvariant data encoding and enhance domain adaptation performance [36, 40, 41] . Domain identity [41] , a one-hot discrete variable vector, differentiates between domains, whereas domain index [41] , a real-valued continuous variable vector, captures domain semantics. Due to the limited information in the discrete domain identity vector, research has increasingly focused on the domain index. Current approaches to incorporating domain index in domain adaptation include: (1) Directly using existing additional information in the dataset as the domain index [36, 40] , which is impractical for datasets lacking such indices [22, 29] , and (2) Treating the domain index as a latent variable to be inferred [26, 41] . However, these methods typically model the domain indices with a simple Gaussian distribution, limiting the domain indices space and thus hindering adaptation to diverse target domains, resulting in suboptimal performance. To address the aforementioned issues, we propose a Bayesian Domain Adaptation with Gaussian Mixture Domain-Indexing (GMDI) algorithm. The proposed adversarial Bayesian algorithm assumes that domain indices follow a mixture of Gaussian distributions, with the number of mixture components dynamically determined by a Chinese Restaurant Process. As shown in Figure 1 , a single Gaussian distribution struggles to adequately fit the domain indices, neglecting the inherent structures among different domains. This observation motivates us to model domain indices from different domains collectively as a Gaussian mixture distribution. To the best of our knowledge, we are the first to model domain indices as a mixture of Gaussian distributions to address the aforementioned challenges. Inspired by [3] , the latent space of the mixture is defined as the "domain-themes" space. The mixtures of distributions provide a higher level of flexibility in a larger latent space, thereby increasing the capability to adapt to various target domains with domain shift. Our theoretical analysis demonstrates that GMDI achieves a more rigorous evidence lower bound, and that maximizing this bound along with adversarial loss effectively infers optimal domain indices. Extensive experimental results validate the significant effectiveness of GMDI.Our key contributions are summarized as: (1) Our proposed GMDI is the first one to consider the entire distributions of domain indices in the "domain-themes" space following a mixture of Gaussian distributions, and dynamically determining the number of components in the mixture with the Chinese Restaurant Process. (2) Our detailed theoretical analysis demonstrates that training with GMDI's superior evidence lower bound together with adversarial loss can yield optimal and more interpretable domain indices. (3) Extensive experiments on classification and regression tasks showcase the strong domain index modeling capability of GMDI, significantly outperforming the state-of-the-art.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: See Abstract and Section 1.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Adversarial domain adaptation. There exists a substantial body of work on domain adaptation [4, 44, 49, 42, 6, 1, 37, 32, 38] . They focus on generating domain-invariant data encoding by aligning the distributions of source and target domains to adapt to target domains. This alignment is achieved by directly matching the statistics of distributions [25, 24] or by employing adversarial loss [33, 31] , which encourages domain confusion through adversarial objective with a discriminator. Adversarial domain adaptation is widely used due to its integration with deep learning, strong theoretical foundation [7] , and superior performance. Various different types of adversarial losses have been explored: [35] uses an inverted label GAN loss, [5] utilizes a minimax loss, and [34] employs a cross-entropy loss against the uniform distribution. Typically, the discriminators in these models rely on domain identity, which contains limited information, to align data encoding distributions. [20] and [10] also pay attention to domain identity. Our work, however, focuses on domain index, providing a more detailed representation of domains. Domain adaptation related to domain indices. Recently, there has been growing interest in using continuous domain index, which contain richer and more interpretable information, to enhance domain adaptation performance. [36] use the rotation angle of images as the domain index for the Rotating MNIST dataset and patients' ages as the domain index for Healthcare Datasets. Their theoretical analysis demonstrates the value of utilizing domain indices to generate domain-invariant features. [40] employ graph node embeddings as domain indices to achieve domain adaptation in graph-relational domains. These methods assume that domain indices are available. However, in practice, domain indices are not always accessible [22, 29] . [26] generates features representing the similarity between different domains but do not formally define the domain index. [41] formally define the domain index and treat it as a latent variable to be inferred. Although [41] takes steps towards Bayesian approximation to parameter distributions, it only assumes a single domain index distribution, limiting its capability to adapt to diverse target domains effectively. In contrast, we address this issue by representing the domain index with a dynamically updated mixture model.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: See Section 7.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
problem setup: 
 We aim at unsupervised domain adaptation: given N domains with different domain shifts, each domain has a domain identity w ∈ W = [N ] ≜ {1, ..., N }, and each domain contains D w data points. Similar to the conventional unsupervised domain adaptation setting, the N domains are divided into source domains with labeled data D S = {(x s i , y s i , w s i )} ns i=1 and target domains with unlabeled dataD T = {(x t i , w t i )} nt i=1. A foundational element that builds up our research problem is the diverse domain shifts [14] between different target domains and source domains. For source domains, the complexity of each target domain varies, which motivates us to dynamically infer domain indices in the "domain-themes" space and model them with dynamic Gaussian Mixture Model. We aim to (1) predict the label {y t i } nt i=1 of target domain data, and (2) infer local domain index u w ∈ R Bu and global domain index θ w ∈ R B θ in the dynamic "domain-themes" space. The summary of the notations is presented in Appendix J.Domain index. The domain index, distinct from domain identity w, represents domain semantics, thus empowering it to significantly enhance domain adaptation performance. As per its definition [36, 41] , the domain index satisfies the following : (1) To acquire domain-invariant data encoding z, the global domain index θ must remain independent of data encoding z , i.e., θ ⊥ ⊥ z or equivalently p(z | θ) = p(z). (2) Effectively representing data point x while averting the occurrence of collapsing.(3) Ensuring optimal performance of downstream tasks utilizing the data encoding z learned by the encoder under the aforementioned constraints, and necessitating the maintenance of sensitivity to labels.In circumstances where the domain index may not be readily accessible, the Variational Domain Index (VDI) [41] is a Bayesian approach to infer the domain index θ and u as latent variables. VDI factorizes the generative model p(x, y, u, θ, z | ε) as:EQUATIONwhere ε denotes the parameters for the prior probability distribution of the domain index θ. As shown in Equation 1, VDI stands capable of significantly enhancing domain adaptation proficiency by leveraging the inferred domain index for generating data encoding z. Note that the independence between the domain index θ and data encoding z, i.e., p(z | θ) = p(z), does not contradict p(z | x, u, θ), given the existence of multiple pathways between the domain index θ and data encoding z. Compared to VDI, which treats the distribution of domain index as a single Gaussian, we focus on a dynamic mixture of Gaussian distributions.Chinese Restaurant Process(CRP). The Dirichlet Process (DP) is a classical method used for clustering. However, DP is difficult to construct directly, we apply the Chinese Restaurant Process(CRP) [16, 17, 18, 46] to implement it. Since similar domains have similar domain indices, the clusters formed by the domain indices correspond one-to-one with the components in mixture of Gaussian distributions. CRP can be employed to determine which cluster a domain belongs to (i.e., which component distribution domain index corresponds to). Specially, CRP is able to dynamically and adaptively determine the number of mixture components. CRP operates as follows:EQUATIONwhere n k is the number of domain contained in cluster k, and parameter α is the concentration parameter of the CRP. A larger α implies a tendency to generate more domain clusters. EQUATIONwhere ε denotes the parameters for the prior probability distribution of the global domain index θ.With the setting of unsupervised domain adaptation, domains are not i.i.d, existing domain shift between domains. It implies that there may be substantial differences in distribution between domains. This leads to a problem that disparate target domains require a more significant degree of adaptation. Although we compute a distribution for domain index θ to enhance the capability of domain adaptation, it may not be effective enough to aid in adapting to a diversity of different target domains. Therefore, if local domain index u are adapted from a simple Gaussian distribution of global domain index θ, it may play a small role in improving the performance of domain adaptation.To better model global domain index and thus enhance the effectiveness of domain adaptation, we propose to maintain a mixture of dynamically updated global domain index θ distributions in the "domain-themes" space. Intuitively, similar domains have similar global domain indices, implying that the mixture of global domain index distributions is associated with a cluster of similar domains. The process of adapting local domain indices from global domain indices is illustrated in Figure 2 . Specifically, we consider a Gaussian Mixture Model (GMM) as the mixture of global domain index distributions. For each distinct domain, we first rank the candidate global domain index distributions in "domain-themes" space and select the distribution with the highest probability. We then derive the local domain index u from the global domain index θ. Therefore, the global domain index is designed to be dynamical GMM.Let v denotes the latent categorical variable indicating the assignment of a domain to a cluster, which is equivalent to selecting components in a mixture distribution. Based on the definition of v, we derive the updated representation of the distribution p(y, x | ε):EQUATIONThe component distribution θ v is selected from the mixture distribution of θ, and afterward, the local domain index u is obtained from global domain index θ v for generating domain-invariant data encoding z. Equation 3 and Equation 4 both represent the factorization of the distribution p(y, x | ε). θ in Equation 3 is the global domain index. While GMDI models the global domain index θ as a mixture of Gaussian distributions, θ v in Equation 4indicates the v-th component of the mixture distribution of θ with the prior p(v). Compared to the single distribution, the mixture of global domain index distributions adequately model the domain index of different domains, enhancing the effectiveness of domain adaptation in the face of varying degrees or even significant domain shifts. However, a remaining challenge is determining the number of components in the mixture distributions, especially when there are numerous domains, or even possibly infinite ones.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes]Answer: [Yes] Justification: Data and code will be publicly available once the paper is accepted.Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
generative process of gmdi: 
 In extreme cases, there may be infinite domains. Due to CRP's flexibility in dynamically determining the number of domain indices mixture components, we employ CRP to determine which cluster a domain belongs to (i.e., which component distribution domain index corresponds to). Specifically, we define the prior for domain indices cluster as a CRP, where the generation of new domain indices clusters is controlled by parameter α. Thus, the probability of a domain belonging to cluster k is calculated by Equation 2. Since CRP is an infinite mixture model, it is able to easily adapted to an infinite number of domains.Mixture of domain indices need a stick-breaking representation of CRP to obtain component weights. Stick-breaking representation indicating an infinite construction, considering the Dirichlet prior with parameter α, each element in the probability vector π = [π 1 , π 2 , ...] is non-negative and the sum of the elements is 1:EQUATIONEQUATIONEquation 6 is equivalent to the weights implied by CRP. Based on Equation 6 and Equation 4, the generative process of GMDI is as follows:EQUATIONEQUATIONEQUATIONEQUATIONwhere µ k and σ 2 k are mean vector and semi-positive covariance matrix of the k-th component in dynamic Gaussian mixture of domain indices, respectively. Figure 3 illustrates the generative process of VDI with a single distribution and GMDI with a mixture of distributions for domain indices. Since CRP is computationally intensive. To improve computational efficiency, we consider the stickbreaking construction to transform the infinite Gaussian mixture of domain indices into a finite one. It can be achieved by directly specifying an upper bound K for the number of components in Gaussian mixture of domain indices. Selecting an appropriate K allows to effectively reduce computational overhead. The finite version of the generative process of GMDI is available in Appendix A.Accordingly, the generative model can be factorized as follows:p(x, y, u, θ, z, v, β | α) = p(β | α)p(v | β)p(θ v )p(u | θ v )p(x | u)p(z | x, u, θ v )p(y | z) . (12)The predictor p(y | z) is a categorical distribution for classification tasks and a Gaussian distribution for regression tasks.The exact posterior of all latent variables, i.e., p(u, θ, z, v, β | x) is intractable, variational inference is used to approximate the posterior. Compared to the Monte Carlo sampling, variational inference allows both uncertainty quantification and computational efficiency. We employ structured variational inference to approximate the exact posterior, factorizing the approximate posterior q(u, θ, z, v, β | x):EQUATIONwhere γ, β, ψ u , ψ θ and ψ z respectively represent the parameters of the variational distributionsq(β), q(v), q(u | x), q(θ v | u) and q(z | x, u, θ v ).We train GMDI by maximizing the evidence lower bound(ELBO) to obtain the optimal variational distributions which best approximate exact posterior distributions. Section 5 demonstrates that our proposed GMDI has a more stringent evidence lower bound. Based on generative and inference process of GMDI, we caluculate the ELBO as follows:EQUATIONwhere ϕ and ξ represent the parameters of the variational distributions q(u, θ v , z|x) and q(u, θ v |x), respectively, and KL [•||•] is the Kullback-Leibler divergence.To ensure the independence between global domain index θ and data encoding z as defined, we follow VDI [41] by training an additional discriminator D with an adversarial loss. As we prove in Section 5 that the independence between global domain index θ and data encoding z relies on the independence between domain identity w and data encoding z, the adversarial loss is simplified to discriminate the domain identity w:EQUATIONCombining Equation 14 and Equation 15, the final objective of GMDI is:EQUATIONwhere λ denotes the hyper-parameter that balances two terms. Since the exact posterior of all latent variables is intractable, we propose to use a structured variational inference method to approximate the exact posterior. More details can be viewed in the appendix B.Variational distribution of β. To derive the optimal variational distribution of β, we only consider the terms related to β in L ELBO , then we can get the posterior q(β k ;γ k ) = Beta(β k ; γ k,1 , γ k,2 ) with parameters γ k,1 = 1 + η k and γ k,2 = α + K i=k+1 η i . Variational distribution of v.Similarly, the variational posterior of v can be calculated as a Categorical distribution q(v; η) = Categorical K (v; η), where the pareameters can be updated as:EQUATIONK k=1 η k = 1 and q(β; γ) = K-1 k=1 q(β k ; γ k ).Variational distribution of θ,u and z. With assuming that the latent parameters are sampled from Gaussian, we have the following forms:EQUATIONq(u|x;EQUATIONEQUATIONwhere µ and σ 2 are mean vector and semi-positive covariance matrix of Gaussian distribution. The parameters are updated by gradient descend. Specifically, we follow VDI [41] by using Earth Mover's Distance (EMD) [30] and Multi-Dimensional Scaling (MDS) [2] to infer θ v from u. 
theory: 
 In this section, we provide significant theoretical guarantees for our GMDI method. First, we give the upper bound for ELBO and adversial loss respectively. Second, we prove the upper bound of the whole loss with mutual information and entropy only. Moreover, we show that the upper bound can be achieved when the conditions are satisfied. Finally, we proove the significant result that our ELBO is better than the VDI, which means that a mixture of Gaussian prior can get better results. See Appendix C for detailed proof.Lemma 1 The ELBO of p(x, y) is bounded by the following formula with the Mutual Information, the Entropy and the KL-divergence:E p(x,y) [LELBO(p(x, y))] ≤I(y; z) + I(x; u, θ, z, v) -(H(x) + H(y)) -E q(x,u,θ,z,v) [KL[q(x|u, θ, v, z)||p(x|u, θ, v, z)]] -KL[q(u, θ, v, z|x)||p(u, θ, v, z)] .The main difference between and Lemma 1 in GMDI and Lemma 4.1 in VDI [41] is the last two KL terms and the inclusion of v.Lemma 2 (Information Decomposition of the Adversarial Loss [41] )We can decompose the global maximum of adversial loss as follows:max D E p(w,x) E q(z|x) [log D(w|z)] = I(z; θ) + I(z; w|θ) -H(w) .The global minimum of the function is achieved if and only if I(z; θ) = 0 and I(z; w|θ) = 0.Theorem 1 The upper bound of the objective function can be decomposed as follows:L GMDI ≤ I(y; z) + I(x; u, θ, z, v) -I(z; θ) -I(z; w|θ) -(H(x) + H(y) -H(w)) .The main difference between Theorem 1 in GMDI and Theorem 4.1 in VDI [41] is the inclusion of v.The global optimum is achieved if and only if: (1)I(z; θ) = I(z; w|θ) = 0, (2)I(y; z) andI(x; u, θ, z, v) are maximized, (3)KL[q(u, θ, v, z|x)||p(u, θ, v, z)] = 0 and KL[q(x|u, θ, v, z)||p(x|u, θ, v, z)] = 0.The main difference between Theorem 2 in GMDI and Theorem 4.2 in VDI [41] is that I(x; u, θ, z, v), which includes v, needs to be maximized, and the two KL divergences should equal zero.  
experimental study: 
 We verify the effectiveness of GMDI via experimental comparison and analysis. In particular, we answer three research questions: (RQ1) Can the performance of GMDI for domain adaptation outperform baselines? (RQ2) How effective is the global domain indices inferred by GMDI? (RQ3) How does the number of mixture components K affect results? Additional experimental results are available in Appendix K.Datasets. We compare GMDI with existing DA methods on the following datasets (see Appendix H and Appendix I for more details): Circle [36] is used for binary classification task. DG-15 and DG-60 [40] are synthetic datasets used for binary classification task. TPT-48 [40] dataset is a real-world dataset used for regression task. W ( 6) → E (42): Adapting models from the 6 states in the west to the 42 states in the east. N (24) → S (24): Adapting models from the 24 states in the north to the 24 states in the south. level-1 target domains: one hop away from the closest source domain. level-2 target domains: two hops away from the closest source domain. level-3 target domains: more than two hops away from the closest source domain. CompCars [43] dataset is a real-world dataset for 4-way classification task.Baselines. To evaluate our proposed GMDI, we compare it against eight state-of-the-art domain adaptation methods: Domain Adversarial Neural Networks (DANN) [4] , Adversarial Discriminative Domain Adaptation (ADDA) [5] , Conditional Domain Adaptation Neural Networks (CDANN) [48] , Margin Disparity Discrepancy (MDD) [47] , SENTRY [27] , Domain to Vector (D2V) [26] , and Variational Domain Index (VDI) [41] . Additionally, we include the results for models trained and tested only on the source domain (Source-only). Note that D2V is not applicable to regression tasks, so its results are not reported on the TPT-48 dataset. Moreover, since our proposed GMDI focuses on inferring domain indices when they are unavailable, whereas [36] and [40] assume domain indices 1 show that, on all three datasets, the performance of baselines other than D2V and VDI is only marginally better or worse than random guess (accuracy of 50% ). This is likely due to the complex relationships between domains within the datasets, making it difficult to adapt to target domains. Additionally, Source-only performs poorly due to overfitting. Compared to VDI, our GMDI improves accuracy by up to 3.4%, achieving very high accuracy (over 96.5%). This improvement is attributed to proposal of modeling the global domain index as the mixture distributions (e.g., Figure 8 ). ( 2) TPT-48. In Table 2 , we report the mean square error (MSE) of the evaluated methods on TPT-48. In both E (6) → W (42) and N (24) → S (24) regression tasks, all methods except DANN, SENTRY, and VDI performed worse than Source-only, indicating the occurrence of negative transfer. In contrast, GMDI significantly reduced the MSE compared to VDI, with average MSE decreases of 16% and 21%, respectively. (3) CompCars. The results in Table 1 show that our method achieves the best classification accuracy. All domain adaptation methods improved to varying degrees compared to Source-only, but our method achieved the highest increase in accuracy, with an improvement of up to 5%. In Figure 7 (left), the data encoding generated by VDI are clustered together, indicating a mixture of points from different class labels. In contrast, in Figure 7 (right), the data encoding of GMDI are separated by class label, demonstrating that GMDI can better distinguish points by class label. Across all datasets, GMDI significantly outperforms baselines, with minimum accuracy of 96.5 % on synthetic datasets , while MSE is reduced by at least 16 % on TPT-48 dataset. RQ3: Number of mixture components. We utilize the stick-breaking representation of the CRP to improve computational efficiency, setting an upper bound K on the number of components in GMM. To study the impact of the number of components on domain adaptation performance, we report classification accuracy on CompCars, more complex dataset for different values of K. For other 4 datasets, the best results are achieved with K = 2, while for CompCars, the best performance is obtained with K = 3. The results in Figure 6 show that accuracy is the lowest when K = 1, suggesting that maintaining a single domain index distribution is insufficient for diverse target domains. The choice of K is related to the concentration parameter of CRP and the dataset; the larger the concentration parameter and the more complex the dataset, the larger the value of K should be. 
conclusion and limitations: 
 In this work, we propose GMDI, a novel Gaussian Mixture Domain-Indexing algorithm, to address the challenge of inferring domain indices when they are unavailable. Unlike existing methods that assume global domain indices are sampled from a single static Gaussian, GMDI is the first one to utilize a mixture of dynamic Gaussians. The number of mixture components is determined adaptively by the Chinese Restaurant Process, enhancing the flexibility and effectiveness of domain adaptation.Our theoretical analysis confirms that GMDI achieves a more stringent evidence lower bound, closer to the log-likelihood. Extensive experiments validate the effectiveness of GMDI in inferring domain indices and highlight its potential practical applications. Specifically, for classification tasks, GMDI outperforms all approaches, and surpasses the state-of-the-art method, VDI, by up to 3.4%, reaching 99.3%. For regression tasks, GMDI reduces MSE by at least 16% (from 2.496 to 2.087) and by 21% (from 3.160 to 2.493), achieving the lowest errors among all methods. Despite these advantages, GMDI still relies on the availability of domain identities and cannot infer them as latent variables. Future work will focus on developing algorithms capable of inferring domain indices together with domain identities to further enhance the robustness and applicability of our approach.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We report performance mean after running experiments for multiple times with different seeds.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: The amount and the type of computing resource used in our experiments are described in Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: See appendix.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification:Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Citations for the existing assets (code, data and baseline models) are provided in Section 6 and Appendix.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: The infinite Chinese Restaurant Process (CRP) requires substantial computational overhead. To leverage CRP with lower computational cost, we use the stick-breaking construction to construct it.We set an upper bound on the number of Gaussian mixture components, eliminating the need for a varying number of mixture components. The finite stick-breaking construction of CRP is given as follows:EQUATIONwhere π = stickbreak(θ) is the prior parameters of the K-dimensional category variable v. Considering the finite stick-breaking construction of CRP mentioned above, we rewrite the generative process of GMDI:EQUATIONwhere µ k and σ 2 k are mean vector and semi-positive covariance matrix of the k-th component in GMM, respectively.Firstly, we factorize the generative model as follows:p(x, y, u, θ, z, v,EQUATION)During the inference process, we need to infer the latent variables. Since the target posterior distributions are intractable, we employ the technique of approximate variational inference. In this framework, we design variational distributions for these latent variables, aiming to approximate their true underlying posterior distributions:EQUATION) Thus we calculate the ELBO as follows:EQUATIONwhere the inequality is given by applying Jensen's inequality. Besides, we apply the adversarial loss to ensure the independence between global domain index and data encoding. The adversarial loss with a discriminator D is designed as follows:EQUATION) Our final objective is derived as:EQUATIONBy optimizing the objective function, we can calculate all the optimal variational distributions of latent variables as follows:Variational distribution of β. To obtain the variational posterior of the latent variable β, We only need to consider the related terms in L GMDI . Note that the adversarial loss L D is independent of the target latent variable, with all terms pertaining to β encompassed within the ELBO loss, which can be formulated as follows:EQUATION) Subsequently, by differentiating the function and setting the derivative equal to zero:EQUATIONwhere q(β k ; γ \k ) is the variational posterior of β without β k , we derive the final variational distribution of β k as follows:log q(β k ; γ k ) ∝ E q(v;η) E q(β \k ;γ \k ) [log p(β k ) + log p(v | β)] = log p(β k ) + E q(v;η) E q(β \k ;γ \k ) K-1 k=1 v k log β k + k-1 l=1 v k log(1 -β l ) + K j=1 v K log(1 -β j ) ∝ log p(β k ) + E q(v;η) v k log β k + K-1 j=k+1 v j log(1 -β k ) + v K log(1 -β k ) = log p(β k ) + η k log β k + K j=k+1 η j log(1 -β k ) .(30) Since we assume that p(β k ) ∼ Beta(•; 1, α), we have:EQUATION) Thus we can derive the optimal variational posterior of β k as:EQUATIONwhich is also a Beta distribution Beta(β k ; γ k,1 , γ k,2 ) with parameters:EQUATION)Variational distribution of v. Since v is a category variable, we assume its variational posterior to be a categorical distribution parameterized by η as:EQUATIONSimilarly, we only consider the terms related to v in L GMDI to derive the optimal varational posterior, which is calculated as:EQUATIONBy taking the derivative function of F (q(v; η)) with respective to zero:EQUATIONwe can finally have:EQUATIONwhereK k=1 η k = 1 and q(β; γ) = K-1 k=1 q(β k ; γ k ).Variational distribution of θ,u and z. The distribution of θ is assumed to include a mixture of components. The distribution of the latent variable θ is assumed to be a mixture of a series of distributions, while the latent variables u and z can be regarded as following conditional Gaussian distributions. Since it is highly intractable to precisely compute the posterior distributions of these latent variables, we employ variational Gaussian distributions to approximate the posterior distribution for each component of θ, as well as the conditional posterior distributions for u and z:EQUATIONEQUATIONEQUATIONwhere µ and Λ are mean vector and semi-positive covariance matrix of Gaussian distribution. All these variational distributions can be updated by maximizing L GMDI with back propagation.The proof process of the lemmas and theorems are partially based on VDI [41] .Lemma C.1 The ELBO of p(x, y) is bounded by the following formula with the Mutual Information, the Entropy and the KL-divergence:E p(x,y) [L ELBO (p(x, y))] ≤I(y; z) + I(x; u, θ, z, v) -(H(x) + H(y)) -E q(x,u,θ,z,v) [KL[q(x|u, θ, v, z)||p(x|u, θ, v, z)]] -KL[q(u, θ, v, z|x)||p(u, θ, z, v)] .The main difference between and Lemma C.1 in GMDI and Lemma B.1 in VDI [41] is the last two KL terms and the inclusion of v.Proof. In order to give an upper bound of p(x, y), we first calculate the ELBO as follows:log p(x, y) = log p(x, u, θ, v, β, z, y)dzdudθ v dvdβ = log p(x, u, θ, v, β, z, y) * q(u, θ, v, β, z|x) q(u, θ, v, β, z|x) dzdudθ v dvdβ = log E q p(x, u, θ, v, β, z, y) q(u, θ, v, β, z|x) ≥ E q log p(x, u, θ, v, β, z, y) q(u, θ, v, β, z|x) = E q log p(y|z)p(x|u, θ, v, β, z)p(u, θ, v, β, z) q(u, θ, v, β, z|x) = E q [log p(y|z)] + E q [log p(x|u, θ, v, β, z)] -KL[p(u, θ, v, β, z)||q(u, θ, v, β, z|x)] .Accordingly, we have the following ELBO objective:L ELBO (p(x, y)) = E q [log p(y|z)]+E q [log p(x|u, θ, v, β, z)]-KL[p(u, θ, v, β, z)||q(u, θ, v, β, z|x)] .(41) Here we aim at giving a formula only including Mutual Information, Entropy and KL-divergence to bound the objective. To achieve this, we first calculate the upper bound of E q log p(y|z), with denoting r(x, y, z) = p(x, y)q(z|x):EQUATIONEQUATION= I(y|z) -H(y) .When p(y|z) = r(y|z) , the maximum of E q [log p(y|z)] is achieved and then we have max E q [log p(y|z)] = I(y; z) -H(y).Secondly, we need to give an upper bound of E q [log p(x|u, θ, v, β, z)]. For convenient, we denote the joint distribution s(x, u, θ, v, β, z) = p(x)q(u, θ, v, β, z), then we can calculate as follows:E p(x,y) E q [log p(x|u, θ, v, β, z)] = E p(x) E q [log p(x|u, θ, v, β, z)] = E p(x) E q [log q(x|u, θ, v, β, z) p(x) p(x)p(x|u, θ, v, β, z) q(x|u, θ, v, β, z) ] = E p(x) E q [log q(x|u, θ, v, β, z) p(x) ] + E p(x) E q [log p(x)] + E p(x) E q [log p(x|u, θ, v, β, z) q(x|u, θ, v, β, z) ] = I s (x; u, θ, v, β, z) + H(x) -KL[q(x|u, θ, v, β, z)||p(x|u, θ, v, β, z)] .Finally, we apply these two bounds on the Eq. 41 and then we can proove the Lemma C.1:E p(x,y) [L ELBO (p(x, y))] ≤I(y; z) + I(x; u, θ, z, v) -(H(x) + H(y)) -E q(x,u,θ,z,v) [KL[q(x|u, θ, v, z)||p(x|u, θ, v, z)]] -KL[q(u, θ, v, z|x)||p(u, θ, z, v)] .Lemma C.2 (Information Decomposition of the Adversarial Loss [41] ) We can decompose the global maximum of adversial loss as follows: maxD E p(w,x) E q(z|x) [log D(w|z)] = I(z; θ, v) + I(z, w|θ, v) -H(w) .The global minimum of the function is achieved if and only if I(z; θ) = 0 and I(z, w|θ) = 0 Proof. With denoting t(w, x, θ, z) = p(w, x)q(z|x)q(θ, v|w), we have:EQUATIONand E p(w,x) E q(z|x) [log D(w|z)] achieves the maximum when t(w, x, θ, z) = D(w|z). To futher analyze the joint distribution t(w, θ, z) := q(z|θ, v, w)q(θ, v|w)p(w), we assume that there is a function v(w) mapping w to a group of domain-related weights v, then we can have: p(z|θ, v, w) = p(z|θ, v(w), w) = p(z|θ v(w) ) = p(z|w) .Accordingly, we can factorize the joint distribution t(w, θ, v, z) = q(z|w)q(θ v |w)p(w). Therefore, we can factorize I(z; θ, v, w) into two different styles with the chain rule for mutual information: I(z; θ, v) + I(z, w|θ, v) = I(z; θ, v, w) = I(z; w) + I q (z, θ, v|w) , (50) where I q (z, θ, v|w) = 0 due to the chain rule. That means:EQUATION)And we also have:EQUATION)Thus we have: maxEQUATION)Accordingly, min max D E p(w,x) E q(z|x) [log D(w|z)] = 0 if and only if I(z; θ, v) = I(z, w|θ, v) = 0 due to the fact that I(•) ≥ 0.Theorem C.1 The upper bound of the objective function can be decomposed as follows:L GMDI ≤ I(y; z) + I(x; u, θ, z, v) -I(z; θ) -I(z; w|θ) -(H(x) + H(y) -H(w)) .The main difference between Theorem C.1 in GMDI and Theorem B.1 in VDI [41] is the inclusion of v.Proof. To proove the theorem, we apply the Lemma C.1 and C.2 to directly get the final upper bound: L GMDI = E p(x,y) [L ELBO (p(x, y))] -max D E p(w,x) E q(z|x[q(u, θ, v, z|x)||p(u, θ, v, z)] = 0 and KL[q(x|u, θ, v, z)||p(x|u, θ, v, z)] = 0.The main difference between Theorem C.2 in GMDI and Theorem B.2 in VDI [41] is that I(x; u, θ, z, v), which includes v, needs to be maximized, and the two KL divergences should equal zero.Proof. The theorem can be prooved by observing the conditions from Lemma C.1, C.2 and Theorem C.1.Theorem C.3 Assuming the ELBO and objective of VDI are L VDI-ELBO and L VDI respectively, where domain indices are sampled from a simple Gaussian prior, we can prove that our objective achieves a more stringent evidence lower bound which is closer to the log-likelihood, and also a tighter upper bound of the objective: L VDI-ELBO ≤ L ELBO ≤ log p(x, y) and L VDI ≤ L GMDI . Proof. To compare our objective loss with the VDI's, we first list the ELBO loss of VDI here and provide an upper bound:L VDI-ELBO (p(x, y)) =E q [log p(y|z)] + E q [log p(x|u, θ, z)] -KL[p(u, θ, z)||q(u, θ, z|x)] ≤ I(y; z) + I(x; u, θ, z, v) -(H(x) + H(y)) -E q(x,u, θ,z) [KL[q(x|u, θ, z)||p(x|u, θ, z)]] -KL[q(u, θ, z|x)||p(u, θ, z)] .We can observe that the most significant difference is the prior distribution, which mainly affects the term of KL-divergence. To further analyze, it is obvious that when θ v=k = θ, VDI is a special case of our proposed method GMDI. Hence we have:EQUATIONFurther more, we can notice that the adversarial loss is independent of the prior distribution of global indices and we can have: The latent variables are estimated by neural networks. Specifically, for the local domain index u, we employ ResNet-18 [9] to approximate its posterior on CompCars dataset, while using multi-layer perceptrons for the other datasets. Additionally, all neural networks are implemented as multi-layer perceptrons. We use the features obtained from the ResNet-18 as inputs to our model. Furthermore, all inputs are uniformly normalized.EQUATIONWe implement our model based on the code of VDI [41] . We appreciate the authors for making their code publicly available. We run experiments on a single machine using 1 NVIDIA GeForce RTX 2080Ti with 11GB memory, 56 Intel Xeon CPUs (E5-2680 v4 @ 2.40GHz). It takes about 30 minutes to train GMDI with K=2 on synthetic datasets , 4 hours on TPT-48, and 6 hours on CompCars with K=3.We set the maximum number of mixture components K from 2,3, and the concentration parameter α to 1 throughout the experiments. Except for DG-15 and DG-60 datasets, which have a batch size of 32, all other datasets use a batch size of 16. Our model is trained with 20 to 100 warmup steps, learning rates ranging from 1 × 10 -5 to 1 × 10 -3 , and λ ranging from 0.1 to 1.Our model has the potential to be applied to various domain shift problems, which also implies the possibility of unintended negative consequences. However, we have not identified any specific societal harms associated with our model. If used maliciously, it could lead to negative impacts.The procedure of our proposed model GMDI is summarized by the pesuedo codes in Algorithm G.Algorithm 1 Bayesian Domain Adaptation with Gaussian Mixture Domain-Indexing Input: Dataset D S and D T , maximum number of mixture components K, concentration parameter α, learning rate ζ.1: Initialize parameters: ψ = {ψ u , ψ θ , ψ z } ; η k , ∀k = 1, ..., K; 2: repeat 3: Update γ k with γ k,1 = 1 + η k and γ k,2 = α + K i=k+1 η i , ∀k = 1, ..., K; 4:Update η with Equation 37; Update ψ ← ψ + ζ∇ ψ L GMDI . 9: until convergeTable 3 [41] summarizes the statistics for all the datasets used in our experiments. Figure 9 [41] visualizes the detailed information of Circle dataset. It contains 30 domains and is used for binary classification task. The data points in the Circle are arranged in a semicircular shape, with each domain occupying a different section of the semicircle. There is a decision boundary that separates the different labels. We use the first six domains as the source domains and the remaining 24 domains as the target domains. DG-15 and DG-60 datasets containing 15 and 60 domains, respectively. Both used for binary classification task. Adjacent domains in the datasets have similar decision boundaries. For these two datasets, we select 6 connected domains as the source domains, while the remaining domains serve as the target domains.Figure 10 [40] visualizes the detailed information of TPT-48 dataset. It contains the monthly average temperatures of 48 contiguous states in the United States from 2008 to 2019. Our regression task is to predict the average temperatures for the next 6 months using the average temperatures of the first 6 months. We divide this task into two finer-grained regression tasks with different adaptation directions:• W (6) → E (42): Adapting models from the 6 states in the west to the 42 states in the east.• N (24) → S (24): Adapting models from the 24 states in the north to the 24 states in the south.To better verify performance, the target domains in the above two regression tasks are divided into three groups based on their distance from the closest source domains. level-1 target domains: one hop away from the closest source domain. level-2 target domains: two hops away from the closest source domain. level-3 target domains: more than two hops away from the closest source domain. CompCars dataset includes three attributes: car types, viewpoints, and years of manufacture (YOMs). We select a subset of CompCars with 4 types (MPV, SUV, sedan and hatchback), 5 viewpoints (front(F), rear (R), side (S), front-side (FS), and rear-side (RS)), and 6 YOMs(2009 and 6 YOMs( , 2010 and 6 YOMs( , 2011 and 6 YOMs( , 2012 and 6 YOMs( , 2013 and 6 YOMs( , 2014) ) for our experiments. This subset is divided into 30 domains(5 viewpoints × 6 YOMs) based on viewpoints and YOMs. The car types are used as labels for prediction. The first domain, which has the front view and YOM 2009, is treated as the source domain, while the remaining 29 domains are target domains.Table 4 provides a summary of the notations used in this paper. Parameter of q(β) ηParameter of q(v)ψ u Parameter of q(u | x) ψ θ Parameter of q(θ | u) ψ z Parameter of q(z | x, u, θ v ) ϕParameter of q(u, θ v , z|x) ξParameter of q(u, θ v |x) λ Hyper-parameter that balances two terms in objective functionTo evaluate the impact and computational cost of the CRP, we conduct ablation and computational cost experiments.We implement GMDI w/o CRP using Gumbel-Softmax [11] . The number of components for GMDI w/o CRP is set to the upper bound K of GMDI, and the hyperparameter temperature τ for Gumbel-Softmax ranges from 0.1 to 50 (with the best performance reported). "Total time" refers to the total training duration, which concludes when the loss converges. The experimental results are shown in Table 5 and Table 6 . We find that although the proposed GMDI has a longer "Time per epoch" compared to GMDI w/o CRP, it converges faster due to the flexible number of components adaptively controlled by CRP. Therefore, the "Total time" is roughly the same as GMDI w/o CRP. On the TPT-48(W->E) dataset, due to faster convergence, the "Total time" of GMDI is less than that of GMDI w/o CRP and is even comparable to VDI. In all three datasets, the performance of GMDI w/o CRP is worse than that of GMDI. On the two TPT-48 datasets, compared to the baseline VDI, GMDI w/o CRP reduces MSE by 1% and 3%, whereas GMDI reduces MSE by 16% and 21%, surpassing GMDI w/o CRP. On the CompCars dataset, GMDI's accuracy is higher than that of GMDI w/o CRP. These results indicate that although using a fixed-component GMM is simpler, the computational costs are roughly equivalent to using CRP, but the performance is inferior to CRP, demonstrating the significance of CRP in GMDI.Additionally, compared to VDI, which models the domain index as a single Gaussian distribution, GMDI's computational costs are only slightly higher, yet its performance is superior. For large-scale datasets with numerous domains, modeling the domain index as a simple single Gaussian distribution may result in poor performance due to the dataset's complexity. The experimental results indicate that GMDI has broad applicability.NeurIPS Paper ChecklistJustification: Our research is conducted with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Reproduced result from VDI. 


Decision: The paper builds upon Variational Domain Indexing (VDI) from ICLR 2023 to introduce the Gaussian Mixture Domain-Indexing (GMDI) algorithm to tackle the challenge of domain adaptation when domain indices are not available. Unlike traditional approaches that assume a single Gaussian prior for domain indices, GMDI models these indices using a Gaussian Mixture Model, with the number of components dynamically determined by a Chinese Restaurant Process. This approach is intended to provide more flexibility and better adaptation to diverse domains.

### Strengths

- GMDI is innovative in its use of a Gaussian Mixture Model to capture the inherent structures among different domains, which allows for a more flexible and powerful inference of domain indices.
- The method is theoretically well-founded, with proofs that demonstrate a more stringent evidence lower bound and a tighter upper bound compared to existing methods.
- Extensive experiments show that GMDI outperforms state-of-the-art methods in both classification and regression tasks, achieving significant improvements in accuracy and mean squared error (MSE).

### Weaknesses

- The model’s learning loss is complex, involving multiple conditional Kullback-Leibler divergences, which could make implementation and interpretation challenging.
- The use of the Chinese Restaurant Process and Gaussian Mixture Model increases computational overhead, potentially limiting the scalability of the method for large-scale datasets or real-time applications.
- The paper lacks clarity in some areas, such as the definitions of 'local' and 'global' domain indices, and certain equations may contain typos, which could confuse readers.

Some concerns have been addressed by the authors during the rebuttal period. 

Near the end of the discussion, there are new concerns on the citation of VDI on the theorems, raised by the reviewers. As prompted by the reviewers, I also briefly checked the paper myself. I think these concerns are kind of valid. For example, most theorems/lemmas in the paper are from the VDI paper, and therefore should be proper attributed to VDI by citation. Another example is Line 255, where "VDI" should be "GMDI"? That said, I do agree with the majority that the paper has its technical merit. What is more important at this point is proper attribution and citation, especially on the theory part. In my opinion, properly attributing the theories will not affect the value of GMDI; on the contrary, it enhances GMDI's credibility and rigor. The authors did reply to me, promising that they will address these concerns fully in the revision. 

Below I summarize several points of revision suggestions after I went over GMDI and VDI:

+ Fix the typo in Line 255 from VDI to GMDI
+ **Lemma 1 and Lemma 4.1 in VDI:** After stating Lemma 1, **cite VDI** and **clearly state** that `“the main difference between and Lemma 1 in GMDI and Lemma 4.1 in VDI [37] is the last two KL terms and the inclusion of $v$”`. 
+ **Lemma 2 and Lemma 4.2 in VDI:** Lemma 2 is exactly the same as Lemma 4.2 in VDI. Therefore, please properly **cite VDI inside the bracket of Lemma 2’s title, i.e., `“(Information Decomposition of the Adversarial Loss **[37]**)”`.
+ **Theorem 1 and Theorem 4.1 in VDI:** After stating Theorem 1, **cite VDI** and **clearly state** that `“the main difference between VDI and GMDI is the inclusion of $v$"`.
+ **Theorem 2 and Theorem 4.2 in VDI:** After stating Theorem 2, **cite VDI** and **summarize** the main difference between VDI and GMDI. There are more differences in this theorem; I believe it would give more insight to readers on the contribution of GMDI beyond VDI if the authors can clarify them well here. 

I agree with the majority that this is an interesting paper, with solid extension to VDI, which itself is a promising new direction in domain adaptation and (hierarchical) Bayesian deep learning. 

I also tend to trust the authors’ promise on the revision and proper citation/attribution and will give a *conditional accept*. Note that 
+ I will help shepherd the revision, and make sure all points, especially those I listed above, are fully incorporated in the revision. Since this is a matter of academic rigor; otherwise this paper would have the risk of plagiarism. 
+ While I recommend acceptance for now, I will stay in touch with the PC Chairs and SAC, and reserve the right to retract the paper if the camera ready does not fully address the concerns above.

Reviews: review: paper_summary: paper_summary: The paper introduces the Gaussian Mixture Domain-Indexing (GMDI) algorithm for domain adaptation when domain indices are unavailable. Unlike traditional methods that use a simple Gaussian prior, GMDI employs a Gaussian Mixture Model adjusted by a Chinese Restaurant Process, enabling adaptive determination of mixture components.
strength_weakness: strength_weakness: - The learning loss for the proposed model is over too complex, featuring multiple conditional Kullback-Leibler divergences, which might complicate implementation and interpretation.
- The paper lacks clear definitions for the 'local' and 'global' domain index within the probabilistic graphical model, which could confuse readers about the model's scope and applicability.
- While introducing the Chinese Restaurant Process (CRP) adds flexibility to the Gaussian Mixture Model, it also increases computational costs. For the problem described in Figure 1, a fixed-component GMM might have been a simpler and more effective solution, though the number of components may not be that flexible. 
- The authors should also seriously consider empirically comparing the proposed method with a fixed-component counterpart, e.g. by simply using Gumbol softmax to infer the Gaussian component.
- The related work may not be sufficiently discussed. For example [a] discussed an end-to-end approach that learns the domain index using adversarial learning; [b] takes the domain index/identity as a latent dynamical system, coupled with adversarial learning. 

[a] Out-of-distribution Representation Learning for Time Series Classification
[b] Extrapolative Continuous-time Bayesian Neural Network for Fast Training-free Test-time Adaptation

----------------Minor------------------

- The connection between Equation (5) and Equation (6) is not clearly explained, leaving a gap in understanding the sequential logic of the model's formulation.
-The paper's clarity and accuracy in writing could be improved. For instance, the statement "DP requires a predefined number of components" is misleading, as Dirichlet Processes are inherently nonparametric and do not require a predefined number of components.
- Several symbols used in the equations are not adequately explained (both in the major paper and appendix), making it difficult to fully grasp the proposed model and its mathematical foundations.
- The proofs and the theoretical part seem to closely follow that of VDI, making it hard to evaluate its novelty.
questions: questions: See weaknesses.
limitations: limitations: The limitations are  addressed in the Sec. conclusion.
paper_summary: paper_summary: This paper proposes a Bayesian Domain Adaptation method with Gaussian Mixture Domain-Indexing (GMDI) to address the challenge of inferring domain indices when they are unavailable. Existing methods often assume a single Gaussian prior for domain indices, ignoring the inherent structures among domains. GMDI models domain indices as a mixture of Gaussian distributions, with the number of components dynamically determined by the Chinese Restaurant Process. This approach provides a higher level of flexibility and effectiveness in adapting to diverse target domains. Theoretical analysis demonstrates that GMDI achieves a more stringent evidence lower bound, closer to the log-likelihood. Extensive experiments on classification and regression tasks show that GMDI significantly outperforms baselines, achieving state-of-the-art performance.
strength_weakness: strength_weakness: 1. GMDI relies on the availability of domain identities but cannot infer them as latent variables. This limits its applicability to scenarios where domain identities are also unknown.

2. The use of the Chinese Restaurant Process and Gaussian Mixture Model can be computationally intensive, especially for large-scale datasets with numerous domains. This could hinder the scalability of GMDI.

3. In the experiment, the binary classification task is not challenging, which makes the performance advantage not convincing. The multi-classification tasks are necessary.
questions: questions: See weaknesses.
limitations: limitations: See weaknesses.
paper_summary: paper_summary: The paper proposes a novel method, "Gaussian Mixture Domain-Indexing" (GMDI), to address domain adaptation with inaccessible domain indices. The technique improves upon prior work by modeling the domain indices prior with a Gaussian Mixture. Empirically, it has been shown that the proposed method achieves state-of-the-art performance in classification and regression tasks.
strength_weakness: strength_weakness: **Clarity**: Some equations in the paper might have typos. Please see the details in the next section.

**Computational Overhead**: The use of CRP and dynamic mixtures increases computational overhead, which might make the method less practical for large-scale or real-time applications without further optimization. It would be helpful if the authors could provide some comments and empirical analysis on the computational overhead of GMDI.
questions: questions: **Typos?** Should the left side of Equation (1) be $p(z|x,\epsilon)$ or should $\int_{x}$ be added to the right side? A similar issue seems to exist in Equation (5) as well.

**Abstract Claim** I do not understand the authors' claim in the abstract, “For classification, GMDI improves accuracy by at least 63% (from 33.5% to 96.5%).” It seems from Table 1, the result for DG-60, the authors picked the worst performance of all baselines, ADDA with an accuracy of 33.5%. It is unclear why the authors claim the improvement is at least 63%.
limitations: limitations: I do not see specific limitations that might lead to potential negative societal impact.

Comments: title: Thanks for the prompt response.
Thanks for the prompt response.
Thanks.
Official Comment by Reviewer 72He
Arbitrary, reckless, and vague reasons for downgrading the rating from Reviewer uzxH.
Thanks
Thanks for the prompt response
Further explanation of the experimental results
Thanks for the prompt response
Incorrect comments and unfair downgraded rating from the reviewer uzxH
Thanks
comment: **4. "Furthermore, the methodology should be tested on data featuring more complex multimodal distributions, which may require a larger number of Gaussian components for accurate representation of the index. Such evaluation would help validate the method’s effectiveness and demonstrate more advantages over a fixed components method."**

Thank you for bringing this to our attention. We have already conducted comparative experiments on the most complex dataset used in VDI, *CompCars*, and the experimental results show that our proposed GMDI demonstrates clear advantages. Due to the scarcity of datasets with ground-truth domain indices, we leave the extension of comparative experiments on new datasets for future research.
Thank you for your detailed comments. The following is our responses to the concerns mentioned in the comments.

**1. "However, the study primarily builds upon the existing framework of VDI, and its theoretical developments, which are closely aligned with those of NDI, are mostly relegated to the appendix without appropriate citations. This lack of clarity suggests that some key proofs might be incorrectly attributed to this work."**

Thank you for pointing out this issue. Our theorems/lemmas are indeed partially based on VDI, and we will carefully review them to ensure that VDI is correctly cited in both the theorems/lemmas in the main body of our paper and the proofs in the appendix in the final version. 

Compared to VDI, the theoretical contribution of the proposed GMDI is that it proves GMDI achieves a more stringent evidence lower bound, as well as a tighter upper bound on the objective. Theoretically, it demonstrates that GMDI can effectively infer optimal domain indices.

**2.  "Additionally, the use of the Chinese Restaurant Process (CRP) to manage an unknown number of Gaussian components is not innovative, as thoroughly discussed in prior research [1]. Therefore, both the technical novelty and theoretical contributions of this paper seem incremental."**

Thanks for pointing out this. The main innovation of our proposed GMDI is that, to the best of our knowledge, GMDI is the first to introduce a CRP-based Gaussian mixture model to represent the domain index.

**3. "The experiments presented are not sufficiently comprehensive. I can understand that the authors have difficulties in conducting experiments using additional GMM variants, yet such comparisons are essential for a robust evaluation. Notably in the experiments, the CRP model frequently identifies the optimal number of components as two or three. It is important to figure out why CRP outperforms fixed-component GMMs, given that the "optimal" number of Gaussian components is known. A more thorough ablation study could clarify these results."**

Thanks for mentioning this. Since fixed-component GMMs are fundamentally similar, none of them can adaptively determine the number of mixture components. It is neither realistic nor meaningful to compare all variants of fixed-component GMMs within a short period. We have already conducted a detailed comparison with the Gumbel softmax version (i.e., GMDI w/o CRP). The experimental results show that our proposed GMDI demonstrates clear advantages. We leave the extension of comparative experiments with additional GMM variants for future research.

Regarding why the CRP model (i.e., GMDI) outperforms the fixed-component GMM (i.e., GMDI w/o CRP), it may be due to the complexities of the training process and parameter search space. Firstly, during training, the varying sample distribution within each batch can make it difficult for the fixed-component GMM (i.e., GMDI w/o CRP) to adapt well under a fixed K, potentially leading to suboptimal performance and convergence challenges. In contrast, the CRP model (i.e., GMDI) can automatically adapt to the sample distribution during training, and experiments have observed that GMDI converges faster to better performance, which aligns with our expectations.

Secondly, due to the complex parameter space, it is challenging to fine-tune both the fixed-component GMM (i.e., GMDI w/o CRP) and GMDI to their optimal performance within a short time frame. However, under fair comparison conditions, GMDI demonstrates superior performance and convergence speed compared to the fixed-component GMM (i.e., GMDI w/o CRP) within a limited search space, while significantly reducing the cost of parameter searching. As shown in the experimental results in the table below, we have tested various hyperparameters, K and $\tau$, as thoroughly as possible. The experiments indicate that even under the fairest possible comparison conditions, the fixed-component GMM (i.e., GMDI w/o CRP) struggles to achieve better performance, thereby validating our conclusions.

（1）GMDI w/o CRP on *CompCars* dataset:

| Accuracy(%) | K=1  | K=2  | K=3      | K=4  | K=5  |
| ----------- | ---- | ---- | -------- | ---- | ---- |
| $\tau$=100  | 42.6 | 42.1 | 42.4     | 41.8 | 40.0 |
| $\tau$=50   | 42.6 | 42.8 | **43.0** | 42.1 | 39.8 |
| $\tau$=10   | 42.6 | 41.8 | 42.0     | 42.9 | 42.4 |
| $\tau$=1    | 42.6 | 42.7 | 36.9     | 30.3 | 30.3 |
| $\tau$=0.1  | 42.6 | 38.5 | 41.8     | 30.4 | 30.3 |
Generally speaking, this paper focuses on an interesting issue within VDI, specifically its neglect of the multimodal nature of the latent index. However, the study primarily builds upon the existing framework of VDI, and its theoretical developments, which are closely aligned with those of NDI, are mostly relegated to the appendix without appropriate citations. This lack of clarity suggests that some key proofs might be incorrectly attributed to this work.

Additionally, the use of the Chinese Restaurant Process (CRP) to manage an unknown number of Gaussian components is not innovative, as thoroughly discussed in prior research [1]. Therefore, both the technical novelty and theoretical contributions of this paper seem incremental.

The experiments presented are not sufficiently comprehensive. I can understand that the authors have difficulties in conducting experiments using additional GMM variants, yet such comparisons are essential for a robust evaluation. Notably in the experiments, the CRP model frequently identifies the optimal number of components as two or three. It is important to figure out why CRP outperforms fixed-component GMMs, given that the "optimal"  number of Gaussian components is known.A more thorough ablation study could clarify these results.

Furthermore, the methodology should be tested on data featuring more complex multimodal distributions, which may require a larger number of Gaussian components for accurate representation of the index. Such evaluation would help validate the method’s effectiveness and demonstrate more advantages over a fixed components method.

Given the above concerns, I would lean towards rejecting this paper, as it requires major revisions.

[1] Li, N., Li, W., Jiang, Y., & Xia, S. T. (2022, August). Deep Dirichlet process mixture models. In Uncertainty in Artificial Intelligence (pp. 1138-1147). PMLR.
Dear Reviewer uzxH,

It seems the authors have clarified some of the potential misunderstandings. Could you please take a look to see if these clarifications change your mind?

Best,

AC
Thank you very much for your positive support.
Thank you. My concerns are well addressed. I stay postive for this manuscript.
The reasons given by the reviewer for downgrading the rating are very arbitrary, reckless, and vague.

During the first-round discussion in the discussion session, the claim made by the reviewer in response to our "Rebuttal"—"I am surprised that GMDI w/o CRP fails to beat VDI in terms of accuracy"—was absolutely incorrect. The reviewer hastily downgraded his/ her rating due to this basic mistake (misreading the results in the table).

In the second-round discussion, although we have addressed the claim raised in the first round and the reviewer admitted he/ she  has made a mistake (“It could be my mistake about the results”), he/ she raised a new but inessential issue: the reviewer guessed that "K=1 for CompCars," and based on this assumption, he/ she still maintained the downgraded rating. However, we clearly stated that K=3 for CompCars, not K=1.

In the third-round discussion, the reviewer stated that "The baselines of fixed-component GMM are not fully investigated or tuned (e.g., hyperparameters of the Gumbel-Softmax version and other GMM-based solutions like GMVAE)." However, we have already thoroughly tuned the hyperparameters of the Gumbel-Softmax version (i.e., GMDI w/o CRP). Additionally, it is unrealistic and unnecessary to compare all variants of fixed-component GMMs within a short time frame. We have already conducted comparative experiments with the Gumbel-Softmax version (i.e., GMDI w/o CRP). Below are our responses to the concerns mentioned in the third-round discussion.

**1. "The current solution is overly complex, which is not necessary."**

We have already addressed this issue in our previous response. The use of CRP in our proposed GMDI is actually essential. We model the domain index as a mixture of Gaussians, but the number of mixture components for the domain index is unknown beforehand. CRP allows us to adaptively determine the number of mixture components. Without CRP, the number of mixture components for the domain index cannot be determined, so while CRP slightly increases the complexity, it is absolutely necessary. 

**2. "The baselines of fixed-component GMM are not fully investigated or tuned (e.g., hyperparameters of the Gumbel-Softmax version and other GMM-based solutions like GMVAE), making the introduction of CRP not well-motivated."**

In our previous responses, we have explained that the hyperparameter K for the Gumbel-Softmax version (i.e., GMDI w/o CRP) ranges from 1 to 5. Additionally, in the global "Author Rebuttal," we stated that the hyperparameter $\tau$ ranges from 0.1 to 50. We have indeed searched for the hyperparameters of the Gumbel-Softmax version (i.e., GMDI w/o CRP) within a sufficiently large space. The experimental results show that our proposed GMDI has a clear advantage over the Gumbel-Softmax version (i.e., GMDI w/o CRP). The motivation for using CRP in GMDI is to adaptively determine the number of mixture components for the domain index. Since the number of mixture components for the domain index is unknown beforehand, the Gumbel-Softmax version (i.e., GMDI w/o CRP) requires constant tuning of the K value, which is extremely time-consuming and not suitable for domain index applications. In contrast, CRP with a simple upper bound on K can adaptively determine the number of mixture components. The GMVAE mentioned by the reviewer also requires constant tuning of the K value, and all fixed-component GMMs, in principle, cannot adaptively determine the number of mixture components. It is unrealistic and unnecessary to conduct comparative experiments with all fixed-component GMM variants within a short timeframe. Furthermore, we have emphasized the motivation for applying CRP in GMDI in the paper.

We believe that we have addressed all the issues raised by the reviewer during the discussion. We hope that the reviewer's evaluation is based on a careful review of the paper and rebuttal, rather than on random, reckless, and vague reasons for rejection and downgrading.
I am sorry that I still decided to keep my current ratings. My major concerns are the following: 1 current solution is over too complex, which is not necessary. 2 The baselines of fixed component GMM are not fully investigated or tuned tuned (e.g. hyperparameters of gumbol softmax version and other GMM based solution e.g.  GMVAE), making the introduction of CRP not well-motivated.
Thank you very much for your positive support.
Thank you for your response. The search space of the hyperparameter K for each dataset ranges from 1 to 5. Due to limitations in computational resources, we did not perform a broader search for the hyperparameter K. It's clear that the hyperparameter K for the *CompCars* dataset is not 1. We have already stated in the global "Author Rebuttal" that "The number of components for GMDI w/o CRP is set to the upper bound K of GMDI," and we have detailed the upper bound K for each dataset in the Experimental Study section of the paper. For the *CompCars* dataset, the hyperparameter K is 3, not 1. In our previous response, based on your suggestion, we have further applied a two-tailed paired t-test and found that the statistical significance of the observed performance difference between GMDI w/o CRP and VDI is significant for alpha=0.05. We are puzzled by your claim in the review that "the results are not significantly different from those of VDI," as this contradicts the results of our significance testing.
Moreover, while the search space of K indeed needs to be sufficiently large for GMM-based variational inference models, this is actually where GMDI shines. Using a fixed-component GMM requires extensive searching for K, a process that is very time-consuming. In contrast, GMDI leverages CRP to dynamically adjust the number of components with just an simple upper bound on K. Although the use of CRP in GMDI is more complex compared to fixed-component GMM, CRP's dynamic adaptability significantly reduces the cost of searching for K, making the use of CRP essential.

You downgraded the rating because of the incorrect statement "I am surprised that GMDI w/o CRP fails to beat VDI in terms of accuracy.". We have addressed this issue raised by you at the first-round discussion (see the previous discussion). And now we believe we also have addressed the issue raised at the second-round discussion (the above discussion). Because the issues have been addressed appropriately,  we wish you are able to keep at least the original rating for our paper rather than the current downgraded score. Thank you very much for your positive support.
Thank you. The authors have addressed all my concerns.
Please let us know about the search space of the hyperparameter K for each dataset.  Is K=1 for CompCars?  If so, it is no wonder the results are not significantly different from those of VDI. Furthermore, the search space of K should be sufficiently large for GMM-based variational inference models.  For now, I still have concerns about the over-complexity of the method. Such complexity may not be necessary for considering the multimodal assumption of the domain index. Therefore, I decided to keep my current ratings.
Dear Reviewer uzxH,

Just wanted to check if the authors' further response addressed your remaining concerns. Feel free to ask follow-up questions if needed. Thanks!

Best,

AC
To avoid misunderstanding and further clarify the experimental results presented in the global "Author Rebuttal," we would like to provide the following additional explanation. VDI is our baseline, which models the domain index as a single Gaussian. Our proposed GMDI models the domain index as a mixture of Gaussians and incorporates CRP. GMDI w/o CRP is a version that follows your suggestion by not using CRP and instead utilizing a fixed-component GMM (Gaussian Mixture Model with a fixed number of components).

We conducted detailed experiments on the relatively complex *TPT-48* and *CompCars* datasets. The results show that GMDI w/o CRP performs better than VDI but falls short of GMDI. This indicates that using a fixed-component GMM improves performance, but the lack of CRP leads to results inferior to those of GMDI, demonstrating the importance of both GMM modeling and the use of CRP. Please let us know if we need to provide further explanation. Thank you very much.
Thank you very much for your prompt response. The performance gap between the two models, i.e., GMDI w/o CRP and  VDI, is significant, which is what we expected and good result. As you suggested, we further applied a two-tailed paired t-test and found that the statistical significance of the observed performance difference between these two models is significant for alpha=0.05. Because of the positive experimental result, we are expecting you to at least upgrade the current rating score to the original rating score. Thank you very much for you kind support.
It could be my mistake about the results. Since the performance gap of these two models are relatively small. Can the authors further provide significant testing results?
Thanks for your response. However, we are quite sure that your following claim is not correct, "I am surprised that GMDI w/o CRP fails to beat VDI in terms of accuracy." In fact, on the *CompCars* dataset, GMDI w/o CRP achieves an accuracy of 43.0%, while the accuracy of VDI is 42.5%. Please see the corresponding details in the global "Author Rebuttal". This indicates that GMDI w/o CRP indeed performs better than VDI, contrary to your claim. We have conducted detailed experiments on *TPT-48* dataset for regression task and *CompCars* dataset for classification task. On all datasets, GMDI w/o CRP slightly outperforms VDI, while it lags far behind GMDI. Because of the above, the downgraded rating by you is unfair for us. Thank you very much and looking forward to your positive response.
Thanks for the efforts. The author's response addressed some of my concerns, but my major concerns about using a simpler solution such as fixed component GMM still exists. Though the additional results are provided, I am surprised that GMDI w/o CRP fails to beat VDI in terms of accuracy, while haveing a smaller MSE. This could due to overfiting. Based on the general quality of the paper and above concerns, i decide to downgrades my ratings.