Question: The authors assume that the source and target distribution have the same number of components with a one-to-one correspondence. How realistic is the working hypothesis?
Evidence: Consider the Gaussian mixtures corresponding to the source and target channel conditional densities, where θ c and θ c are the parameter vectors of the original and adapted MDN. Here µ i (x) ∈ R d is the mean vector, σ 2 i (x) ∈ R d + is the variance vector, and π i (x) ∈ [0, 1] is the prior probability (weight) of component i for the original mixture. We have assumed that the Gaussian components have a diagonal covariance matrix, with σ 2 i (x) being the diagonal elements 2 . The mixture weights are parameterized using the softmax function as π i (x) = e αi(x) / k j=1 e αj (x) , ∀i. The MDN simply predicts the un-normalized weights α i (x) ∈ R or the prior logits. The parameter vector of component i is defined as Affine and Inverse-Affine Feature Transfomations. Applying the above result to our MDN with k components, we define the affine feature transformation for a given symbol x and component i,
(3)
It is straightforward to also define the inverse-affine transformation from y to y as
For the case of diagonal covariances, we constrain C i to be diagonal. These feature transformations will be used for aligning the target and source class-conditional distributions of the decoder input.
Parameter Transformations. The corresponding transformations between the source and target Gaussian mixture parameters for any symbol x ∈ X and component i ∈ [k] are given by
where
) is a diagonal scale matrix for the variances; and β i ∈ R and γ i ∈ R are the scale and offset for the prior logits.
The vector of all adaptation parameters to be optimized is defined as
, where ψ T i contains the affine-transformation parameters from component i. The number of adaptation parameters (dimension of ψ) is given by k (d 2 + 2 d + 2). This is typically much smaller than the number of MDN parameters (weights and biases from all layers), even for shallow fully-connected NNs. In Fig., the adaptation layer mapping φ(x) to φ(x) basically implements the parameter transformations defined in Eq. ().
Assumptions and Key Insight. The proposed adaptation of the MDN is based on the affinetransformation property of multivariate Gaussians, i.e., one can transform between any two multivariate Gaussians through an affine transformation. Assumption 1: The source and target Gaussian mixtures have the same number of components. This is a practical assumption we make in order to not have to change the architecture of the MDN. Adding or removing components would require a change to the output layer of the MDN. Also, this assumption can be practically justified when k is chosen to be sufficiently large. Assumption 2: The two mixtures have a one-to-one correspondence between the components. This assumption makes in mathematically convenient to derive a closed-form expression for the KL-divergence between two Gaussian mixtures, which would not be possible in the general case.
Based on the above assumptions, we can formulate the MDN adaptation as an equivalent problem of finding the optimal set of affine transformations (one per-component) from the source to the target Gaussian mixture. This is a much smaller problem compared to optimizing the weights of all the MDN layers. Moreover, the affine transformations are bijective, allowing the feature and parameter mapping to be applied in the inverse direction. To reduce the possibility of the adapted MDN finding bad solutions due to the small-sample regime, we include a regularization term based on the KL-divergence (KLD) in the adaptation objective.
Assumptions and Key Insight. The proposed adaptation of the MDN is based on the affinetransformation property of multivariate Gaussians, i.e., one can transform between any two multivariate Gaussians through an affine transformation. Assumption 1: The source and target Gaussian mixtures have the same number of components. This is a practical assumption we make in order to not have to change the architecture of the MDN. Adding or removing components would require a change to the output layer of the MDN. Also, this assumption can be practically justified when k is chosen to be sufficiently large. Assumption 2: The two mixtures have a one-to-one correspondence between the components. This assumption makes in mathematically convenient to derive a closed-form expression for the KL-divergence between two Gaussian mixtures, which would not be possible in the general case. Based on the above assumptions, we can formulate the MDN adaptation as an equivalent problem of finding the optimal set of affine transformations (one per-component) from the source to the target Gaussian mixture. This is a much smaller problem compared to optimizing the weights of all the MDN layers. Moreover, the affine transformations are bijective, allowing the feature and parameter mapping to be applied in the inverse direction. To reduce the possibility of the adapted MDN finding bad solutions due to the small-sample regime, we include a regularization term based on the KL-divergence (KLD) in the adaptation objective
Answer: In the revised paper, we state these two assumptions and provide a brief justification for the same. The assumption of same number of components is a practical one that we make in order to not have to change the architecture of the MDN. Adding or removing components would require a change to the output layer of the MDN. Also, this assumption can be practically justified when the number of components is chosen to be sufficiently large. With a large number of components, the appearance or disappearance of a few clusters of data from the true data distribution can be compensated for by other components in the Gaussian mixture.