Question: What’s difference between \hat{\pi_i} and \hat{\alpha_i}?
Evidence: Consider the Gaussian mixtures corresponding to the source and target channel conditional densities, where θ c and θ c are the parameter vectors of the original and adapted MDN. Here µ i (x) ∈ R d is the mean vector, σ 2 i (x) ∈ R d + is the variance vector, and π i (x) ∈ [0, 1] is the prior probability (weight) of component i for the original mixture. We have assumed that the Gaussian components have a diagonal covariance matrix, with σ 2 i (x) being the diagonal elements 2 . The mixture weights are parameterized using the softmax function as π i (x) = e αi(x) / k j=1 e αj (x) , ∀i. The MDN simply predicts the un-normalized weights α i (x) ∈ R or the prior logits. The parameter vector of component i is defined as Affine and Inverse-Affine Feature Transfomations. Applying the above result to our MDN with k components, we define the affine feature transformation for a given symbol x and component i,
(3)
It is straightforward to also define the inverse-affine transformation from y to y as
For the case of diagonal covariances, we constrain C i to be diagonal. These feature transformations will be used for aligning the target and source class-conditional distributions of the decoder input.
Parameter Transformations. The corresponding transformations between the source and target Gaussian mixture parameters for any symbol x ∈ X and component i ∈ [k] are given by
where
) is a diagonal scale matrix for the variances; and β i ∈ R and γ i ∈ R are the scale and offset for the prior logits.
The vector of all adaptation parameters to be optimized is defined as
, where ψ T i contains the affine-transformation parameters from component i. The number of adaptation parameters (dimension of ψ) is given by k (d 2 + 2 d + 2). This is typically much smaller than the number of MDN parameters (weights and biases from all layers), even for shallow fully-connected NNs. In Fig., the adaptation layer mapping φ(x) to φ(x) basically implements the parameter transformations defined in Eq. ().
Assumptions and Key Insight. The proposed adaptation of the MDN is based on the affinetransformation property of multivariate Gaussians, i.e., one can transform between any two multivariate Gaussians through an affine transformation. Assumption 1: The source and target Gaussian mixtures have the same number of components. This is a practical assumption we make in order to not have to change the architecture of the MDN. Adding or removing components would require a change to the output layer of the MDN. Also, this assumption can be practically justified when k is chosen to be sufficiently large. Assumption 2: The two mixtures have a one-to-one correspondence between the components. This assumption makes in mathematically convenient to derive a closed-form expression for the KL-divergence between two Gaussian mixtures, which would not be possible in the general case.
Based on the above assumptions, we can formulate the MDN adaptation as an equivalent problem of finding the optimal set of affine transformations (one per-component) from the source to the target Gaussian mixture. This is a much smaller problem compared to optimizing the weights of all the MDN layers. Moreover, the affine transformations are bijective, allowing the feature and parameter mapping to be applied in the inverse direction. To reduce the possibility of the adapted MDN finding bad solutions due to the small-sample regime, we include a regularization term based on the KL-divergence (KLD) in the adaptation objective.
Here µi(x) ∈ R d is the mean vector, σ 2 i (x) ∈ R d + is the variance vector, and πi(x) ∈ [0, 1] is the prior probability (weight) of component i for the original mixture. We have assumed that the Gaussian components have a diagonal covariance matrix, with σ 2 i (x) being the diagonal elements 2 . The mixture weights are parameterized using the softmax function as πi(x) = e αi(x) / Pk j=1 e αj (x) , ∀i. The MDN simply predicts the un-normalized weights αi(x) ∈ R or the prior logits. The parameter vector of component i is defined as φi(x) T = [αi(x), µi(x) T ,σ 2 i (x) T ], and the MDN output φ(x) has dimension k (2d + 1). The adapted MDN predicts the parameters of the target Gaussian mixture φb(x) = Mθbc (x) as shown in Fig. 1. The parameters of the adapted MDN and Gaussian mixture are similarly defined, with a hat in the notation.
Answer: πi∈[0,1] is the mixture prior probability of component i, and αi∈R is the corresponding un-normalized logit. They are related via the softmax function as πi=eαi/∑jeαj. Adapting the unconstrained logits is easier than adapting the corresponding probabilities. We apologize that this was not clearly defined in the paper. We have now revised the paper with this definition in section 3.1.