## 3 Semi-Interpretable Generators
### GMM Case  
Moving forwards, we present the more expressive case of GMMs as generators. The probability density of a GMM is a weighted sum of \(K\)\(d\)-dimensional Gaussians given by \(\sum_{k=1}^{K}p_{k}\mathcal{N}(I_{k},\Sigma_{k})\) where \(p_{k}>0\) is the normalized frequency of the \(k\)-th Gaussian while \((\mu_{k},\Sigma_{k})\) are the respective mean vector and covariance matrix. A sample via the reparameterization trick for the GMM is obtained as  
\[x^{\prime}=\sum_{k=1}^{K}\mathbb{1}_{[n_{k-1},n_{k}]}(u)\Big{(}\mu_{k}+L_{k}z_ {k}\Big{)}\,,\text{with}\,u\sim\mathcal{U}(0,1)\,,z_{k}\sim\mathcal{N}(0,I_{ d})\,,\text{i.i.d.} \tag{6}\]  
where \(\mathbb{1}_{[a,b]}(\cdot)\) indicates the indicator function of the interval \([a,b]\) while \(w_{k}\) corresponds to the cumulative mass function given by \(w_{k}=\sum_{j=1}^{k}p_{k}\) with \(w_{0}=0\) and \(w_{K}=1\). Notice that the indicator function is non-differentiable because it is discontinuous at the extreme points \(a\) and \(b\).  
The generator parameters to be learned are the probability and the parameters of the affine transformation for each Gaussian whereas \(K\) is fixed. The choice of \(K\) will respect the data set characteristics and particularly, as a rule of thumb, \(K\) could take a value close to the number of clusters in the data set. A clustering algorithm such as k-means (or Louvain clustering or similar algorithm) could be used on the training data set prior training, in order to determine the value of \(K\). Moreover, we experimentally observed that utilizing a higher value of \(K\) leads to many weights, \(p_{k}\), to be close to zero. Therefore, we typically choose a large value for \(K\) without instilling instabilities in the training process. In other words, we could select \(K\) using our framework iteratively by keeping only the modes that are above a certain threshold.  
While the training of \(\mu_{k}\) and \(L_{k}\) is straightforward, the training of the probabilities is more elaborate. First, the softmax function is used on every training step to normalize the values of \(p_{k}=\frac{\sigma_{k}}{\sum_{k=1}^{K}e^{i}}\) where \(q_{k}\)' are the unconstrained trainable variables which can be trained with standard stochastic back-propagation algorithms. Indeed, the direct training of the probabilities, \(p_{k}\) cannot be carried over with the standard unconstrained algorithms because of the constraint requirements for \(p_{k}\). Second and in order to avoid the non-differentiability issue of the indicator function, we approximate it via the difference of two scaled sigmoid functions with a scaling factor, \(c\). Mathematically, the approximation is given by  
\[\mathbb{1}_{[u_{k-1},u_{k}]}(u)\approx\sigma_{c}(u-w_{k-1})-\sigma_{c}(u-w_{k}) =\frac{1}{1+e^{-c(u-w_{k-1})}}-\frac{1}{1+e^{-c(u-w_{k})}}\,. \tag{7}\]  
Figure 1 shows both the indication function and its approximation for two values of the scaling factor. We set the value for the scaling factor to \(c=300\) which is a reasonable compromise between accuracy of the indicator function and the efficient propagation of the (non-zero) gradients. We also remark that the authors in [36] followed another approach in similar rationale to avoid differentiation issues associated with sampling from discrete distributions.