## 2 Regularized MoLE
### Formulation  
The MoLE approximates the true probability distribution function (PDF), defined as \(p(y_{i}|\mathbf{x}_{i})\), by a superposition of PDF's:  
\[p(y_{i}|\mathbf{x}_{i},\boldsymbol{\Omega})=\sum_{p=1}^{p}g_{p}(\mathbf{x}_{i },\mathbf{G})\,p(y_{i}|\mathbf{x}_{i},\boldsymbol{\Theta}_{p}), \tag{3}\]  
where \(p(y_{i}|\mathbf{x}_{i},\boldsymbol{\Theta}_{p})=\mathcal{N}(y_{i}|\mathbf{x}_ {i}^{T}\theta_{p},\sigma_{p}^{2})\) is the conditional PDF of expert \(p\), governed by the parameters \(\boldsymbol{\Theta}_{p}=\{\theta_{p},\sigma_{p}^{2}\}\), where \(\sigma_{p}^{2}\) is the error variance of expert \(p\), assumed to be zero mean. The collection of gates parameters is represented by \(\mathbf{G}=\{\mathbf{v}_{1},\ldots,\mathbf{v}_{p}\}\). The collection of all parameters is defined as \(\boldsymbol{\Omega}=\{\mathbf{\theta}_{1},\ldots,\mathbf{\theta}_{p},\sigma_{ 1}^{2},\ldots,\sigma_{p}^{2},\mathbf{v}_{1},\ldots,\mathbf{v}_{p}\}\). The estimation of \(\boldsymbol{\Omega}\) by maximum likelihood is unfeasible. Instead, the expectation maximization (EM) is commonly employed [33]. The EM uses a two-steps iterative procedure to perform the likelihood maximization of MoLE. In the first, called the expectation step (E-Step), the expectation of the log-likelihood (ELL) is evaluated, while in the second step, called maximization step (M-Step), new parameters are determined by maximum likelihood estimation from the ELL. Therefore, the maximization of likelihood in EM is achieved trough successive improvements of ELL; see [33] for further details on the application of the EM algorithm. Since the EM is an iterative procedure, the superscript \(t\) is used to indicate the \(t\)-th iteration of the EM algorithm, e.g., \(\mathbf{\Omega}^{t}\) is the vector of the estimated parameters of MoLE at iteration \(t\). The ELL at the \(t\)-th E-Step, \(Q^{t}(\mathbf{\Omega})\), is given by  
\[Q^{t}(\mathbf{\Omega}) =Q^{t}_{e}(\mathbf{\Theta})+Q^{t}_{g}(\mathbf{G}), \tag{4}\] \[Q^{t}_{e}(\mathbf{\Theta}) =\sum_{i=1}^{n}\sum_{p=1}^{p}\gamma^{t}_{p}(\mathbf{x}_{i})\log\mathcal{ N}(y_{i}|\mathbf{x}_{i}^{\mathsf{T}}\mathbf{\theta}_{p},\sigma_{p}^{2}),\] \[Q^{t}_{g}(\mathbf{G}) =\sum_{i=1}^{n}\sum_{p=1}^{p}\gamma^{t}_{p}(\mathbf{x}_{i})\log g_{p} (\mathbf{x}_{i},\mathbf{G}),\] \[\gamma^{t}_{p}(\mathbf{x}_{i}) =\frac{g^{t}_{p}(\mathbf{x}_{i})\mathcal{N}\Big{(}y_{i}|\mathbf{x}_{i}^{ \mathsf{T}}\mathbf{\theta}_{p^{\prime}}^{t},\sigma_{p}^{2}\Big{)}}{\sum_{l=1}^{p}g ^{t}_{l}(\mathbf{x}_{i})\mathcal{N}\big{(}y_{i}|\mathbf{x}_{i}^{\mathsf{T}}\mathbf{\theta} _{p}^{t},\sigma_{l}^{2}\big{)}}, \tag{5}\]  
where \(Q^{t}_{e}(\,\cdot\,)\) and \(Q^{t}_{g}(\,\cdot\,)\) account for the expert and gate contributions to the ELL, respectively, and \(\gamma^{t}_{p}(\mathbf{x}_{i})\) is the responsibility, which accounts for the probability of sample \(\mathbf{x}_{i}\) belonging to the region covered by expert \(p\); from now on \(\gamma^{t}_{p}(\mathbf{x}_{i})\) will also be referred as \(\gamma^{t}_{pi}\): The convergence of the ELL, can be measured by computing the ELL difference \(|Q^{t}(\mathbf{\Omega}^{t})-Q^{t-1}(\mathbf{\Omega}^{t-1})|\) at each iteration. The complete derivation of ELL, and its relation to the log-likelihood of (3) can be found in ([9], Section 4). In the M-Step, the objective is to estimate the new parameters \(\mathbf{\Omega}^{t+1}=\{\mathbf{\Theta}^{t+1},\mathbf{G}^{t+1}\}\), that maximize Equation (4). This is stated as the following optimization problem:  
\[\mathbf{\Theta}^{t+1} =\operatorname*{arg\,max}_{\mathbf{\Theta}^{*}}Q^{t}_{e}(\mathbf{\Theta}^ {*}), \tag{6}\] \[\mathbf{G}^{t+1} =\operatorname*{arg\,max}_{\mathbf{G}^{*}}Q^{t}_{g}(\mathbf{G}^{*}). \tag{7}\]  
The EM guarantees that the ELL, and consequently the log-likelihood of Equation (3), increases monotonically. The EM algorithm runs until the convergence of the ELL, where the convergence detection condition is defined as \(|Q^{t}(\mathbf{\Omega}^{t})-Q^{t-1}(\mathbf{\Omega}^{t-1})|\leq\xi_{Q}\), where \(\xi_{Q}\) is a convergence threshold.