## 2 Regularized MoLE
### Regularization on Experts  
The next step is to solve the maximization (6) for the experts. The experts' contribution to the ELL, \(Q_{e}^{t}(\mathbf{\Theta})\), can be further decomposed to account for the contribution of each expert separately, as follows  
\[Q_{e}^{t}(\mathbf{\Theta}) =\sum_{p=1}^{p}Q_{ep}^{t}(\mathbf{\Theta}), \tag{8}\] \[Q_{ep}^{t}(\mathbf{\Theta}) =\sum_{i=1}^{n}\gamma_{pi}^{t}\log\mathcal{N}\Big{(}y_{i}|\mathbf{ x}_{i}^{T}\mathbf{\theta}_{p},\sigma_{p}^{2}\Big{)}, \tag{9}\]  
where \(Q_{ep}^{t}(\,\cdot\,)\) is the individual contribution of expert \(p\) to the ELL. The new vector of expert coefficients \(\mathbf{\theta}_{p}^{t+1}\) is the one that maximizes \(Q_{ep}^{t}(\mathbf{\Theta})\), which is the solution of the following weighted least squares problem  
\[\mathbf{\theta}_{p}^{t+1}=\operatorname*{arg\,min}_{\mathbf{\theta}_{p}^{t}}\frac{1}{ 2}\sum_{i=1}^{n}\gamma_{pi}^{t}(y_{i}-\mathbf{x}_{i}^{T}\mathbf{\theta}_{p}^{*})^{ 2}=(\mathbf{X}^{T}\mathbf{\Gamma}_{p}\mathbf{X})^{-1}\mathbf{X}^{T}\mathbf{\Gamma}_{p} \mathbf{y}, \tag{10}\]  
where \(\mathbf{\Gamma}_{p}=\text{diag}\Big{(}\gamma_{p1}^{t},\ldots,\gamma_{pn}^{t}\Big{)}\) is the matrix of responsibilities of expert \(p\). However, in presence of collinearity, the inverse \((\mathbf{X}^{T}\mathbf{\Gamma}_{p}\mathbf{X})^{-1}\) becomes ill conditioned. To overcome this situation, a EN regularization is added as follows to penalize the loss function  
\[\mathbf{\theta}_{p}^{t+1}=\operatorname*{arg\,min}_{\mathbf{\theta}_{p}^{t}}\frac{1}{ 2}\sum_{i=1}^{n}\gamma_{pi}^{t}(y_{i}-\mathbf{x}_{i}^{T}\mathbf{\theta}_{p}^{*})^{ 2}+\lambda_{p}^{e}\Bigg{(}\alpha\sum_{i}|\theta_{pi}^{*}|+(1-\alpha)\sum_{i}| \theta_{pi}^{*}|^{2}\Bigg{)}, \tag{11}\]  
where \(\lambda_{p}^{e}\) is the regularization parameter that controls the sparsity of the solution, and \(\alpha\) is the EN penalty. The EN regularization allows the use of the Lasso penalty when \(\alpha=1\), the RR penalty when \(\alpha=0\), or the EN penalty when \(\alpha=0.5\) (a trade-off between Lasso and RR). The Lasso penalty is known to promote sparse solutions by shrinking the regression coefficient towards zero, being adequate when dealing with a large number of inputs. However, the Lasso penalty does not consider the group effect, i.e. for correlated features, it will tend to select one input while shrinking the coefficient of others to zero. The RR penalty alleviates the ill posed problem of \((\mathbf{X}^{T}\mathbf{\Gamma}_{p}\mathbf{X}+\lambda_{p}^{e}\mathbf{\Gamma})^{-1}\) by adding a regularization factor to it. In RR, all coefficients will have a contribution to prediction. On the other hand, the EN penalty integrates the benefits of both, it provides sparse solutions (Lasso penalty) while considering the effect group problem (RR penalty). The error variance update becomes the solution of the following maximization problem  
\[\mathbf{\sigma}_{p}^{2\,t+1}=\operatorname*{arg\,max}_{\sigma_{p}^{2*}}\,\sum_{i= 1}^{n}\gamma_{pi}^{t}\log\mathcal{N}\Big{(}y_{i}|\mathbf{x}_{i}^{T}\mathbf{\theta} _{p},{\sigma_{p}^{2*}}\Big{)}, \tag{12}\]  
which is equal to  
\[\mathbf{\sigma}_{p}^{2\,t+1}=\frac{\sum_{i=1}^{n}\gamma_{pi}^{t}(y_{i}-\mathbf{x}_ {i}^{T}\mathbf{\theta}_{p}^{t+1})^{2}}{\sum_{i=1}^{n}\gamma_{pi}^{t}}, \tag{13}\]  
where the updated \(\mathbf{\theta}_{p}^{t+1}\) is used to compute the updated variance term. The maximization problem in Equation (11) can be achieved using the coordinate gradient optimization descent algorithm, described in [34]. The coordinate gradient descent algorithm minimizes the loss function for each coordinate at a time. It converges to theoptimal value if the loss function is convex and differentiable, conditions that hold for the loss function (11).