## 2 Regularized MoLE
### Regularization on Gates  
On the experts' update, the EN regularization was easily added to penalize the experts' parameters (Section 2.4). On the other hand, during the gates learning, the application of the regularization term is not explicit. The contribution of gates to the ELL, \(Q_{g}^{t}(\mathbf{G})\), is given by  
\[ \begin{split} Q_{g}^{t}(\mathbf{G})&=\sum_{i=1}^{n} \sum_{p=1}^{p}\gamma_{pi}^{t}\log g_{pi}\\ &=\sum_{i=1}^{n}\Bigg{[}\sum_{p=2}^{p}\gamma_{pi}^{t}\mathbf{x} ^{T}\mathbf{v}_{p}-\sum_{p=1}^{p}\gamma_{pi}^{t}\log\left(1+\sum_{k=2}^{p} \exp\left(\mathbf{x}_{i}^{T}\mathbf{v}_{k}\right)\right)\Bigg{]}.\end{split}  \tag{17}\]  
The solution for the new parameters \(\mathbf{G}^{t+1}\) by direct maximization of Equation (17) is not straightforward. Instead, the iterative re-weighted least squares (IRLS) method will be employed. The IRLS algorithm works in the following way. First, define the following auxiliary variable \(\hat{\mathbf{v}}_{p}^{k}\) as the auxiliary gate parameter in the \(k\)-th iteration of the IRLS algorithm. It is updated as follows:  
\[\hat{\mathbf{v}}_{p}^{k+1} =\hat{\mathbf{v}}_{p}^{k}+\Bigg{[}\frac{\partial^{2}Q_{g}^{t}( \mathbf{G})}{\partial\mathbf{v}_{p}^{k}(\mathbf{v}_{p}^{k})^{T}}\Bigg{]}^{-1} \Bigg{[}\frac{\partial Q_{g}^{t}(\mathbf{G})}{\partial\mathbf{v}_{p}^{k}} \Bigg{]}, \tag{18}\] \[=(\mathbf{X}^{T}\mathbf{R}_{p}\mathbf{X})\mathbf{X}^{T}\mathbf{u} _{p}=(\mathbf{X}^{T}\mathbf{R}_{p}\mathbf{X})\mathbf{X}^{T}\mathbf{R}_{p}\hat{ \mathbf{z}}_{p}^{k},\]where  
\[\mathbf{R}_{p} =\text{diag}(g_{p1}(1-g_{p1}),\dots,g_{pn}(1-g_{pn})),\] \[\mathbf{u}_{p} =[g_{p1}-\gamma_{p1}^{t},\dots,g_{pn}-\gamma_{pn}^{t}]^{T},\] \[\hat{\mathbf{z}}_{p}^{k} =\mathbf{X}\hat{\mathbf{v}}_{p}^{k}-\mathbf{R}_{p}^{-1}\mathbf{u}_ {p}.\]  
The IRLS algorithm runs until convergence. The employed convergence detection condition is \(||\hat{\mathbf{v}}_{p}^{k+1}-\hat{\mathbf{v}}_{p}^{k}||_{2}<\xi_{I}\). At the end of the algorithm, the new gate parameter is updated as \(\mathbf{v}_{p}^{t+1}=\hat{\mathbf{v}}_{p}^{k}\), where \(K\) is the last iteration of the IRLS algorithm. In practice, few \(K\) iterations are necessary in the IRLS algorithm. The IRLS solution can become unstable due to the ill-conditioned inverse \(\mathbf{R}_{p}^{-1}\). Many authors have proposed different alternatives to IRLS, to overcome this problem, such as the proximal-Newton EM in [31] that avoids the matrix inversion in the gates update. Here, a regularization term is added, which works well in practice. Specifically, the regularization term \(\xi\mathbf{I}\) is added such that the inverse becomes \((\mathbf{R}_{p}+\xi\mathbf{I})^{-1}\). In experiments, \(\zeta=10^{-3}\) is used. However, similarly to the experts, the solution for the gates at each iteration of IRLS becomes ill conditioned in the presence of collinearity. Thus, through the closed form solution of the inner loops of the IRLS algorithm in Equation (18), the results derived for the experts are mimicked to the gates. By modifying so, the value of \(\hat{\mathbf{v}}_{p}^{k}\) to be found at the \(k\)-th IRLS algorithm iteration with the EN regularization added, becomes  
\[\hat{\mathbf{v}}_{p}^{k+1}=\underset{\hat{\mathbf{v}}_{p}^{k}}{\arg\min}\frac {1}{2}\sum_{i=1}^{n}r_{pi}(\hat{z}_{pi}^{k}-\mathbf{x}_{i}^{T}\hat{\mathbf{v}} _{p}^{*})^{2}+\lambda_{p}^{S}\Bigg{(}\alpha\sum_{i}|\hat{\mathbf{v}}_{pi}^{*} |+(1-\alpha)\sum_{i}|\hat{\mathbf{v}}_{pi}^{*}|^{2}\Bigg{)}. \tag{19}\]  
Then, at each iteration of IRLS, an EN penalty is added to the loss function. In total, \(Kp\) EN maximization problems are computed. In a way similar to case of the experts, the solution of (19) can be achieved by using the coordinate gradient descent algorithm, described in [34].