On Expert Estimation in Hierarchical Mixture of Experts: Beyond Softmax Gating Functions

## Abstract

Abstract With the growing prominence of the Mixture of Experts (MoE) architecture in developing large-scale foundation models, we investigate the Hierarchical Mixture of Experts (HMoE), a specialized variant of MoE that excels in handling complex inputs and improving performance on targeted tasks. Our investigation highlights the advantages of using varied gating functions, moving beyond softmax gating within HMoE frameworks. We theoretically demonstrate that applying tailored gating functions to each expert group allows HMoE to achieve robust results, even when optimal gating functions are applied only at select hierarchical levels. Empirical validation across diverse scenarios supports these theoretical claims. This includes large-scale multimodal tasks, image classification, and latent domain discovery and prediction tasks, where our modified HMoE models show great performance improvements.

## 1 Introduction

In recent years, the integration of mixture-of-experts (MoE) within large-scale foundation models has markedly advanced the machine learning field [, , , , ]. MoE architectures, known for their ability to efficiently handle diverse and complex datasets, have facilitated significant improvements in model performance without a proportional increase in computational demand. They address bottlenecks associated with traditional deep learning architectures by dynamically allocating resources to parts of the model for which they are most relevant [, ]. The Hierarchical Mixture of Experts (HMoE) model [] is a special type of MoE architecture that is characterized by a layered structure of decision modules and expert networks that operate in tandem to refine decision-making at each level, optimizing the allocation of computational resources and enhancing specialization for complex tasks. Unlike the standard MoE, which typically involves a single gating network directing inputs to various expert networks, HMoE introduces multiple layers of gating mechanisms and experts. This hierarchical design divides the problem space recursively, allowing different experts to specialize in subspaces of the input space, leading to enhanced flexibility and model generalization [, ].

Figure [1](#S1.F1) compares HMoE and standard MoE in processing multimodal input data. The hierarchical structure of HMoE makes it particularly effective at handling complex inputs, such as data that can be divided into semantically meaningful subgroups. This recursive partitioning enables HMoE to select features and specialize in various segments of the input space more effectively, especially in high-dimensional data scenarios []. The unique capability of HMoE to handle complex datasets makes it particularly valuable across a range of applications. Historically, HMoE has been applied on image classification [], speech recognition [, ], and complex decision-making tasks [, ].
However, there is a notable lack of recent studies on HMoE in the literature, partly due to its more complex structure as compared to standard MoE. For instance, while standard MoE requires the selection of a single gating function, HMoE necessitates the choice of multiple gating functions, introducing additional hyperparameters and therefore greater complication in model specification. Given the increasing complexity of input data in the modern era, such as multiple modalities or subgroups defined by ambiguous latent domains, there is a growing demand for models that can deliver accurate and individualized predictions for each subgroup. Therefore, it is worthwhile to study HMoE, which can leverage the intrinsic information within complex input structures and achieve superior performance on corresponding tasks.

Figure: Figure 1: Comparison of HMoE and standard MoE in managing multimodal input: MoE excels at processing homogeneous inputs. However, it faces challenges with more intricate structures, such as inputs that can be split into subgroups or those with inherently hierarchical configurations. By contrast, HMoE improves upon this by decomposing tasks into subproblems and directing subsets of data to specialized groups of experts. This approach allows for more granular specialization and enhances the model’s capability to handle complex inputs.
Refer to caption: /html/2410.02935/assets/x1.png

In this paper, we investigate distinct selections of gating functions within HMoE and their impact on overall performance. This is a critical issue and will lay the groundwork for future research in this relatively unstudied domain.
It is important to note that expert specialization, as discussed in [], is a critical problem that involves understanding how quickly an expert becomes specialized in specific tasks or aspects of the data. To address this, we conduct a comprehensive analysis of the convergence behavior of experts within two-level HMoE models, using three different combinations of the conventional softmax gating [] and the Laplace gating as suggested in [].
Our theoretical analysis reveals that employing Laplace gating at both levels of the HMoE framework accelerates expert convergence and significantly improves performance relative to baseline. We further validate this through extensive empirical evaluations across diverse scenarios, demonstrating HMoE’s effectiveness on complex datasets, such as those with inherent hierarchies or clustered data that can be partitioned into subgroups. By incorporating the three aforementioned combinations of gating functions, our experiments confirm that using Laplace gating at both levels consistently improves performance across multiple downstream tasks compared to the standard softmax gating baseline. Additionally, we observe that different combinations of Laplace and softmax gating can also noticeably enhance results, leading to better and more robust performance by offering a broader selection of gating function combinations. These findings highlight the practical benefits of selecting appropriate gating functions to enhance HMoE’s capabilities.

Notations. We let $[n]$ stand for the set $\{1,2,\ldots,n\}$ for any $n\in\mathbb{N}$. Next, for any set $S$, we denote $|S|$ as its cardinality. For any vector $v\in\mathbb{R}^{d}$ and $\alpha:=(\alpha_{1},\alpha_{2},\ldots,\alpha_{d})\in\mathbb{N}^{d}$, we let $v^{\alpha}=v_{1}^{\alpha_{1}}v_{2}^{\alpha_{2}}\ldots v_{d}^{\alpha_{d}}$, $|v|:=v_{1}+v_{2}+\ldots+v_{d}$ and $\alpha!:=\alpha_{1}!\alpha_{2}!\ldots\alpha_{d}!$, while $\|v\|$ stands for its $L^{2}$-norm value. For any two positive sequences $(a_{n})_{n\geq 1}$ and $(b_{n})_{n\geq 1}$, we write $a_{n}=\mathcal{O}(b_{n})$ or $a_{n}\lesssim b_{n}$ if there exists $C>0$ such that $a_{n}\leq Cb_{n}$ for all $n\in\mathbb{N}$. Meanwhile, the notation $a_{n}=\mathcal{O}_{P}(b_{n})$ indicates that $a_{n}/b_{n}$ is stochastically bounded. Lastly, for any two probability density functions $p,q$ dominated by the Lebesgue measure $\mu$, we denote $h^{2}(p,q)=\frac{1}{2}\int(\sqrt{p}-\sqrt{q})^{2}d\mu$ as their squared Hellinger distance and $V(p,q)=\frac{1}{2}\int|p-q|d\mu$ as their Total Variation distance.

## 2 Theoretical Contributions and Methods

We conduct a convergence analysis of expert estimation in the two-level Gaussian HMoE under three settings of alternatively using the Softmax gating and Laplace gating in the two levels of the model. Our goal is to find which gating combination would induce the fastest expert estimation rate.

### 2.1 Softmax-Softmax HMoE

We begin by considering the scenario when the two-level Gaussian HMoE is equipped with the Softmax gating in both levels. More specifically, let us assume that an i.i.d. sample of size $n$: $(\boldsymbol{X}_{1},Y_{1}),(\boldsymbol{X}_{2},Y_{2}),\ldots,(\boldsymbol{X}_{n},Y_{n})$ in $\mathbb{R}^{d}\times\mathbb{R}$, where $\boldsymbol{X}_{i}$ is an input and $Y_{i}$ is a response variable, is generated from that model whose conditional density function is given by

$$ $\displaystyle p^{SS}_{G_{*}}(y|\boldsymbol{x})$ $\displaystyle:=\sum_{i_{1}=1}^{k^{*}_{1}}\sigma((\boldsymbol{a}_{i_{1}}^{*})^{\top}\boldsymbol{x}+b_{i_{1}}^{*})\sum_{i_{2}=1}^{k^{*}_{2}}\sigma((\boldsymbol{\omega}_{i_{2}|i_{1}}^{*})^{\top}\boldsymbol{x}+\beta_{i_{2}|i_{1}}^{*})\pi(y|(\boldsymbol{\eta}_{i_{1}i_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{i_{1}i_{2}}^{*},\nu_{i_{1}i_{2}}^{*}).$ (1) $$

Above, the abbreviation $SS$ stands for “Softmax-Softmax”, indicating that the softmax gating is used in both levels of the Gaussian HMoE. Next, we define

$$ $\displaystyle G_{*}:=\sum_{i_{1}=1}^{k^{*}_{1}}\exp(b_{i_{1}}^{*})\sum_{i_{2}=1}^{k^{*}_{2}}\exp(\beta_{i_{2}|i_{1}}^{*})\delta_{(\boldsymbol{a}_{i_{1}}^{*},\boldsymbol{\omega}_{i_{2}|i_{1}}^{*},\tau_{i_{1}i_{2}}^{*},\boldsymbol{\eta}_{i_{1}i_{2}}^{*},\nu_{i_{1}i_{2}}^{*})}$ $$

as a corresponding *mixing measure*, i.e., a weighted sum of Dirac measures $\delta$ with true yet unknown parameters $(b_{i_{1}}^{*},\boldsymbol{a}_{i_{1}}^{*},\beta_{i_{2}|i_{1}}^{*},\boldsymbol{\omega}_{i_{2}|i_{1}}^{*},\tau_{i_{1}i_{2}}^{*},\boldsymbol{\eta}_{i_{1}i_{2}}^{*},\nu_{i_{1}i_{2}}^{*})$ in the parameter space $\Theta\subseteq\mathbb{R}\times\mathbb{R}^{d}\times\mathbb{R}\times\mathbb{R}^{d}\times\mathbb{R}^{q}\times\mathbb{R}_{+}$. Additionally, $k^{*}_{1}$ denotes the number of mixtures in the two-level Gaussian HMoE, whereas $k^{*}_{2}$ is the number of experts in each mixture.
For any integer $k\in\mathbb{N}$ and real-valued vector $(v_{i})_{i=1}^{k}$, we denote by $\sigma(v_{i}):={\exp(v_{i})}/{\sum_{j=1}^{k}\exp(v_{j})}$ the softmax function. Meanwhile, $\pi(\cdot|\mu,\nu)$ is an univariate Gaussian density function with mean $\mu$ and variance $\nu$.

Recall that expert specialization is an essential problem in the MoE literature where we explore how fast an expert specializes in some tasks or some aspects of the data [, ], which can be captured through the convergence analysis of expert estimation.

Maximum likelihood estimation (MLE). To estimate the unknown parameters, or equivalently the unknown mixing measure $G_{*}$, we utilize the maximum likelihood method []. For simplicity, we assume that the value of $k^{*}_{1}$ is known as the analysis would become unnecessarily complicated otherwise. At the same time, the value of $k^{*}_{2}$ remains unknown. Then, we over-specify the true model ([1](#S2.E1)) by considering an MLE within a class of mixing measures with at most $k^{*}_{1}k_{2}$ components, where $k_{2}>k^{*}_{2}$, as follows:

$$ $\displaystyle\widehat{G}^{SS}_{n}:=\operatorname*{arg\,max}_{G\in\mathcal{G}_{k^{*}_{1},k_{2}}(\Theta)}\frac{1}{n}\sum_{i=1}^{n}\log(p^{SS}_{G}(Y_{i}|\boldsymbol{X}_{i})),$ (2) $$

in which

$$ $\displaystyle\mathcal{G}_{k^{*}_{1},k_{2}}(\Theta)$ $\displaystyle:=\Big{\{}G=\sum_{i_{1}=1}^{k^{*}_{1}}\exp(b_{i_{1}})\sum_{i_{2}=1}^{k^{\prime}_{2}}\exp(\beta_{i_{2}|i_{1}})\delta_{(\boldsymbol{a}_{i_{1}},\boldsymbol{\omega}_{i_{2}|i_{1}},\boldsymbol{\eta}_{i_{1}i_{2}},\tau_{i_{1}i_{2}},\nu_{i_{1}i_{2}})}:k^{\prime}_{2}\in[k_{2}],$ $\displaystyle\hskip 142.26378pt(b_{i_{1}},\boldsymbol{a}_{i_{1}},\beta_{i_{2}|i_{1}},\boldsymbol{\omega}_{i_{1}i_{2}},\tau_{i_{1}i_{2}},\boldsymbol{\eta}_{i_{1}i_{2}},\nu_{i_{1}i_{2}})\in\Theta\Big{\}}.$ $$

Assumptions. For the sake of theory, we make some following standard assumptions on the data as well as the model parameters throughout this paper:

*(A.1) We assume that the parameter space $\Theta$ is compact and the input space $\mathcal{X}$ is bounded to guarantee the MLE convergence.*

*(A.2) In order that the Gaussian HMoE is identifiable, that is, $p^{SS}_{G}(y|\boldsymbol{x})=p^{SS}_{G_{*}}(y|\boldsymbol{x})$ for almost every $(\boldsymbol{x},y)$ implies $G\equiv G_{*}$, the softmax gating value must not be invariant to parameter translation. Therefore, we let $\boldsymbol{a}^{*}_{k^{*}_{1}}=\boldsymbol{0}_{d},b^{*}_{k^{*}_{1}}=0$ and $\boldsymbol{\omega}^{*}_{k^{*}_{2}|i_{1}}=\boldsymbol{0}_{d},\beta^{*}_{k^{*}_{2}|i_{1}}=0$ for any $i_{1}\in[k^{*}_{1}]$.*

*(A.3) For any $i_{1}\in[k^{*}_{1}]$, let $(\boldsymbol{\eta}^{*}_{i_{1}1},\tau^{*}_{i_{1}1},\nu^{*}_{i_{1}1}),\ldots,(\boldsymbol{\eta}^{*}_{i_{1}k^{*}_{2}},\tau^{*}_{i_{1}k^{*}_{2}},\nu^{*}_{i_{1}k^{*}_{2}})$ be distinct parameters so that the Gaussian distributions associated with the same parent node are different from each other.*

*(A.4) To ensure that the gating depend on the input, we assume at least one among gating parameters in the first level $\boldsymbol{a}^{*}_{1},\ldots,\boldsymbol{a}^{*}_{k^{*}_{1}}$ (resp. those in the second level $\boldsymbol{\omega}^{*}_{1},\ldots,\boldsymbol{\omega}^{*}_{k^{*}_{1}}$) is different from zero.*

Now, we investigate the convergence behavior of the density estimation $p^{SS}_{\widehat{G}_{n}}$ to the true density $p^{SS}_{G_{*}}$ in Theorem [1](#Thmtheorem1) whose proof can be found in Appendix [F](#A6).

###### Theorem 1 .

Given an MLE $\widehat{G}^{SS}_{n}$ defined in equation ([2](#S2.E2)), the corresponding density estimation $p^{SS}_{\widehat{G}_{n}}$ converges to the true density $p^{SS}_{G_{*}}$ under the Hellinger distance $h$ at following rate:

$$ $\displaystyle\mathbb{P}(\mathbb{E}_{\boldsymbol{X}}[h(p^{SS}_{\widehat{G}^{SS}_{n}}(\cdot|\boldsymbol{X}),p^{SS}_{G_{*}}(\cdot|\boldsymbol{X}))]>C_{1}\sqrt{\log(n)/n})\lesssim\exp(-c_{1}\log n),$ $$

where $C_{1}$ and $c_{1}$ are universal constants.

Theorem [1](#Thmtheorem1) indicates that the rate for estimating the true conditional density of the Gaussian HMoE is of parametric order $\widetilde{\mathcal{O}}_{P}(n^{-1/2})$. Consequently, if we are able to construct a loss function among parameters denoted by, for example, $\mathcal{L}(\widehat{G}_{n},G_{*})$
, and establish the bound
$\mathcal{L}(\widehat{G}_{n},G_{*})\lesssim\mathbb{E}_{\boldsymbol{X}}[h(p^{SS}_{\widehat{G}^{SS}_{n}}(\cdot|\boldsymbol{X}),p^{SS}_{G_{*}}(\cdot|\boldsymbol{X}))]$, then we will obtain the parameter estimation rates $\mathcal{L}(\widehat{G}_{n},G_{*})=\widetilde{\mathcal{O}}_{P}(n^{-1/2})$, which leads to our desired rates for estimating experts. However, while such Hellinger bound has been well studied under the setting of one-level Gaussian MoE [, , ], it has remained elusive for the hierarchical setting. In the following paragraph, we will point out fundamental obstacles for deriving that bound.

Challenges.
Our main technique for deriving the parameter estimation rates is to decompose the density estimation and the true density, i.e. $p^{SS}_{\widehat{G}^{SS}_{n}}(y|\boldsymbol{x})-p^{SS}_{G_{*}}(y|\boldsymbol{x})$, into a combination of linearly independent terms by applying the Taylor expansion to the function $u(\boldsymbol{x};\boldsymbol{a},\boldsymbol{\omega},\boldsymbol{\eta},\tau,\nu):=\exp(\boldsymbol{a}^{\top}\boldsymbol{x})\exp(\boldsymbol{\omega}^{\top}\boldsymbol{x})\pi(y|\boldsymbol{\eta}^{\top}\boldsymbol{x}+\tau,\nu)$ with respect to its parameters. In previous works [, , ], it is well-known that there is an interaction between the mean parameter $\tau$ and variance $\nu$ of the Gaussian density via the partial differential equation (PDE) $\frac{\partial u}{\partial\nu}=\frac{1}{2}\cdot\frac{\partial^{2}u}{\partial\tau^{2}}$. Such PDE induces several linearly dependent terms in the aforementioned decomposition, thereby leading to significantly slow rates for estimating those parameters. In this paper, we discover that the first-level gating parameter $a$ also interacts with the second-level parameters $\boldsymbol{\omega},\nu,\tau$, that is,

$$ $\displaystyle\frac{\partial u}{\partial\boldsymbol{\eta}}=\frac{\partial^{2}u}{\partial\boldsymbol{a}\partial\tau},\quad\frac{\partial u}{\partial\boldsymbol{a}}=\frac{\partial u}{\partial\boldsymbol{\omega}}.$ (3) $$

To the best of our knowledge, these intrinsic interactions have not been noted before in the literature. Therefore, we have to take the solvability of the unforeseen system of poylnomial equations ([4](#S2.E4)) into account to capture that interaction.

System of polynomial equations. For each $m\geq 2$, we define $r^{SS}(m)$ as the smallest natural number $r$ such that the following system does not have any non-trivial solutions for the unknown variables $(p_{i_{2}},\boldsymbol{q}_{1i_{2}},\boldsymbol{q}_{2},\boldsymbol{q}_{3i_{2}},q_{4i_{2}},q_{5i_{2}})_{i_{2}=1}^{m}$

$$ $\displaystyle\sum_{i_{2}=1}^{m}\sum_{(\boldsymbol{\alpha}_{1},\boldsymbol{\alpha}_{2},\boldsymbol{\alpha}_{3},\alpha_{4},\alpha_{5})\in\mathcal{I}^{SS}_{\boldsymbol{\rho}_{1},\rho_{2}}}\frac{1}{\boldsymbol{\alpha}!}\cdot p^{2}_{i_{2}}\boldsymbol{q}_{1i_{2}}^{\boldsymbol{\alpha}_{1}}\boldsymbol{q}_{2}^{\boldsymbol{\alpha}_{2}}\boldsymbol{q}_{3i_{2}}^{\boldsymbol{\alpha}_{3}}q_{4i_{2}}^{\alpha_{4}}q_{5i_{2}}^{\alpha_{5}}=0,\quad 1\leq|\boldsymbol{\rho}_{1}|+\rho_{2}\leq r,$ (4) $$

where $\mathcal{I}^{SS}_{\boldsymbol{\rho}_{1},\rho_{2}}:=\{(\boldsymbol{\alpha}_{1},\boldsymbol{\alpha}_{2},\boldsymbol{\alpha}_{3},\alpha_{4},\alpha_{5})\in\mathbb{R}^{d}\times\mathbb{R}^{d}\times\mathbb{R}^{d}\times\mathbb{R}\times\mathbb{R}_{+}:\boldsymbol{\alpha}_{1}+\boldsymbol{\alpha}_{2}+\boldsymbol{\alpha}_{3}=\boldsymbol{\rho}_{1},|\boldsymbol{\alpha}_{3}|+\alpha_{4}+2\alpha_{5}=\rho_{2}\}$. Here, a solution is categorized as non-trivial if all the values of $p_{i_{2}}$ are different from zero and at least one among $q_{4i_{2}}$ is non-zero. Note that $r^{SS}(m)$ is a monotonically increasing function. However, finding the exact value of $r^{SS}(m)$ is a demanding problem in the field of algebraic geometry []. Thus, we provide in Lemma [1](#Thmlemma1) (whose proof is in Appendix [G](#A7)) some specific values of $r^{SS}(m)$ when $m$ is small, while those for larger $m$ are left for future development.

###### Lemma 1 .

For any $d\geq 1$, we have that $r^{SS}(2)=4$ and $r^{SS}(3)=6$, while we conjecture that $r^{SS}(m)\geq 7$ for $m\geq 4$.

Voronoi loss. To precisely characterize the convergence rate of parameter estimation, it is necessary to capture the number of fitted parameters approaching each individual true parameter in both levels of Gaussian HMoE. For that purpose, let us introduce the concept of Voronoi cells []. In particular, given an arbitrary mixing measure $G\in\mathcal{G}_{k^{*}_{1}k_{2}}(\Theta)$, we distribute its atoms across the Voronoi cells $\{\mathcal{V}_{j_{1}}(G),j_{1}\in[k^{*}_{1}]\}$ and $\{\mathcal{V}_{j_{2}|j_{1}}(G),j_{1}\in[k^{*}_{1}],j_{2}\in[k^{*}_{2}]\}$ generated by the atoms of $G_{*}$, where

$$ $\displaystyle\mathcal{V}_{j_{1}}\equiv\mathcal{V}_{j_{1}}(G)$ $\displaystyle:=\{i_{1}\in[k^{*}_{1}]:\|\boldsymbol{a}_{i_{1}}-\boldsymbol{a}^{*}_{j_{1}}\|\leq\|\boldsymbol{a}_{i_{1}}-\boldsymbol{a}^{*}_{\ell_{1}}\|,\forall\ell_{1}\neq j_{1}\},$ $\displaystyle\mathcal{V}_{j_{2}|j_{1}}\equiv\mathcal{V}_{j_{2}|j_{1}}(G)$ $\displaystyle:=\{i_{2}\in[k_{2}]:\|\boldsymbol{\zeta}_{i_{2}|j_{1}}-\boldsymbol{\zeta}^{*}_{j_{2}|j_{1}}\|\leq\|\boldsymbol{\zeta}_{i_{2}|j_{1}}-\boldsymbol{\zeta}^{*}_{\ell_{2}|j_{1}}\|,\forall\ell_{2}\neq j_{2}\},$ $$

with $\boldsymbol{\zeta}_{i_{2}|j_{1}}:=(\boldsymbol{\omega}_{i_{2}|j_{1}},\boldsymbol{\eta}_{j_{1}i_{2}},\tau_{j_{1}i_{2}},\nu_{j_{1}i_{2}})$ and $\boldsymbol{\zeta}^{*}_{j_{2}|j_{1}}:=(\boldsymbol{\omega}^{*}_{j_{2}|j_{1}},\boldsymbol{\eta}^{*}_{j_{2}|j_{1}},\tau^{*}_{j_{1}j_{2}},\nu^{*}_{j_{1}j_{2}})$. Note that when the MLE $\widehat{G}_{n}$ is sufficiently close to its true counterpart $G_{*}$, since the value of $k^{*}_{1}$ is known, we have $|\mathcal{V}_{j_{1}}(\widehat{G}_{n})|=1$ for any $j_{1}\in[k^{*}_{1}]$, meaning that each parameter $a^{*}_{j_{1}}$ is fitted by exactly one parameter. On the other hand, as $k^{*}_{2}$ is unknown and we over-specify it by a larger value $k_{2}$, a Voronoi cell $\mathcal{V}_{j_{2}|j_{1}}$ could have more than one element. Furthermore, the cardinality of $\mathcal{V}_{j_{2}|j_{1}}$ is exactly the number of fitted parameters converging to $\zeta^{*}_{j_{2}|j_{1}}$. For instance, $|\mathcal{V}_{j_{2}|j_{1}}|=2$ indicates that $\zeta^{*}_{j_{2}|j_{1}}$ is fitted by two parameters.
Now, we define a Voronoi loss function based on the Voronoi cells as follows:

$$ $\displaystyle\mathcal{L}_{(r_{1},r_{2},r_{3})}(G,G_{*}):=\sum_{j_{1}=1}^{k^{*}_{1}}\Big{|}\sum_{i_{1}\in\mathcal{V}_{j_{1}}}\exp(b_{i_{1}})-\exp(b^{*}_{j_{1}})\Big{|}+\sum_{j_{1}=1}^{k^{*}_{1}}\sum_{i_{1}\in\mathcal{V}_{j_{1}}}\exp(b_{i_{1}})\|\Delta\boldsymbol{a}_{i_{1}j_{1}}\|$ $\displaystyle+$ $\displaystyle\sum_{j_{1}=1}^{k^{*}_{1}}\sum_{i_{1}\in\mathcal{V}_{j_{1}}}\exp(b_{i_{1}})\Bigg{[}\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|=1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{i_{2}|j_{1}})\Big{(}\|\Delta\boldsymbol{\omega}_{i_{2}j_{2}|j_{1}}\|+\|\Delta\boldsymbol{\eta}_{j_{1}i_{2}j_{2}}\|+|\Delta\tau_{j_{1}i_{2}j_{2}}|+|\Delta\nu_{j_{1}i_{2}j_{2}}|\Big{)}$ $\displaystyle+$ $\displaystyle\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|>1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{i_{2}|j_{1}})\Big{(}\|\Delta\boldsymbol{\omega}_{i_{2}j_{2}|j_{1}}\|^{2}+\|\Delta\boldsymbol{\eta}_{j_{1}i_{2}j_{2}}\|^{r_{1}(|\mathcal{V}_{j_{2}|j_{1}}|)}+|\Delta\tau_{j_{1}i_{2}j_{2}}|^{r_{2}(|\mathcal{V}_{j_{2}|j_{1}}|)}$ $\displaystyle+$ $\displaystyle|\Delta\nu_{j_{1}i_{2}j_{2}}|^{r_{3}(|\mathcal{V}_{j_{2}|j_{1}}|)}\Big{)}\Bigg{]}+\sum_{j_{1}=1}^{k^{*}_{1}}\sum_{i_{1}\in\mathcal{V}_{j_{1}}}\exp(b_{i_{1}})\sum_{j_{2}=1}^{k^{*}_{2}}\Big{|}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{i_{2}|j_{1}})-\exp(\beta^{*}_{j_{2}|j_{1}})\Big{|},$ (5) $$

where $r_{1},r_{2},r_{3}:\mathbb{N}\to\mathbb{N}$ are some integer-valued functions and we denote $\Delta\boldsymbol{a}_{i_{1}j_{1}}:=\boldsymbol{a}_{i_{1}}-\boldsymbol{a}^{*}_{j_{1}}$, $\Delta\boldsymbol{\omega}_{i_{2}j_{2}|j_{1}}:=\boldsymbol{\omega}_{i_{2}|j_{1}}-\boldsymbol{\omega}_{j_{2}|j_{1}}$, $\Delta\boldsymbol{\eta}_{j_{1}i_{2}j_{2}}:=\boldsymbol{\eta}_{j_{1}i_{2}}-\boldsymbol{\eta}^{*}_{j_{1}j_{2}}$, $\Delta\tau_{j_{1}i_{2}j_{2}}:=\tau_{j_{1}i_{2}}-\tau^{*}_{j_{1}j_{2}}$ and $\Delta\nu_{j_{1}i_{2}j_{2}}:=\nu_{j_{1}i_{2}}-\nu^{*}_{j_{1}j_{2}}$. Given the above loss function, we are ready to characterize the convergence behavior of expert estimation in the following theorem.

###### Theorem 2 .

The following Hellinger lower bounds hold true for any $G\in\mathcal{G}_{k^{*}_{1},k_{2}}(\Theta)$:

$$ $\displaystyle\mathbb{E}_{\boldsymbol{X}}[h(p^{SS}_{G}(\cdot|\boldsymbol{X}),p^{SS}_{G_{*}}(\cdot|\boldsymbol{X}))]$ $\displaystyle\gtrsim\mathcal{L}_{(\frac{1}{2}r^{SS},r^{SS},\frac{1}{2}r^{SS})}(G,G_{*}).$ $$

As a result, we obtain that $\mathcal{L}_{(\frac{1}{2}r^{SS},r^{SS},\frac{1}{2}r^{SS})}(\widehat{G}^{SS}_{n},G_{*})=\widetilde{\mathcal{O}}_{P}(n^{-1/2})$.

Proof of Theorem [2](#Thmtheorem2) is in Appendix [E](#A5). The above results together with the formulation of the Voronoi loss $\mathcal{L}_{(\frac{1}{2}r^{SS},r^{SS},\frac{1}{2}r^{SS})}$ in equation ([5](#S2.E5)) implies that

(i) Exact-specified parameters: The rates for estimating exact-specified parameters $\boldsymbol{a}^{*}_{j_{1}}$, $\boldsymbol{\omega}^{*}_{j_{2}|j_{1}}$, $\boldsymbol{\eta}^{*}_{j_{1}j_{2}}$, $\tau^{*}_{j_{1}j_{2}}$, $\nu^{*}_{j_{1}j_{2}}$ which are approached by exactly one fitted parameter, i.e. their Voronoi cells have only one element $|\mathcal{V}_{j_{1}}|=|\mathcal{V}_{j_{2}|j_{1}}|=1$, are parametric on the sample size $n$, standing at the order $\widetilde{\mathcal{O}}_{P}(n^{-1/2})$. Additionally, the gating bias parameters $\exp(b^{*}_{j_{1}})$ and $\exp(\beta^{*}_{j_{2}|j_{1}})$ also share the same parametric estimation rates.

(ii) Over-specified parameters: For over-specified parameters $\boldsymbol{\omega}^{*}_{j_{2}|j_{1}},\boldsymbol{\eta}^{*}_{j_{1}j_{2}},\tau^{*}_{j_{1}j_{2}},\nu^{*}_{j_{1}j_{2}}$ which are fitted by more than one parameter, i.e. $|\mathcal{V}_{j_{2}|j_{1}}|>1$, their estimation rates are not homogeneous. In particular, the rates for estimating $\boldsymbol{\omega}^{*}_{j_{2}|j_{1}}$ are of order $\widetilde{\mathcal{O}}_{P}(n^{-1/4})$. At the same time, those for $\boldsymbol{\eta}^{*}_{j_{1}j_{2}},\tau^{*}_{j_{1}j_{2}},\nu^{*}_{j_{1}j_{2}}$ depend on their number of fitted parameters $|\mathcal{V}_{j_{2}|j_{1}}|$ and the solvability of the polynomial equation system in equation ([4](#S2.E4)), standing at the orders of $\widetilde{\mathcal{O}}_{P}(n^{-1/r^{SS}(|\mathcal{V}_{j_{2}|j_{1}}|)})$, $\widetilde{\mathcal{O}}_{P}(n^{-1/2r^{SS}(|\mathcal{V}_{j_{2}|j_{1}}|)})$, $\widetilde{\mathcal{O}}_{P}(n^{-1/r^{SS}(|\mathcal{V}_{j_{2}|j_{1}}|)})$, respectively. For instance, when $|\mathcal{V}_{j_{2}|j_{1}}|=3$, these rates become $\widetilde{\mathcal{O}}_{P}(n^{-1/6}),\widetilde{\mathcal{O}}_{P}(n^{-1/12}),\widetilde{\mathcal{O}}_{P}(n^{-1/6})$, which are significantly slower than those for exact-specified parameters. These slow rates occur due to the interactions mentioned in the “Challenges” paragraph.

(iii) Expert estimation: Recall that expert specialization is an essential problem where we learn how fast an expert specializes in some tasks or some aspects of the data. Therefore, it is important to understand the convergence behavior of the expert estimation, particularly its data-dependent term $(\boldsymbol{\eta}^{*}_{j_{1}j_{2}})^{\top}\boldsymbol{x}$. According to the Cauchy-Schwarz inequality, we have

$$ $\displaystyle|(\hat{\boldsymbol{\eta}}^{SS,n}_{i_{1}i_{2}})^{\top}\boldsymbol{x}-(\boldsymbol{\eta}^{*}_{j_{1}j_{2}})^{\top}\boldsymbol{x}|\leq\|\hat{\boldsymbol{\eta}}^{SS,n}_{i_{1}i_{2}}-\boldsymbol{\eta}^{*}_{j_{1}j_{2}}\|\cdot\|\boldsymbol{x}\|,$ (6) $$

where $\hat{\boldsymbol{\eta}}^{SS,n}_{i_{1}i_{2}}$ is an MLE of $\boldsymbol{\eta}^{*}_{j_{1}j_{2}}$.
Since the input space is bounded and
from the estimation rate of $\boldsymbol{\eta}^{*}_{j_{1}j_{2}}$ in the above two remarks, we deduce that $(\boldsymbol{\eta}^{*}_{j_{1}j_{2}})^{\top}\boldsymbol{x}$ admits an estimation rate of order $\widetilde{\mathcal{O}}_{P}(n^{-1/2})$ when $|\mathcal{V}_{j_{2}|j_{1}}|=1$ or $\widetilde{\mathcal{O}}_{P}(n^{-1/r^{SS}(|\mathcal{V}_{j_{2}|j_{1}}|)})$ when $|\mathcal{V}_{j_{2}|j_{1}}|>1$. Note that the latter rate is significantly slow since the term $r^{SS}(|\mathcal{V}_{j_{2}|j_{1}}|)$ grows as the number of fitted experts $|\mathcal{V}_{j_{2}|j_{1}}|$ increases.

###### Theorem 1 .

###### Lemma 1 .

###### Theorem 2 .

### 2.2 Softmax-Laplace HMoE

Moving to this section, we study the effects of replacing the softmax gating in the second level with the Laplace gating on the convergence of expert estimation under the Gaussian HMoE. In particular, the conditional density function in equation ([1](#S2.E1)) becomes

$$ $\displaystyle p^{SL}_{G_{*}}(y|\boldsymbol{x})$ $\displaystyle:=\sum_{i_{1}=1}^{k^{*}_{1}}\sigma((\boldsymbol{a}_{i_{1}}^{*})^{\top}\boldsymbol{x}+b_{i_{1}}^{*})\sum_{i_{2}=1}^{k^{*}_{2}}\sigma(-\|\boldsymbol{\omega}_{i_{2}|i_{1}}^{*}-\boldsymbol{x}\|+\beta_{i_{2}|i_{1}}^{*})\pi(y|(\boldsymbol{\eta}_{i_{1}i_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{i_{1}i_{2}}^{*},\nu_{i_{1}i_{2}}^{*}),$ (7) $$

where the abbreviation $SL$ stands for “Softmax-Laplace”. Additionally, the MLE under this setting, denoted by $\widehat{G}^{SL}_{n}$, is determined similarly to that in equation ([2](#S2.E2)).
The main difference between the density $p^{SL}_{G_{*}}(y|\boldsymbol{x})$ from its counterpart $p^{SS}_{G_{*}}(y|\boldsymbol{x})$ is the Laplace gating function $\sigma(-\|\boldsymbol{\omega}_{i_{2}|i_{1}}^{*}-\boldsymbol{x}\|+\beta_{i_{2}|i_{1}}^{*})$ in the second level. Due to this gating change, the interaction between parameters $\boldsymbol{a}$ and $\boldsymbol{\omega}$ via the PDE $\frac{\partial u}{\partial\boldsymbol{a}}=\frac{\partial u}{\partial\boldsymbol{\omega}}$ in equation ([3](#S2.E3)) no longer holds true, while others still exist. As a consequence, we only need to consider a simpler (fewer variables) system of polynomial equations than that in equation ([4](#S2.E4)). More specifically, for each $m\geq 2$, we define $r^{SL}(m)$ as the smallest natural number $r$ such that the following system does not have any non-trivial solutions for the unknown variables $(p_{i_{2}},\boldsymbol{q}_{2},\boldsymbol{q}_{3i_{2}},q_{4i_{2}},q_{5i_{2}})_{i_{2}=1}^{m}$:

$$ $\displaystyle\sum_{i_{2}=1}^{m}\sum_{(\boldsymbol{\alpha}_{2},\boldsymbol{\alpha}_{3},\alpha_{4},\alpha_{5})\in\mathcal{I}^{SL}_{\boldsymbol{\rho}_{1},\rho_{2}}}\frac{1}{\boldsymbol{\alpha}!}\cdot p^{2}_{i_{2}}\boldsymbol{q}_{2}^{\boldsymbol{\alpha}_{2}}\boldsymbol{q}_{3i_{2}}^{\boldsymbol{\alpha}_{3}}q_{4i_{2}}^{\alpha_{4}}q_{5i_{2}}^{\alpha_{5}}=0,\quad 1\leq|\boldsymbol{\rho}_{1}|+\rho_{2}\leq r,$ (8) $$

where $\mathcal{I}^{SL}_{\boldsymbol{\rho}_{1},\rho_{2}}:=\{(\boldsymbol{\alpha}_{2},\boldsymbol{\alpha}_{3},\alpha_{4},\alpha_{5})\in\mathbb{R}^{d}\times\mathbb{R}^{d}\times\mathbb{R}\times\mathbb{R}_{+}:\boldsymbol{\alpha}_{2}+\boldsymbol{\alpha}_{3}=\boldsymbol{\rho}_{1},|\boldsymbol{\alpha}_{3}|+\alpha_{4}+2\alpha_{5}=\rho_{2}\}$. Here, a solution is called non-trivial if all the values of $p_{i_{2}}$ are different from zero and at least one among $q_{4i_{2}}$ is non-zero. This system has been considered in [] where they show that $r^{SL}(2)=4$ and $r^{SL}(3)=6$.
We observe that the function $r^{SL}$ admits identical behavior to the function $r^{SS}$ in Lemma [1](#Thmlemma1) at some particular points. Nevertheless, it is challenging to make an explicit comparison between these two functions, which requires further technical tools in algebraic geometry [] to be developed.

Next, note that we can achieve the density estimation rate $\mathbb{E}_{\boldsymbol{X}}[h(p^{SL}_{\widehat{G}^{SL}_{n}}(\cdot|\boldsymbol{X}),p^{SL}_{G_{*}}(\cdot|\boldsymbol{X}))]=\widetilde{\mathcal{O}}_{P}(n^{-1/2})$ using similar arguments for Theorem [1](#Thmtheorem1) (see Appendix [F](#A6)). Thus, we will present only the convergence of parameter and expert estimation under the setting of this section in Theorem [3](#Thmtheorem3).

###### Theorem 3 .

The following Hellinger lower bounds hold true for any $G\in\mathcal{G}_{k^{*}_{1},k_{2}}(\Theta)$:

$$ $\displaystyle\mathbb{E}_{\boldsymbol{X}}[h(p^{SL}_{G}(\cdot|\boldsymbol{X}),p^{SL}_{G_{*}}(\cdot|\boldsymbol{X}))]$ $\displaystyle\gtrsim\mathcal{L}_{(\frac{1}{2}r^{SL},r^{SL},\frac{1}{2}r^{SL})}(G,G_{*}).$ $$

As a result, we obtain that $\mathcal{L}_{(\frac{1}{2}r^{SL},r^{SL},\frac{1}{2}r^{SL})}(\widehat{G}^{SL}_{n},G_{*})=\widetilde{\mathcal{O}}_{P}(n^{-1/2})$.

Proof of Theorem [3](#Thmtheorem3) is in Appendix [E](#A5). From the above results, it can be seen that the parameter and expert estimation when using the softmax gating and Laplace gating in the first and second levels of the Gaussian HMoE share the same convergence behavior as those when using the softmax gating in both levels in Theorem [2](#Thmtheorem2). In particular, by arguing analogously to equation ([6](#S2.E6)), we get that the data-dependent term of expert $(\boldsymbol{\eta}^{*}_{j_{1}j_{2}})^{\top}\boldsymbol{x}$ has an estimation rate of order $\widetilde{\mathcal{O}}_{P}(n^{-1/2})$ when $|\mathcal{V}_{j_{2}|j_{1}}|=1$ or $\widetilde{\mathcal{O}}_{P}(n^{-1/r^{SL}(|\mathcal{V}_{j_{2}|j_{1}}|)})$ when $|\mathcal{V}_{j_{2}|j_{1}}|>1$. Thus, we can see that substituting the softmax gating with the Laplace gating in the second level is not enough to accelerate the expert estimation rate (see Table [1](#S2.T1)). This is because the interaction $\frac{\partial u}{\partial\boldsymbol{\eta}}=\frac{\partial^{2}u}{\partial\boldsymbol{a}\partial\tau}$ between $\boldsymbol{\eta}$ and other parameters in equation ([3](#S2.E3)) still occurs under the setting of softmax-Laplace gating Gaussian HMoE.

###### Theorem 3 .

### 2.3 Laplace-Laplace HMoE

In this section, we consider the two-level Gaussian HMoE equipped with the Laplace gating in both levels. More specifically, the conditional density function in equation ([7](#S2.E7)) turns into

$$ $\displaystyle p^{LL}_{G_{*}}(y|\boldsymbol{x})$ $\displaystyle:=\sum_{i_{1}=1}^{k^{*}_{1}}\sigma(-\|\boldsymbol{a}_{i_{1}}^{*}-\boldsymbol{x}\|+b_{i_{1}}^{*})\sum_{i_{2}=1}^{k^{*}_{2}}\sigma(-\|\boldsymbol{\omega}_{i_{2}|i_{1}}^{*}-\boldsymbol{x}\|+\beta_{i_{2}|i_{1}}^{*})\pi(y|(\boldsymbol{\eta}_{i_{1}i_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{i_{1}i_{2}}^{*},\nu_{i_{1}i_{2}}^{*}),$ (9) $$

where the abbreviation $LL$ stands for “Laplace-Laplace”. Furthermore, the definition of the MLE under this setting, denoted by $\widehat{G}^{LL}_{n}$, is determined similarly to that in equation ([2](#S2.E2)). Under this setting, the first-level softmax gating $\sigma((\boldsymbol{a}_{i_{1}}^{*})^{\top}\boldsymbol{x}+b_{i_{1}}^{*})$ used in previous sections is replaced with the Laplace gating $\sigma(-\|\boldsymbol{a}_{i_{1}}^{*}-\boldsymbol{x}\|+b_{i_{1}}^{*})$, leading to the disappearance of the interaction $\frac{\partial u}{\partial\boldsymbol{\eta}}=\frac{\partial^{2}u}{\partial\boldsymbol{a}\partial\tau}$ between $\boldsymbol{\eta}$ and other parameters mentioned in equation ([3](#S2.E3)). Therefore, we only need to cope with $\frac{\partial u}{\partial\nu}=\frac{1}{2}\cdot\frac{\partial^{2}u}{\partial\tau^{2}}$ as in []. Consequently, it is sufficient to take account of the following system of polynomial equations with substantially fewer variables than those in equations ([4](#S2.E4)) and ([8](#S2.E8)). In particular, for each $m\geq 2$, we define $r^{LL}(m)$ as the smallest natural number $r$ such that the following system does not have any non-trivial solutions for the unknown variables $(p_{i_{2}},q_{4i_{2}},q_{5i_{2}})_{i_{2}=1}^{m}$:

$$ $\displaystyle\sum_{i_{2}=1}^{m}\sum_{(\alpha_{4},\alpha_{5})\in\mathcal{I}^{LL}_{\rho}}\frac{1}{\boldsymbol{\alpha}!}\cdot p^{2}_{i_{2}}q_{4i_{2}}^{\alpha_{4}}q_{5i_{2}}^{\alpha_{5}}=0,\quad 1\leq\rho\leq r,$ (10) $$

where $\mathcal{I}^{LL}_{\rho}:=\{(\alpha_{4},\alpha_{5})\in\mathbb{R}\times\mathbb{R}_{+}:\alpha_{4}+2\alpha_{5}=\rho\}$. Here, a solution is called non-trivial if all the values of $p_{i_{2}}$ are different from zero and at least one among $q_{4i_{2}}$ is non-zero. The above system has been studied in [] which show that $r^{LL}(2)=4$ and $r^{LL}(3)=6$. These values are similar to those of the aforementioned functions $r^{SS}$ and $r^{SL}$.

As demonstrated in Appendix [F](#A6), we also obtain the convergence rate of density estimation $\mathbb{E}_{\boldsymbol{X}}[h(p^{LL}_{\widehat{G}^{LL}_{n}}(\cdot|\boldsymbol{X}),p^{LL}_{G_{*}}(\cdot|\boldsymbol{X}))]=\widetilde{\mathcal{O}}_{P}(n^{-1/2})$ under this setting. Given that result, we are ready to investigate the impacts of using the Laplace gating in both levels on the convergence behavior of parameter and expert estimation in the below theorem.

###### Theorem 4 .

The following Hellinger lower bounds hold true for any $G\in\mathcal{G}_{k^{*}_{1},k_{2}}(\Theta)$:

$$ $\displaystyle\mathbb{E}_{\boldsymbol{X}}[h(p^{LL}_{G}(\cdot|\boldsymbol{X}),p^{LL}_{G_{*}}(\cdot|\boldsymbol{X}))]$ $\displaystyle\gtrsim\mathcal{L}_{(2,r^{LL},\frac{1}{2}r^{LL})}(G,G_{*}).$ $$

As a result, we obtain that $\mathcal{L}_{(2,r^{LL},\frac{1}{2}r^{LL})}(\widehat{G}^{LL}_{n},G_{*})=\widetilde{\mathcal{O}}_{P}(n^{-1/2})$.

Proof of Theorem [4](#Thmtheorem4) is in Appendix [E](#A5). From the formulation of the loss function $\mathcal{L}_{(2,r^{LL},\frac{1}{2}r^{LL})}$ in equation ([5](#S2.E5)), we observe that all the parameter estimations share the same convergence behavior as those under the previous two settings, except for the estimation of $\boldsymbol{\eta}^{*}_{j_{1}j_{2}}$ which enjoys a convergence rate of order $\widetilde{\mathcal{O}}_{P}(n^{-1/2})$ when $|\mathcal{V}_{j_{2}|j_{1}}|=1$ or $\widetilde{\mathcal{O}}_{P}(n^{-1/4})$ when $|\mathcal{V}_{j_{2}|j_{1}}|>1$. By employing the same arguments as in equation ([6](#S2.E6)), we deduce that the data-dependent term of expert $(\boldsymbol{\eta}^{*}_{j_{1}j_{2}})^{\top}\boldsymbol{x}$ also admits these rates. Compared to those when using the softmax gating in either level or both levels, the expert estimation rates when using the Laplace gating in both levels are improved significantly as they no longer depend on the term $r^{LL}(|\mathcal{V}_{j_{2}|j_{1}}|)$ (see Table [1](#S2.T1)). This rate acceleration occurs since the interaction $\frac{\partial u}{\partial\boldsymbol{\eta}}=\frac{\partial^{2}u}{\partial\boldsymbol{a}\partial\tau}$ between $\boldsymbol{\eta}$ and other parameters mentioned in equation ([3](#S2.E3)) does not exist under this setting. As a result, we claim that the convergence of expert estimation under the two-level Gaussian HMoE is benefited the most when equipped with the Laplace gating in both levels.

**Table 1: Summary of estimation rates for the data-dependent term $(\boldsymbol{\eta}^{*}_{j_{1}j_{2}})^{\top}\boldsymbol{x}$ in experts. Below, experts are called exact-specified when $|\mathcal{V}_{j_{2}|j_{1}}|=1$ and over-specified when $|\mathcal{V}_{j_{2}|j_{1}}|>1$.**
| Gating | Softmax-Softmax | Softmax-Laplace | Laplace-Laplace |  |
| --- | --- | --- | --- | --- |
| Expert estimation rates | Exact-specified | $\widetilde{\mathcal{O}}_{P}(n^{-1/2})$ | $\widetilde{\mathcal{O}}_{P}(n^{-1/2})$ | $\widetilde{\mathcal{O}}_{P}(n^{-1/2})$ |
| Over-specified | $\widetilde{\mathcal{O}}_{P}(n^{-1/r^{SS}(|\mathcal{V}_{j_{2}|j_{1}}|)})$ | $\widetilde{\mathcal{O}}_{P}(n^{-1/r^{SL}(|\mathcal{V}_{j_{2}|j_{1}}|)})$ | $\boldsymbol{\widetilde{\mathcal{O}}_{P}(n^{-1/4})}$ |  |

###### Theorem 4 .

## 3 Experiments

In this section, we empirically demonstrate the effects of employing various combinations of gating functions in HMoE to validate our theoretical findings and discuss empirical insights. First, we show that HMoE outperforms standard MoE and other alternatives, particularly in cases with inherent subgroups or multilevel structures, where HMoE excels. We then conduct comprehensive ablation studies to analyze the impact of different gating function combinations and perform case studies across various scenarios. Beyond performance improvements, these experiments provide valuable insights into how different gating function combinations influence the distribution of input modules, offering explanations for the performance variations observed with different gating configurations.

**Table 2: Comparison of HMoE-based methods (gray) and baselines, utilizing vital signs and clinical notes of MIMIC-IV []. The best results are highlighted in bold font, and the second-best results are underlined. All results are averaged across 5 random experiments.**
|  | MulT | MAG | TFN | HAIM | MISTS | MoE | HMoE |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 48-IHM | AUROC | 75.56 $\pm$ 0.34 | 79.36 $\pm$ 0.25 | 79.12 $\pm$ 0.56 | 78.87 $\pm$ 0.00 | 77.23 $\pm$ 0.82 | 83.13 $\pm$ 0.36 | 85.59 $\pm$ 0.44 |
| F1 | 38.65 $\pm$ 0.25 | 40.87 $\pm$ 0.17 | 40.96 $\pm$ 0.37 | 39.78 $\pm$ 0.00 | 45.98 $\pm$ 0.49 | 46.82 $\pm$ 0.28 | 47.57 $\pm$ 0.32 |  |
| LOS | AUROC | 82.12 $\pm$ 0.98 | 81.94 $\pm$ 0.36 | 81.65 $\pm$ 0.43 | 82.46 $\pm$ 0.00 | 80.34 $\pm$ 0.61 | 83.76 $\pm$ 0.59 | 86.26 $\pm$ 0.61 |
| F1 | 73.16 $\pm$ 0.51 | 72.78 $\pm$ 0.22 | 73.89 $\pm$ 0.52 | 72.75 $\pm$ 0.00 | 73.22 $\pm$ 0.43 | 74.32 $\pm$ 0.44 | 76.07 $\pm$ 0.29 |  |
| 25-PHE | AUROC | 70.41 $\pm$ 0.44 | 71.17 $\pm$ 0.36 | 72.26 $\pm$ 0.27 | 63.57 $\pm$ 0.00 | 71.49 $\pm$ 0.59 | 73.87 $\pm$ 0.71 | 73.81 $\pm$ 0.51 |
| F1 | 32.33 $\pm$ 0.62 | 32.86 $\pm$ 0.19 | 34.24 $\pm$ 0.14 | 42.80 $\pm$ 0.00 | 33.29 $\pm$ 0.23 | 35.96 $\pm$ 0.23 | 35.64 $\pm$ 0.18 |  |
|  |  |  |  |  |  |  |  |  |

HMoE Implementation.
We implement the two-level HMoE module, inspired by []. Algorithm [1](#alg1) in Appendix outlines the procedure, which employs a recursive computation strategy to process inputs in a coarse-to-fine manner. The inputs are first partitioned by the outer dispatcher, followed by the inner dispatcher, into subgroups, which are then sent to specialized groups and experts for independent processing. The outputs from the experts are recursively combined using inner and outer combination tensors to produce the final output. Gating losses from both levels are integrated and scaled to regularize training, promoting balanced expert utilization.

### 3.1 Primary Results

HMoE Improves Multimodal Fusion.
We first evaluate the effectiveness of HMoE on the MIMIC-IV dataset, a comprehensive database containing records from nearly 300k patients admitted to a medical center between 2008 and 2019, focusing on a subset of 73,181 ICU stays. We integrated diverse patient modalities, including vital signs (time series), clinical notes, and CXR (chest X-ray images). Our tasks of interest in the MIMIC dataset include 48-hour in-hospital mortality prediction (48-IHM), 25-type phenotype classification (25-PHE), and length-of-stay (LOS) prediction. The baselines include: (1) the Multimodal Transformer (MulT), which models modality interactions []; (2) the Multimodal Adaptation Gate (MAG), which addresses consistency and differences across modalities []; (3) the early fusion method Tensor Fusion Network (TFN) []; (4) the HAIM data pipeline [], specifically designed for integrating multimodal data from MIMIC-IV; (5) MISTS, a cross-attention approach combined with irregular sequence modeling []; and (6) multimodal fusion using MoE []. The data is first processed by modality-specific encoders, with the obtained modality embeddings then fed into 12 stacked HMoE modules with residual connections to produce the outcome. Details of the building blocks are provided in the appendix.
Table [2](#S3.T2) presents the outcomes of integrating time series, clinical notes, and CXR data into various prediction tasks. The HMoE (Laplace-Laplace) outperforms the baselines in most scenarios, often by a significant margin. While the MoE-based fusion method [] has proven effective in multimodal fusion, the inherent hierarchical structure of the HMoE module further enhances its ability to process multimodal inputs, allowing for more specialized expert assignment and improved performance.

HMoE Enhances Clinical Latent Domain Discovery.
Many datasets in high-stakes applications can be categorized into different latent domains. For instance, in clinical prediction tasks, patients can be grouped based on latent domains such as age, medical history, treatment, and symptoms. Training a generic model on heterogeneous patient data is often less effective than using a domain-specific model, as demonstrated by the SLDG method proposed by []. However, SLDG assigns a fixed classifier to each domain without considering the interactions between them. Moreover, it relies heavily on a separate hierarchical clustering process, which is separated from model training and limits input data to low-dimensional forms like short time series, failing to utilize a broader range of patient modalities. We extend this framework by evaluating HMoE for latent domain modeling tasks, using the HMoE module as a substitute for domain-specific classifiers. The HMoE module partitions inputs based on the similarity-driven top-$k$ routing mechanism, allowing tokens from each patient sample to be shared across multiple inner and outer experts simultaneously. In addition to MIMIC-IV, we also evaluated our methods on the eICU dataset [], which covers over 139k patients admitted to ICUs across the United States between 2014 and 2015. We followed the experimental settings used by []. For predictive tasks, we tested our method on readmission prediction and mortality prediction, and included representative baselines: Oracle (trained directly on the target test data), Base (trained solely on the source training data), as well as domain generalization methods that require domain IDs: DANN [] and MLDG [], and those that do not require IDs: IRM []. Table [3](#S3.T3) presents the results for both datasets. Among all the tested methods, HMoE with the Softmax-Laplace gating combination achieved the best overall performance on both tasks. Given HMoE’s advantage in processing multimodal inputs, we further added clinical notes and CXR modalities to the MIMIC-IV dataset (HMoE-M in Table [3](#S3.T3)), which led to additional performance improvements thanks to the joint benefit of customized modeling and the inclusion of extra modality information.

**Table 3: We apply HMoE to multi-domain and multi-modal patient data. HMoE delivers customized predictions for each group, while effectively accounting for the interactions and uniqueness of each group. This approach greatly improves results compared to current state-of-the-art methods.**
| Dataset | eICU | MIMIC-IV |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Task | Readmission | Mortality | Readmission | Mortality |  |  |  |  |
| Metric | AUPRC | AUROC | AUPRC | AUROC | AUPRC | AUROC | AUPRC | AUROC |
| Oracle | 21.92 $\pm$ 0.15 | 67.72 $\pm$ 0.42 | 27.14 $\pm$ 0.06 | 83.87 $\pm$ 0.57 | 28.21 $\pm$ 0.34 | 69.31 $\pm$ 0.53 | 42.83 $\pm$ 0.48 | 89.82 $\pm$ 0.75 |
| Base | 10.41 $\pm$ 0.12 | 51.01 $\pm$ 0.31 | 23.02 $\pm$ 0.24 | 80.31 $\pm$ 0.43 | 23.70 $\pm$ 0.23 | 66.54 $\pm$ 0.41 | 37.40 $\pm$ 0.20 | 86.10 $\pm$ 0.64 |
| DANN | 13.50 $\pm$ 0.09 | 53.79 $\pm$ 0.19 | 24.47 $\pm$ 0.08 | 80.82 $\pm$ 0.27 | 24.68 $\pm$ 0..09 | 67.31 $\pm$ 0.33 | 38.01 $\pm$ 0.17 | 87.34 $\pm$ 0.39 |
| MLDG | 10.41 $\pm$ 0.07 | 52.54 $\pm$ 0.43 | 22.41 $\pm$ 0.12 | 79.73 $\pm$ 0.39 | 20.50 $\pm$ 0.14 | 63.72 $\pm$ 0.29 | 35.98 $\pm$ 0.31 | 85.72 $\pm$ 0.68 |
| IRM | 13.62 $\pm$ 0.13 | 53.78 $\pm$ 0.22 | 25.18 $\pm$ 0.09 | 80.09 $\pm$ 0.47 | 24.23 $\pm$ 0.21 | 66.80 $\pm$ 0.22 | 38.72 $\pm$ 0.19 | 87.59 $\pm$ 0.43 |
| SLDG | 18.57 $\pm$ 0.10 | 62.30 $\pm$ 0.46 | 26.79 $\pm$ 0.16 | 82.44 $\pm$ 0.19 | 27.41 $\pm$ 0.10 | 69.02 $\pm$ 0.40 | 41.56 $\pm$ 0.12 | 89.85 $\pm$ 0.59 |
| HMoE | 19.39 $\pm$ 0.05 | 63.61 $\pm$ 0.23 | 26.60 $\pm$ 0.08 | 81.92 $\pm$ 0.28 | 27.82 $\pm$ 0.24 | 69.13 $\pm$ 0.21 | 42.23 $\pm$ 0.32 | 89.47 $\pm$ 0.18 |
| HMoE-M | - | - | - | - | 27.97 $\pm$ 0.18 | 69.19 $\pm$ 0.26 | 42.47 $\pm$ 0.35 | 89.65 $\pm$ 0.13 |
|  |  |  |  |  |  |  |  |  |

Figure: Figure 2: We evaluate the impact of using different gating function combinations in HMoE and compare it with standard MoE on (a) CIFAR-10 and (b) ImageNet. First, we present the results of one-layer MoE models (left side of each figure), where the model contains only the module of that specific setting. For the one-layer results, we use Tiny-ImageNet as a substitute for the full ImageNet. Next, we integrate these MoE modules into the state-of-the-art Vision MoE model (right) [] and compare the performance on the full datasets.
Refer to caption: /html/2410.02935/assets/fig/CIFAR-10-Dataset.png

### 3.2 Quantatitive Analysis

Combinations of Different Gating Mechanisms.
Figure [2](#S3.F2) compares the performance of different gating function combinations on the commonly used CIFAR-10 and ImageNet datasets. We first evaluate a single module (i.e., a one-layer MoE model) on CIFAR-10 and Tiny-ImageNet, followed by integrating these modules into the Vision-MoE framework []: in the Vision Transformer (ViT) models, we selectively replace an even number of FFN layers with targeted MoE layers and test the models on the full datasets. The performance gap between different gating functions is more pronounced in the one-layer MoE models due to the amplified effect of the module differences, while the difference becomes smaller after incorporating them into Vision MoE. The results show that the Laplace-Laplace gating combination achieves the best performance, while the combination of Laplace and Softmax gating also yields competitive results. Overall, HMoE demonstrates its potential to enhance the capacity of image classification models.

Multimodal Routing Distributions.
We then analyze how modality tokens are distributed across different experts and groups. Figure [3](#S3.F3) displays the distribution of three modality tokens in the best-performing HMoE block for corresponding tasks from the MIMIC-IV dataset. The HMoE module consists of two expert groups, each containing four experts. The results are taken from the final HMoE block of the trained model, using the first batch of data. Most vital signs and clinical notes tokens are routed to expert group 1, while CXR tokens are predominantly routed to expert group 2. For tasks (a) and (b), vital signs and clinical notes contribute more heavily to the overall HMoE prediction, particularly in task (b). However, for task (c), CXR tokens play a more significant role, contributing almost as much as vital signs, despite being present in smaller quantities. Additionally, due to the load-balancing loss applied during training, the total token count is nearly uniformly distributed among experts, with minimal token dropping because of exceeding capacity limits.

Distribution of Clinical Events.
Given that the number of clinical event categories is much larger than the number of modalities, it is more intuitive to visualize the impact of different gating function combinations on the distribution of clinical events. Figure [4](#S3.F4) (a) illustrates the routing distribution for the most commonly observed clinical events using the best-performing Softmax-Laplace gating combination of HMoE in latent domain discovery, compared to the Softmax gating function. The results indicate that the Softmax-Laplace combination promotes greater diversification in routing clinical event samples to experts while encouraging expert sharing across different categories. We further conduct ablation studies by varying the number of inner and outer experts in the best-performing HMoE across four tasks, as shown in Figure [4](#S3.F4) (b) and (c), where their number of outer and inner experts is fixed at 2 and 4, respectively. The results demonstrate that increasing the number of experts has a positive impact on performance, particularly for inner experts, though this improvement comes with an increase in computational demands.

Figure: Figure 3: Token distribution (time series, CXR, clinical notes) of HMoE blocks of a multimodal transformer. We present the best-performing gating combinations for three tasks evaluated on MIMIC-IV, where the HMoE block comprises 2 outer expert groups, each containing 4 inner experts. Expert IDs 1 to 4 (left section of each figure) represent token distributions from expert group 1, and expert IDs 5 to 8 (middle section) represent token distributions from expert group 2. The right section shows the relative weights assigned to each expert group.
Refer to caption: /html/2410.02935/assets/x2.png

Figure: Figure 4: (a) Distribution of top clinical events across expert IDs under heterogeneous versus homogeneous gating functions. (b)/(c) Performance variations as the number of inner/outer experts increases.
Refer to caption: /html/2410.02935/assets/fig/latent_domain_dist.png

## 4 Discussions and Limitations

In this work, we explored diverse gating function combinations beyond Softmax in a two-level hierarchical mixture of experts (HMoE). Our theoretical analysis demonstrated that using Laplace gating in HMoE improves convergence behavior, and employing Laplace gating at both levels significantly optimizes performance. We validated this theoretical finding on multiple real-world tasks, while also showcasing the effectiveness of HMoE in handling complex inputs, such as multimodal and multidomain data. However, the enhanced ability to process complex inputs comes with increased computational demands, which is a key limitation of HMoE. For future work, we plan to explore techniques like pruning to reduce computational costs in large-scale multimodal tasks and to identify more suitable downstream applications for HMoE.

Supplementary to
“On Expert Estimation in Hierarchical Mixture of Experts: Beyond Softmax Gating Functions”

In this supplementary material, we first introduce some related works to this paper in Appendix [A](#A1). The dataset information, preprocessing procedures, and implementation details can be found in Appendices [B](#A2), [C](#A3), and [D](#A4), respectively. Next, we provide the proof for the convergence of expert estimation in Appendix [E](#A5), while that for the convergence of density estimation is presented in Appendix [F](#A6). Then, we continue to streamline the proof of Lemma [1](#Thmlemma1) in Appendix [G](#A7) before investigating the identifiability of the Gaussian HMoE in Appendix [H](#A8).

## Appendix A Related Works

MoE [, ] has gained significant popularity for managing complex tasks since its introduction three decades ago. Unlike traditional models that reuse the same parameters for all inputs, MoE selects distinct parameters for each specific input. This results in a sparsely activated layer, enabling a substantial scaling of model capacity without a corresponding increase in computational cost. Recent studies [, , , , , ] have demonstrated the effectiveness of integrating MoE with cutting-edge models across a diverse range of tasks. [, , ] have also tackled key challenges such as accuracy and training instability. With the growing prevalence of MoE, the HMoE architecture has also been utilized to enhance model generalization performance in complex data structures. For instance, [] leveraged HMoE to more effectively manage hierarchical data, thereby improving classification accuracy in medical datasets. Similarly, [] introduced regularized HMoE models with embedded local feature selection, which enhanced model performance in high-dimensional scenarios. Due to its ability to assign input partitions to specialized experts, HMoE is particularly well-suited for multi-modal or multi-domain applications []. Prior research has demonstrated that HMoE can ensure robust generalization capabilities []. However, existing studies have primarily assessed HMoE in small-scale experiments and have not shown its effectiveness in large-scale real-world settings.

While MoE has been widely employed to scale up large models, its theoretical foundations have remained relatively underdeveloped. First of all, [] studied the maximum likelihood estimator for parameters of the MoE with each expert being a polynomial regression model. In particular, they investigated the convergence rate of the estimated density to the true density under the Kullback-Leibler (KL) divergence and gave some insights on how many experts should be chosen. Next, [] conducted a similar convergence analysis for input-free gating Gaussian MoE but using the Hellinger distance for the density estimation problem instead of the KL divergence. Additionally, they utilized the generalized Wasserstein distance to capture the parameter estimation rates which were negatively affected by the algebraic interactions among parameters.
[, , ] then generalized these results to a more popular setting known as softmax gating Gaussian MoE. Rather than leveraging the generalized Wasserstein distance for the parameter estimation problem, they proposed novel Voronoi-based loss functions which were shown to characterize the parameter estimation rates more accurately. Recently, [] advocated using a new Laplace gating function which induced faster convergence rates than the softmax gating functions due to a reduced number of parameter interactions. However, to the best of our knowledge, a comprehensive convergence analysis for HMoE has remained elusive in the literature.

## Appendix B Dataset Information

### B.1 MIMIC-IV

MIMIC-IV [] is a comprehensive database containing records from nearly 300,000 patients admitted to a medical center between 2008 and 2019, focusing on a subset of 73,181 ICU stays. We linked core ICU records, including lab results and vital signs, with corresponding chest X-rays [], radiological notes [], and electrocardiogram (ECG) data [] recorded during the same ICU stay.

Tasks of Interest.
We design an in-hospital mortality prediction task (referred to as 48-IHM) to assess our method’s capability in forecasting short-term patient deterioration. Additionally, accurately predicting patient discharge times is vital for improving patient outcomes and managing hospital resources efficiently [], leading us to implement the length-of-stay (LOS) task. Both the 48-IHM and LOS tasks are framed as binary classification problems, utilizing a 48-hour observation window (for patients staying at least 48 hours in the ICU) to predict in-hospital mortality (48-IHM) and patient discharge (without death) within the subsequent 48 hours (LOS). Moreover, recognizing the presence of specific acute care conditions in patient records is key for several clinical goals, such as forming cohorts for studies and identifying comorbidities []. Traditional approaches, which often rely on manual chart reviews or billing codes, are increasingly being complemented by machine learning models []. Automating this process demands high-accuracy classifications, which drives the development of our 25-type phenotype classification (25-PHE) task. This multilabel classification problem involves predicting one of 25 acute care conditions using data from the entire ICU stay. We summarize the details of these tasks below:

- •
48-IHM: This is a binary classification task where we aim to predict in-hospital mortality based on data collected during the first 48 hours of ICU admission, applicable only to patients who remained in the ICU for at least 48 hours.
- •
LOS: The length-of-stay task is structured similarly to 48-IHM. For patients who stayed in the ICU for a minimum of 48 hours, the objective is to predict whether they will be discharged (without death) within the next 48 hours.
- •
25-PHE: This multilabel classification task involves predicting one of 25 acute care conditions [, ], such as congestive heart failure, pneumonia, or shock, at the conclusion of each patient’s ICU stay. Since the original task was developed for diagnoses based on ICD-9 codes, and MIMIC-IV includes both ICD-9 and ICD-10 codes, we convert diagnoses coded in ICD-10 using the conversion database from [].

Evaluation.
We concentrated on patients with complete data across all modalities, which yielded a dataset of 8,770 ICU stays for the 48-IHM and LOS tasks, and 14,541 stays for the 25-PHE task. To assess the performance of the single-label tasks, 48-IHM and LOS, we utilize the F1-score and AUROC as our evaluation metrics. For the 25-PHE task, following prior research [, , ], we rely on macro-averaged F1-score and AUROC as the primary measures of evaluation. For the multimodal fusion task, we allocated 70% data for training, while the remaining 30% was evenly divided between validation and testing. For clinical latent domain discovery, similar to [], we segment the dataset into four temporal groups: 2008-2010, 2011-2013, 2014-2016, and 2017-2019. Each group is then divided into training, validation, and testing sets, following a 70%, 10%, and 20% split, respectively. Patients admitted after 2014 are treated as the target test data, while all earlier patients are used as the source training data.

### B.2 eICU

The eICU dataset [] includes over 200,000 visits from 139,000 patients admitted to ICUs in 208 hospitals across the United States. The data was gathered between 2014 and 2015. The 208 hospitals are categorized into four regions based on their geographic location: Midwest, Northeast, West, and South. We define our cohorts by excluding visits from patients younger than 18 or older than 89, as well as visits exceeding 10 days in length or containing fewer than 3 or more than 256 timestamps. Additionally, we omit visits shorter than 12 hours, since predictions are made 12 hours post-admission.

Tasks of Interest.
For the readmission task using the eICU dataset, our goal is to predict whether a patient will be readmitted within 15 days after discharge. Similar to the MIMIC-IV dataset, the mortality prediction task focuses on determining whether a patient will pass away following discharge.

Evaluation.
The eICU dataset is divided into four regional groups: Midwest, Northeast, West, and South. Each region is further split into 70% for training, 10% for validation, and 20% for testing. To assess the performance gap between regions, we compare the backbone model’s performance when trained on data from the same region versus data from other regions, as proposed by []. The region with the largest performance gap (Midwest) is selected as the target test data, while the remaining regions (Northeast, West, and South) are used as the source training data. To compare with baselines from [], we use the same evaluation metrics: Area Under the Precision-Recall Curve (AUPRC) and the Area Under the Receiver Operating Characteristic Curve (AUROC) scores.

### B.3 Image Classification Datasets

CIFAR-10.
CIFAR-10 [] is a well-known dataset in computer vision, commonly used for object recognition tasks. It contains 60,000 color images, each with a resolution of 32x32 pixels, representing one of 10 object categories (“plane,” “car,” “bird,” “cat,” “deer,” “dog,” “frog,” “horse,” “ship,” “truck”), with 6,000 images per class.

ImageNet.
We use the ImageNet database from ILSVRC2012 [], where the task is to classify images into 1,000 distinct categories, using a vast dataset of over 1.2 million training images and 150,000 validation and test images sourced from the ImageNet database.

Tiny-ImageNet.
The Tiny-ImageNet is a smaller, more manageable subset of the ImageNet dataset. It contains 100,000 images and 200 classes selected from full ImageNet dataset. All images are resized to 64×64 pixels to reduce computational demands.

## Appendix C Data Preprocessing for Clinical Tasks

During preprocessing, we selected 30 relevant lab and chart events from each patient’s ICU records to capture vital sign measurements. For chest X-rays, we employed a pre-trained DenseNet-121 model [], which had been fine-tuned on the CheXpert dataset [], to extract 1024-dimensional image embeddings. Additionally, we used the BioClinicalBERT model [] to generate 768-dimensional embeddings for the radiological notes.

Time Series.
We selected 30 time-series events for analysis, as outlined in []. This included nine vital signs: heart rate, mean/systolic/diastolic blood pressure, respiratory rate, oxygen saturation, and Glasgow Coma Scale (GCS) verbal, eye, and motor response. Additionally, 21 laboratory values were incorporated: potassium, sodium, chloride, creatinine, urea nitrogen, bicarbonate, anion gap, hemoglobin, hematocrit, magnesium, platelet count, phosphate, white blood cell count, total calcium, MCH, red blood cell count, MCHC, MCV, RDW, platelet count, neutrophil count, and vancomycin. Each time series value was standardized to have a mean of 0 and a standard deviation of 1, based on values from the training set. We use the Transformer as an encoder for time series data.

Chest X-Rays.
To integrate medical imaging into our analysis, we use the MIMIC-CXR-JPG module [] available through Physionet [], which contains 377,110 JPG images derived from the DICOM-based MIMIC-CXR database []. As described in [], each image is resized to 224 $\times$ 224 pixels, and we extract embeddings from the final layer of the DenseNet121 model. To identify X-rays taken during the patient’s ICU stay, we match subject IDs from MIMIC-CXR-JPG with the core MIMIC-IV database and then filter the X-rays to those captured between the ICU admission and discharge times.

Clinical Notes
To incorporate text data, we use the MIMIC-IV-Note module [], which includes 2,321,355 deidentified radiology reports for 237,427 patients. These reports can be linked to patients in the main MIMIC-IV dataset using a similar matching method as employed for chest X-rays. It is important to note that we were unable to access intermediate clinical notes (i.e., notes recorded by clinicians during the patient’s stay), as they have not yet been made publicly available. We extract note embeddings using the Bio-Clinical BERT model [].

## Appendix D Implementation Details

### D.1 Model Architecture

Once embeddings from each input modality or domain are generated, we address the issue of irregularity in the data. To do this, we use a discretized multi-time attention (mTAND) module [], which applies a time attention mechanism [] to convert irregularly sampled observations into discrete time intervals. This approach has been employed in previous works such as [, ]. The mTAND module transforms the irregular sequences into fixed-length representations, which are then passed into the MoE fusion layer with a residual connection. This fusion layer comprises multi-head self-attention followed by the HMoE module. In total, there are 12 MoE fusion layers, and the output from this layer is optimized using task-specific loss and load imbalance loss. We apply a dropout rate of 0.1 and use the Adam optimizer with a learning rate of 1e-4 and a weight decay of 1e-5. All models are trained for 100 epochs. For the multimodal experiment, we use a batch size of 2, while for the latent domain discovery experiment, the batch size is set to 256.

### D.2 HMoE Module

The detailed implementation procedure of the two-level HMoE module of the MoE fusion layer can be found in Algorithm [1](#alg1). We have also provided Python code as part of the supplementary material.

Figure: Algorithm 1 Computation Procedure for the 2-Level Hierarchical MoE Module

## Appendix E Proofs for Convergence of Expert Estimation

###### Proof of Theorems 2 , 3 and 4 .

Overview. We will focus on establishing the following inequality:

$$ $\displaystyle\inf_{G\in\mathcal{G}_{k^{*}_{1},k_{2}}(\Theta)}\mathbb{E}_{\boldsymbol{X}}[h(p^{type}_{G}(\cdot|\boldsymbol{X}),p^{type}_{G_{*}}(\cdot|\boldsymbol{X}))]/\mathcal{L}_{(r_{1},r_{2},r_{3})}(G,G_{*})>0,$ $$

where the value of $(r_{1},r_{2},r_{3})$ varies with the variable $type\in\{SS,SL,LL\}$. Note that the Hellinger distance $h$ is lower bounded by the Total Variation distance $V$, that is, $h\geq V$, it suffices to demonstrate that

$$ $\displaystyle\inf_{G\in\mathcal{G}_{k^{*}_{1},k_{2}}(\Theta)}\mathbb{E}_{\boldsymbol{X}}[V(p^{type}_{G}(\cdot|\boldsymbol{X}),p^{type}_{G_{*}}(\cdot|\boldsymbol{X}))]/\mathcal{L}_{(r_{1},r_{2},r_{3})}(G,G_{*})>0.$ (11) $$

To this end, we first show that

$$ $\displaystyle\lim_{\varepsilon\to 0}\inf_{G\in\mathcal{G}_{k^{*}_{1},k_{2}}(\Theta):\mathcal{L}_{(r_{1},r_{2},r_{3})}(G,G_{*})\leq\varepsilon}\mathbb{E}_{\boldsymbol{X}}[V(p^{type}_{G}(\cdot|\boldsymbol{X}),p^{type}_{G_{*}}(\cdot|\boldsymbol{X}))]/\mathcal{L}_{(r_{1},r_{2},r_{3})}(G,G_{*})>0.$ (12) $$

The proof of this result will be presented later. Now, suppose that it holds true, then there exists a positive constant $\varepsilon^{\prime}$ that satisfies

$$ $\displaystyle\inf_{G\in\mathcal{G}_{k^{*}_{1},k_{2}}(\Theta):\mathcal{L}_{1}(G,G_{*})\leq\varepsilon^{\prime}}\mathbb{E}_{\boldsymbol{X}}[V(p^{type}_{G}(\cdot|\boldsymbol{X}),p^{type}_{G_{*}}(\cdot|\boldsymbol{X}))]/\mathcal{L}_{(r_{1},r_{2},r_{3})}(G,G_{*})>0.$ $$

Thus, it suffices to establish the following inequality:

$$ $\displaystyle\inf_{G\in\mathcal{G}_{k^{*}_{1},k_{2}}(\Theta):\mathcal{L}_{1}(G,G_{*})>\varepsilon^{\prime}}\mathbb{E}_{\boldsymbol{X}}[V(p^{type}_{G}(\cdot|\boldsymbol{X}),p^{type}_{G_{*}}(\cdot|\boldsymbol{X}))]/\mathcal{L}_{(r_{1},r_{2},r_{3})}(G,G_{*})>0.$ (13) $$

Assume by contrary that the inequality ([13](#A5.E13)) does not hold true, then we can seek a sequence of mixing measures $G^{\prime}_{n}\in\mathcal{G}_{k^{*}_{1},k_{2}}(\Theta)$ that satisfy $\mathcal{L}_{1}(G^{\prime}_{n},G_{*})>\varepsilon^{\prime}$ and

$$ $\displaystyle\lim_{n\to\infty}\mathbb{E}_{\boldsymbol{X}}[V(p^{type}_{G^{\prime}_{n}}(\cdot|\boldsymbol{X}),p^{type}_{G_{*}}(\cdot|\boldsymbol{X}))]/\mathcal{L}_{(r_{1},r_{2},r_{3})}(G^{\prime}_{n},G_{*})=0.$ $$

Thus, we deduce that $\mathbb{E}_{\boldsymbol{X}}[V(p^{type}_{G^{\prime}_{n}}(\cdot|\boldsymbol{X}),p^{type}_{G_{*}}(\cdot|\boldsymbol{X}))]\to 0$ as $n\to\infty$. Since $\Theta$ is a compact set, we can substitute the sequence $(G^{\prime}_{n})$ by one of its subsequences that converges to a mixing measure $G^{\prime}\in\mathcal{G}_{k^{*}_{1},k_{2}}(\Theta)$. Recall that $\mathcal{L}_{(r_{1},r_{2},r_{3})}(G^{\prime}_{n},G_{*})>\varepsilon^{\prime}$, then we deduce that $\mathcal{L}_{(r_{1},r_{2},r_{3})}(G^{\prime},G_{*})>\varepsilon^{\prime}$.
By employing the Fatou’s lemma, it follows that

$$ $\displaystyle 0$ $\displaystyle=\lim_{n\to\infty}\mathbb{E}_{\boldsymbol{X}}[V(p^{type}_{G^{\prime}_{n}}(\cdot|\boldsymbol{X}),p^{type}_{G_{*}}(\cdot|\boldsymbol{X}))]/\mathcal{L}_{(r_{1},r_{2},r_{3})}(G^{\prime}_{n},G_{*})$ $\displaystyle\geq\frac{1}{2}\int\liminf_{n\to\infty}\Big{|}p^{type}_{G^{\prime}_{n}}(y|\boldsymbol{x})-p^{type}_{G_{*}}(y|\boldsymbol{x})\Big{|}^{2}~{}\mathrm{d}(\boldsymbol{x},y).$ $$

Thus, we obtain that $p^{type}_{G^{\prime}}(y|\boldsymbol{x})=p^{type}_{G_{*}}(y|\boldsymbol{x})$ for almost surely $(\boldsymbol{x},y)$. According to Proposition [1](#Thmproposition1), we get that $G^{\prime}\equiv G_{*}$, which yields that $\mathcal{L}_{(r_{1},r_{2},r_{3})}(G^{\prime},G_{*})=0$. This result contradicts the fact that $\mathcal{L}_{(r_{1},r_{2},r_{3})}(G^{\prime},G_{*})>\varepsilon^{\prime}>0$. Hence, we obtain the result in equation ([13](#A5.E13)), which together with the inequality ([12](#A5.E12)) leads to the conclusion in equation ([11](#A5.E11)).

Now, we are going back to the proof of the inequality ([12](#A5.E12)).

Proof of the inequality ([12](#A5.E12)) Suppose that the inequality ([12](#A5.E12)) does not hold, then we can find a sequence of mixing measures $(G_{n})$ in $\mathcal{G}_{k^{*}_{1},k_{2}}(\Theta)$ that satisfies $\mathcal{L}_{(r_{1},r_{2},r_{3})}(G_{n},G_{*})\to 0$ and

$$ $\displaystyle\mathbb{E}_{\boldsymbol{X}}[V(p^{type}_{G_{n}}(\cdot|\boldsymbol{X}),p^{type}_{G_{*}}(\cdot|\boldsymbol{X}))]/\mathcal{L}_{(r_{1},r_{2},r_{3})}(G_{n},G_{*})\to 0,$ (14) $$

as $n\to\infty$. For each $j_{1}\in[k^{*}_{1}]$, let $\mathcal{V}^{n}_{j_{1}}:=\mathcal{V}_{j_{1}}(G_{n})$ be a Voronoi cell of $G_{n}$ generated by the $j_{1}$-th components of $G_{*}$. As the Voronoi loss $\mathcal{V}^{n}_{j_{1}}$ has only one element and our arguments are asymptotic, we may assume WLOG that $\mathcal{V}^{n}_{j_{1}}=\mathcal{V}_{j_{1}}=\{j_{1}\}$ for any $j_{1}\in[k^{*}_{1}]$. Then, the Voronoi loss becomes

$$ $\displaystyle\mathcal{L}_{(r_{1},r_{2},r_{3})}(G_{n},G_{*})=\sum_{j_{1}=1}^{k^{*}_{1}}\Big{|}\exp(b^{n}_{j_{1}})-\exp(b^{*}_{j_{1}})\Big{|}+\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b^{n}_{j_{1}})\|\Delta\boldsymbol{a}_{j_{1}}^{n}\|+\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b^{n}_{j_{1}})$ $\displaystyle\times\Bigg{[}\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|=1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta^{n}_{i_{2}|j_{1}})\Big{(}\|\Delta\boldsymbol{\omega}_{i_{2}j_{2}|j_{1}}^{n}\|+\|\Delta\boldsymbol{\eta}_{j_{1}i_{2}j_{2}}^{n}\|+|\Delta\tau_{j_{1}i_{2}j_{2}}^{n}|+|\Delta\nu_{j_{1}i_{2}j_{2}}^{n}|\Big{)}$ $\displaystyle+\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|>1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta^{n}_{i_{2}|j_{1}})\Big{(}\|\Delta\boldsymbol{\omega}_{i_{2}j_{2}|j_{1}}^{n}\|^{2}+\|\Delta\boldsymbol{\eta}_{j_{1}i_{2}j_{2}}^{n}\|^{r_{1}}+|\Delta\tau_{j_{1}i_{2}j_{2}}^{n}|^{r_{2}}$ $\displaystyle+|\Delta\nu_{j_{1}i_{2}j_{2}}^{n}|^{r_{3}}\Big{)}\Bigg{]}+\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b^{n}_{j_{1}})\sum_{j_{2}=1}^{k^{*}_{2}}\Big{|}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta^{n}_{i_{2}|j_{1}})-\exp(\beta^{*}_{j_{2}|j_{1}})\Big{|}.$ (15) $$

Since $\mathcal{L}_{(r_{1},r_{2},r_{3})}(G_{n},G_{*})\to 0$ as $n\to\infty$, it follows that $\exp(b_{j_{1}}^{n})\to\exp(b_{j_{1}}^{*})$, $\boldsymbol{a}_{j_{1}}^{n}\to\boldsymbol{a}_{j_{1}}^{*}$, $\exp(\beta_{i_{2}|j_{1}}^{n})\to\exp(\beta_{j_{2}|j_{1}}^{*})$, $\boldsymbol{\omega}_{i_{2}|j_{1}}^{n}\to\boldsymbol{\omega}_{j_{2}|j_{1}}^{*}$, $\boldsymbol{\eta}_{j_{1}i_{2}}^{n}\to\boldsymbol{\eta}_{j_{1}j_{2}}^{*}$, $\tau_{j_{1}i_{2}}^{n}\to\tau_{j_{1}j_{2}}^{*}$ and $\nu_{j_{1}i_{2}}^{n}\to\nu_{j_{1}j_{2}}^{*}$ for all $j_{1}\in[k^{*}_{1}]$, $j_{2}\in[k^{*}_{2}]$ and $i_{2}\in\mathcal{V}_{j_{2}|j_{1}}$.

Subsequently, we consider three different settings where the variable $type$ takes the value in the set $\{SS,SL,LL\}$ in Appendices [E.1](#A5.SS1), [E.2](#A5.SS2) and [E.3](#A5.SS3), respectively. In each appendix, the proof will be divided into three main stages.

###### Proof of Theorems 2 , 3 and 4 .

### E.1 When t ​ y ​ p ​ e = S ​ S 𝑡 𝑦 𝑝 𝑒 𝑆 𝑆 type=SS

When $type=SS$, the corresponding Voronoi loss function is $\mathcal{L}_{(\frac{1}{2}r^{SS},r^{SS},\frac{1}{2}r^{SS})}(G_{n},G_{*})=\mathcal{L}_{1n}$ where we define

$$ $\displaystyle\mathcal{L}_{1n}:=\sum_{j_{1}=1}^{k^{*}_{1}}\Big{|}\exp(b^{n}_{j_{1}})-\exp(b^{*}_{j_{1}})\Big{|}+\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b^{n}_{j_{1}})\|\Delta\boldsymbol{a}_{j_{1}}^{n}\|+\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b^{n}_{j_{1}})$ $\displaystyle\times\Bigg{[}\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|=1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta^{n}_{i_{2}|j_{1}})\Big{(}\|\Delta\boldsymbol{\omega}_{i_{2}j_{2}|j_{1}}^{n}\|+\|\Delta\boldsymbol{\eta}_{j_{1}i_{2}j_{2}}^{n}\|+|\Delta\tau_{j_{1}i_{2}j_{2}}^{n}|+|\Delta\nu_{j_{1}i_{2}j_{2}}^{n}|\Big{)}$ $\displaystyle+\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|>1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta^{n}_{i_{2}|j_{1}})\Big{(}\|\Delta\boldsymbol{\omega}_{i_{2}j_{2}|j_{1}}^{n}\|^{2}+\|\Delta\boldsymbol{\eta}_{j_{1}i_{2}j_{2}}^{n}\|^{\frac{r^{SS}_{j_{2}|j_{1}}}{2}}+|\Delta\tau_{j_{1}i_{2}j_{2}}^{n}|^{r^{SS}_{j_{2}|j_{1}}}$ $\displaystyle+|\Delta\nu_{j_{1}i_{2}j_{2}}^{n}|^{\frac{r^{SS}_{j_{2}|j_{1}}}{2}}\Big{)}\Bigg{]}+\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b^{n}_{j_{1}})\sum_{j_{2}=1}^{k^{*}_{2}}\Big{|}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta^{n}_{i_{2}|j_{1}})-\exp(\beta^{*}_{j_{2}|j_{1}})\Big{|}.$ (16) $$

Step 1 - Taylor expansion: In this stage, we aim to decompose the term

$$ $\displaystyle Q_{n}:=\left[\sum_{j_{1}=1}^{k^{*}_{1}}\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x}+b_{j_{1}}^{*})\right][p^{SS}_{G_{n}}(y|\boldsymbol{x})-p^{SS}_{G_{*}}(y|\boldsymbol{x})]$ $$

into a combination of linearly independent terms using the Taylor expansion. For that purpose, let us denote

$$ $\displaystyle p^{SS,n}_{j_{1}}(y|\boldsymbol{x})$ $\displaystyle:=\sum_{j_{2}=1}^{k^{*}_{2}}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\sigma((\boldsymbol{\omega}_{i_{2}|j_{1}}^{n})^{\top}\boldsymbol{x}+\beta^{n}_{i_{2}|j_{1}})\pi(y|(\boldsymbol{\eta}_{j_{1}i_{2}}^{n})^{\top}\boldsymbol{x}+\tau_{j_{1}i_{2}}^{n},\nu_{j_{1}i_{2}}^{n}),$ $\displaystyle p^{SS,*}_{j_{1}}(y|\boldsymbol{x})$ $\displaystyle:=\sum_{j_{2}=1}^{k^{*}_{2}}\sigma((\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})^{\top}\boldsymbol{x}+\beta_{j_{2}|j_{1}}^{*})\pi(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*}).$ $$

Then, it can be checked that the quantity $Q_{n}$ is divided as

$$ $\displaystyle Q_{n}$ $\displaystyle=\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\left[\exp((\boldsymbol{a}_{j_{1}}^{n})^{\top}\boldsymbol{x})p^{SS,n}_{j_{1}}(y|\boldsymbol{x})-\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})p^{SS,*}_{j_{1}}(y|\boldsymbol{x})\right]$ $\displaystyle-\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\left[\exp((\boldsymbol{a}_{j_{1}}^{n})^{\top}\boldsymbol{x})-\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})\right]p^{SS}_{G_{n}}(y|\boldsymbol{x})$ $\displaystyle+\sum_{j_{1}=1}^{k^{*}_{1}}\left(\exp(b_{j_{1}}^{n})-\exp(b_{j_{1}}^{*})\right)\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})\left[p^{SS,n}_{j_{1}}(y|\boldsymbol{x})-p^{SS}_{G_{n}}(y|\boldsymbol{x})\right]$ $\displaystyle:$ $\displaystyle=A_{n}-B_{n}+C_{n}.$ (17) $$

Step 1A - Decompose $A_{n}$: Using the same techniques for decomposing $Q_{n}$, we can decompose $A_{n}$ as follows:

$$ $\displaystyle A_{n}$ $\displaystyle:=\sum_{j_{1}=1}^{k^{*}_{1}}\frac{\exp(b_{j_{1}}^{n})}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp((\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}})^{\top}\boldsymbol{x}+\beta^{*}_{j^{\prime}_{2}|j_{1}})}[A_{n,j_{1},1}+A_{n,j_{1},2}+A_{n,j_{1},3}],$ $$

where

$$ $\displaystyle A_{n,j_{1},1}$ $\displaystyle:=\sum_{j_{2}=1}^{k^{*}_{2}}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta^{n}_{i_{2}|j_{1}})\Big{[}\exp((\boldsymbol{\omega}_{i_{2}|j_{1}}^{n})^{\top}\boldsymbol{x})\exp((\boldsymbol{a}_{j_{1}}^{n})^{\top}\boldsymbol{x})\pi(y|(\boldsymbol{\eta}_{j_{1}i_{2}}^{n}){\top}\boldsymbol{x}+\tau_{j_{1}i_{2}}^{n},\nu_{j_{1}i_{2}}^{n})$ $\displaystyle\hskip 85.35826pt-\exp((\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})^{\top}\boldsymbol{x})\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})\pi(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})\Big{]},$ $\displaystyle A_{n,j_{1},2}$ $\displaystyle:=\sum_{j_{2}=1}^{k^{*}_{2}}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta^{n}_{i_{2}|j_{1}})\Big{[}\exp((\boldsymbol{\omega}_{i_{2}|j_{1}}^{n})^{\top}\boldsymbol{x})-\exp((\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})^{\top}\boldsymbol{x})\Big{]}$ $\displaystyle\hskip 199.16928pt\times\exp((\boldsymbol{a}_{j_{1}}^{n})^{\top}\boldsymbol{x})p^{SS,n}_{j_{1}}(y|\boldsymbol{x}),$ $\displaystyle A_{n,j_{1},3}$ $\displaystyle:=\sum_{j_{2}=1}^{k^{*}_{2}}\Big{(}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{i_{2}|j_{1}}^{n})-\exp(\beta_{j_{2}|j_{1}}^{*})\Big{)}\exp((\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})^{\top}\boldsymbol{x})$ $\displaystyle\hskip 28.45274pt\times[\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})\pi(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})-\exp((\boldsymbol{a}_{j_{1}}^{n})^{\top}\boldsymbol{x})p^{SS,n}_{j_{1}}(y|\boldsymbol{x})].$ $$

Based on the cardinality of the Voronoi cells $\mathcal{V}_{j_{2}|j_{1}}$, we continue to divide the term $A_{n,j_{1},1}$ into two parts as

$$ $\displaystyle A_{n,j_{1},1}$ $\displaystyle=\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|=1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{i_{2}|j_{1}}^{n})\Big{[}\exp((\boldsymbol{\omega}_{i_{2}|j_{1}}^{n})^{\top}\boldsymbol{x})\exp((\boldsymbol{a}_{j_{1}}^{n})^{\top}\boldsymbol{x})\pi(y|(\boldsymbol{\eta}_{j_{1}i_{2}}^{n}){\top}\boldsymbol{x}+\tau_{j_{1}i_{2}}^{n},\nu_{j_{1}i_{2}}^{n})$ $\displaystyle\hskip 85.35826pt-\exp((\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})^{\top}\boldsymbol{x})\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})\pi(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})\Big{]},$ $\displaystyle+\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|>1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{i_{2}|j_{1}}^{n})\Big{[}\exp((\boldsymbol{\omega}_{i_{2}|j_{1}}^{n})^{\top}\boldsymbol{x})\exp((\boldsymbol{a}_{j_{1}}^{n})^{\top}\boldsymbol{x})\pi(y|(\boldsymbol{\eta}_{j_{1}i_{2}}^{n}){\top}\boldsymbol{x}+\tau_{j_{1}i_{2}}^{n},\nu_{j_{1}i_{2}}^{n})$ $\displaystyle\hskip 85.35826pt-\exp((\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})^{\top}\boldsymbol{x})\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})\pi(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})\Big{]}$ $\displaystyle:$ $\displaystyle=A_{n,j_{1},1,1}+A_{n,j_{1},1,2}.$ $$

Let $\xi(\boldsymbol{\eta},\tau)=\boldsymbol{\eta}^{\top}\boldsymbol{x}+\tau$. By applying the first-order Taylor expansion, the term $A_{n,j_{1},1,1}$ can be rewritten as

$$ $\displaystyle A_{n,j_{1},1,1}=\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|=1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\sum_{|\boldsymbol{\alpha}|=1}\frac{\exp(\beta_{i_{2}|j_{1}}^{n})}{2^{\alpha_{5}}\boldsymbol{\alpha}!}(\Delta\boldsymbol{\omega}_{i_{2}j_{2}|j_{1}}^{n})^{\boldsymbol{\alpha}_{1}}(\Delta\boldsymbol{a}_{j_{1}}^{n})^{\boldsymbol{\alpha}_{2}}(\Delta\boldsymbol{\eta}_{j_{1}i_{2}j_{2}}^{n})^{\boldsymbol{\alpha}_{3}}(\Delta\tau_{j_{1}i_{2}j_{2}}^{n})^{\alpha_{4}}$ $\displaystyle\times(\Delta\nu_{j_{1}i_{2}j_{2}}^{n})^{\alpha_{5}}\boldsymbol{x}^{\boldsymbol{\alpha}_{1}+\boldsymbol{\alpha}_{2}+\boldsymbol{\alpha}_{3}}\exp((\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})^{\top}\boldsymbol{x})\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})\frac{\partial^{|\boldsymbol{\alpha}_{3}|+\alpha_{4}+2\alpha_{5}}\pi}{\partial\xi^{|\boldsymbol{\alpha}_{3}|+\alpha_{4}+2\alpha_{5}}}(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})$ $\displaystyle\hskip 341.43306pt+R_{n,1,1}(\boldsymbol{x})$ $\displaystyle=\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|=1}\sum_{|\boldsymbol{\rho}_{1}|+\boldsymbol{\rho}_{2}=1}^{2}S_{n,j_{2}|j_{1},\boldsymbol{\rho}_{1},\rho_{2}}\cdot\boldsymbol{x}^{\boldsymbol{\rho}_{1}}\cdot\exp((\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})^{\top}\boldsymbol{x})\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})$ $\displaystyle\hskip 170.71652pt\times\frac{\partial^{\rho_{2}}\pi}{\partial\xi^{\rho_{2}}}(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})+R_{n,1,1}(\boldsymbol{x}),$ $$

where $R_{n,1,1}(\boldsymbol{x})$ is a Taylor remainder satisfying $R_{n,1,1}(\boldsymbol{x})/\mathcal{L}_{1n}\to 0$ as $n\to\infty$, and

$$ $\displaystyle S_{n,j_{2}|j_{1},\boldsymbol{\rho}_{1},\rho_{2}}$ $\displaystyle:=\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\sum_{(\boldsymbol{\alpha}_{1},\boldsymbol{\alpha}_{2},\boldsymbol{\alpha}_{3},\alpha_{4},\alpha_{5})\in\mathcal{I}^{SS}_{\boldsymbol{\rho}_{1},\rho_{2}}}\frac{\exp(\beta_{i_{2}|j_{1}}^{n})}{2^{\alpha_{5}}\boldsymbol{\alpha}!}(\Delta\boldsymbol{\omega}_{i_{2}j_{2}|j_{1}}^{n})^{\boldsymbol{\alpha}_{1}}(\Delta\boldsymbol{a}_{j_{1}}^{n})^{\boldsymbol{\alpha}_{2}}(\Delta\boldsymbol{\eta}_{j_{1}i_{2}j_{2}}^{n})^{\boldsymbol{\alpha}_{3}}$ $\displaystyle\hskip 199.16928pt\times(\Delta\tau_{j_{1}i_{2}j_{2}}^{n})^{\alpha_{4}}(\Delta\nu_{j_{1}i_{2}j_{2}}^{n})^{\alpha_{5}},$ $$

for any $(\boldsymbol{\rho}_{1},\rho_{2})\neq(\boldsymbol{0}_{d},0)$ and $j_{1}\in[k^{*}_{1}],j_{2}\in[k^{*}_{2}]$ in which

$$ $\displaystyle\mathcal{I}^{SS}_{\boldsymbol{\rho}_{1},\rho_{2}}:=\{(\boldsymbol{\alpha}_{1},\boldsymbol{\alpha}_{2},\boldsymbol{\alpha}_{3},\alpha_{4},\alpha_{5})\in\mathbb{R}^{d}\times\mathbb{R}^{d}\times\mathbb{R}^{d}\times\mathbb{R}:\boldsymbol{\alpha}_{1}+\boldsymbol{\alpha}_{2}+\boldsymbol{\alpha}_{3}=\boldsymbol{\rho}_{1},|\boldsymbol{\alpha}_{3}|+\alpha_{4}+2\alpha_{5}=\rho_{2}\}.$ $$

For each $(j_{1},j_{2})\in[k^{*}_{1}]\times[k^{*}_{2}]$, by applying the Taylor expansion of order $r^{SS}(|\mathcal{V}_{j_{2}|j_{1}}|):=r^{SS}_{j_{2}|j_{1}}$, we can represent the term $A_{n,j_{1},1,2}$ as

$$ $\displaystyle A_{n,j_{1},1,2}=\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|>1}\sum_{|\boldsymbol{\rho}_{1}|+\boldsymbol{\rho}_{2}=1}^{2r^{SS}_{j_{2}|j_{1}}}S_{n,j_{2}|j_{1},\boldsymbol{\rho}_{1},\rho_{2}}\cdot\boldsymbol{x}^{\boldsymbol{\rho}_{1}}\cdot\exp((\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})^{\top}\boldsymbol{x})\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})$ $\displaystyle\hskip 199.16928pt\times\frac{\partial^{\rho_{2}}\pi}{\partial\xi^{\rho_{2}}}(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})+R_{n,1,2}(\boldsymbol{x}),$ $$

where $R_{n,1,2}(\boldsymbol{x})$ is a Taylor remainder such that $R_{n,1,2}(\boldsymbol{x})/\mathcal{L}_{1n}\to 0$ as $n\to\infty$.

Subsequently, we rewrite the term $A_{n,j_{1},2}$ as follows:

$$ $\displaystyle\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|=1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{i_{2}|j_{1}}^{n})\Big{[}\exp((\boldsymbol{\omega}_{i_{2}|j_{1}}^{n})^{\top}\boldsymbol{x})-\exp((\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})^{\top}\boldsymbol{x})\Big{]}\exp((\boldsymbol{a}_{j_{1}}^{n})^{\top}\boldsymbol{x})p^{SS,n}_{j_{1}}(y|\boldsymbol{x})$ $\displaystyle+\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|>1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{i_{2}|j_{1}}^{n})\Big{[}\exp((\boldsymbol{\omega}_{i_{2}|j_{1}}^{n})^{\top}\boldsymbol{x})-\exp((\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})^{\top}\boldsymbol{x})\Big{]}\exp((\boldsymbol{a}_{j_{1}}^{n})^{\top}\boldsymbol{x})p^{SS,n}_{j_{1}}(y|\boldsymbol{x})$ $\displaystyle:$ $\displaystyle=A_{n,j_{1},2,1}+A_{n,j_{1},2,2}.$ $$

By means of the first-order Taylor expansion, we have

$$ $\displaystyle A_{n,j_{1},2,1}=\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|=1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\sum_{|\boldsymbol{\psi}|=1}\frac{\exp(\beta_{i_{2}|j_{1}}^{n})}{\boldsymbol{\psi}!}(\Delta\boldsymbol{\omega}_{i_{2}j_{2}|j_{1}}^{n})^{\boldsymbol{\psi}}$ $\displaystyle\hskip 142.26378pt\times\boldsymbol{x}^{\boldsymbol{\psi}}\exp((\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})^{\top}\boldsymbol{x})\exp((\boldsymbol{a}_{j_{1}}^{n})^{\top}\boldsymbol{x})p^{SS,n}_{j_{1}}(y|\boldsymbol{x})+R_{n,2,1}(\boldsymbol{x}),$ $\displaystyle=\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|=1}\sum_{|\boldsymbol{\psi}|=1}T_{n,j_{2}|j_{1},\boldsymbol{\psi}}\cdot\boldsymbol{x}^{\boldsymbol{\psi}}\exp((\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})^{\top}\boldsymbol{x})\exp((\boldsymbol{a}_{j_{1}}^{n})^{\top}\boldsymbol{x})p^{SS,n}_{j_{1}}(y|\boldsymbol{x})+R_{n,2,1}(\boldsymbol{x}),$ $$

where $R_{n,2,1}(\boldsymbol{x})$ is a Taylor remainder such that $R_{n,2,1}(\boldsymbol{x})/\mathcal{L}_{1n}\to 0$ as $n\to\infty$, and

$$ $\displaystyle T_{n,j_{2}|j_{1},\boldsymbol{\psi}}:=\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\frac{\exp(\beta_{i_{2}|j_{1}}^{n})}{\boldsymbol{\psi}!}(\Delta\boldsymbol{\omega}_{i_{2}j_{2}|j_{1}}^{n})^{\boldsymbol{\psi}},$ $$

for any $j_{2}\in[k^{*}_{2}]$ and $\boldsymbol{\psi}\neq\boldsymbol{0}_{d}$.

At the same time, we apply the second-order Taylor expansion to $A_{n,j_{1},2,2}$:

$$ $\displaystyle A_{n,j_{1},2,2}$ $\displaystyle=\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|>1}\sum_{|\boldsymbol{\psi}|=1}^{2}T_{n,j_{2}|j_{1},\boldsymbol{\psi}}\cdot\boldsymbol{x}^{\boldsymbol{\psi}}\exp((\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})^{\top}\boldsymbol{x})\exp((\boldsymbol{a}_{j_{1}}^{n})^{\top}\boldsymbol{x})p^{SS,n}_{j_{1}}(y|\boldsymbol{x})+R_{n,2,2}(\boldsymbol{x}),$ $$

where $R_{n,2,2}(\boldsymbol{x})$ is a Taylor remainder such that $R_{n,2,2}(\boldsymbol{x})/\mathcal{L}_{1n}\to 0$ as $n\to\infty$.

As a result, the term $A_{n}$ can be rewritten as

$$ $\displaystyle A_{n}=\sum_{j_{1}=1}^{k^{*}_{1}}\sum_{j_{2}=1}^{k^{*}_{2}}\frac{\exp(b_{j_{1}}^{n})}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp((\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}})^{\top}\boldsymbol{x}+\beta^{*}_{j^{\prime}_{2}|j_{1}})}\Bigg{[}\sum_{|\boldsymbol{\rho}_{1}|+\boldsymbol{\rho}_{2}=1}^{2r^{SS}_{j_{2}|j_{1}}}S_{n,j_{2}|j_{1},\boldsymbol{\rho}_{1},\rho_{2}}\cdot\boldsymbol{x}^{\boldsymbol{\rho}_{1}}\cdot\exp((\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})^{\top}\boldsymbol{x})$ $\displaystyle\times\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})\frac{\partial^{\rho_{2}}\pi}{\partial\xi^{\rho_{2}}}(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})+R_{n,1,1}(\boldsymbol{x})+R_{n,1,2}(\boldsymbol{x})$ $\displaystyle-\sum_{|\boldsymbol{\psi}|=0}^{2}T_{n,j_{2}|j_{1},\boldsymbol{\psi}}\cdot\boldsymbol{x}^{\boldsymbol{\psi}}\exp((\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})^{\top}\boldsymbol{x})\exp((\boldsymbol{a}_{j_{1}}^{n})^{\top}\boldsymbol{x})p^{SS,n}_{j_{1}}(y|\boldsymbol{x})-R_{n,2,1}(\boldsymbol{x})-R_{n,2,2}(\boldsymbol{x})\Bigg{]},$ (18) $$

where $S_{n,j_{2}|j_{1},\boldsymbol{\rho}_{1},\rho_{2}}=T_{n,j_{2}|j_{1},\boldsymbol{\psi}}=\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{i_{2}|j_{1}}^{n})-\exp(\beta_{j_{2}|j_{1}}^{*})$ for any $j_{2}\in[k^{*}_{2}]$ where $(\boldsymbol{\alpha}_{1},\boldsymbol{\rho}_{1},\rho_{2})=(\boldsymbol{0}_{d},\boldsymbol{0}_{d},0)$ and $\boldsymbol{\psi}=\boldsymbol{0}_{d}$.

Step 1B - Decompose $B_{n}$: By invoking the first-order Taylor expansion, the term $B_{n}$ defined in equation ([17](#A5.E17)) can be rewritten as

$$ $\displaystyle B_{n}$ $\displaystyle=\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\sum_{|\boldsymbol{\gamma}|=1}(\Delta\boldsymbol{a}_{j_{1}}^{n})^{\boldsymbol{\gamma}}\cdot\boldsymbol{x}^{\boldsymbol{\gamma}}\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})p^{SS}_{G_{n}}(y|\boldsymbol{x})+R_{n,3}(\boldsymbol{x}),$ (19) $$

where $R_{n,3}(\boldsymbol{x})$ is a Taylor remainder such that $R_{n,3}(\boldsymbol{x})/\mathcal{L}_{1n}\to 0$ as $n\to\infty$.

From the decomposition in equations ([17](#A5.E17)), ([18](#A5.E18)) and ([19](#A5.E19)), we realize that $A_{n}$, $B_{n}$ and $C_{n}$ can be viewed as a combination of elements from the following set union:

$$ $\displaystyle\Bigg{\{}\boldsymbol{x}^{\boldsymbol{\rho}_{1}}\cdot\exp((\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})^{\top}\boldsymbol{x})\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})\frac{\partial^{\rho_{2}}\pi}{\partial\xi^{\rho_{2}}}(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*}):j_{1}\in[k^{*}_{1}],\ j_{2}\in[k^{*}_{2}],$ $\displaystyle\hskip 256.0748pt0\leq|\boldsymbol{\rho}_{1}|+\rho_{2}\leq 2r^{SS}_{j_{2}|j_{1}}\Bigg{\}}$ $\displaystyle\cup~{}$ $\displaystyle\Bigg{\{}\frac{\boldsymbol{x}^{\boldsymbol{\psi}}\exp((\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})^{\top}\boldsymbol{x})\exp((\boldsymbol{a}_{j_{1}}^{n})^{\top}\boldsymbol{x})p^{SS,n}_{j_{1}}(y|\boldsymbol{x})}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp((\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}})^{\top}\boldsymbol{x}+\beta^{*}_{j^{\prime}_{2}|j_{1}})}:j_{1}\in[k^{*}_{1}],\ j_{2}\in[k^{*}_{2}],\ 0\leq|\boldsymbol{\psi}|\leq 2\Bigg{\}}$ $\displaystyle\cup~{}$ $\displaystyle\left\{\boldsymbol{x}^{\boldsymbol{\gamma}}\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})p^{SS,n}_{j_{1}}(y|\boldsymbol{x}),\ \boldsymbol{x}^{\boldsymbol{\gamma}}\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})p^{SS}_{G_{n}}(y|\boldsymbol{x}):j_{1}\in[k^{*}_{1}],\ 0\leq|\boldsymbol{\gamma}|\leq 1\right\}.$ $$

Step 2 - Non-vanishing coefficients: In this stage, we show that not all the coefficients in the representation of $A_{n}/\mathcal{L}_{1n}$, $B_{n}/\mathcal{L}_{1n}$ and $C_{n}/\mathcal{L}_{1n}$ go to zero as $n\to\infty$. Assume that all of them approach zero, then by looking into the coefficients associated with the term

- •
$\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})p^{SS,n}_{j_{1}}(y|\boldsymbol{x})$ in $C_{n}/\mathcal{L}_{1n}$, we have
$\displaystyle\frac{1}{\mathcal{L}_{1n}}\cdot\sum_{j_{1}=1}^{k^{*}_{1}}\Big{|}\exp(b_{j_{1}}^{n})-\exp(b_{j_{1}}^{*})\Big{|}\to 0.$
(20)
- •
$\dfrac{\exp((\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})^{\top}\boldsymbol{x})\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})\pi(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp((\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}})^{\top}\boldsymbol{x}+\beta^{*}_{j^{\prime}_{2}|j_{1}})}$ in $A_{n}/\mathcal{L}_{1n}$, we get that
$\displaystyle\frac{1}{\mathcal{L}_{1n}}\cdot\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\sum_{j_{2}=1}^{k^{*}_{2}}\Big{|}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{i_{2}|j_{1}}^{n})-\exp(\beta_{j_{2}|j_{1}}^{*})\Big{|}\to 0.$
(21)
- •
$\dfrac{\boldsymbol{x}^{\boldsymbol{\psi}}\exp((\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})^{\top}\boldsymbol{x})\exp((\boldsymbol{a}_{j_{1}}^{n})^{\top}\boldsymbol{x})p^{SS,n}_{j_{1}}(y|\boldsymbol{x})}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp((\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}})^{\top}\boldsymbol{x}+\beta^{*}_{j^{\prime}_{2}|j_{1}})}$ in $A_{n}/\mathcal{L}_{1n}$ for $j_{1}\in[k^{*}_{1}],j_{2}\in[k^{*}_{2}]:|\mathcal{V}_{j_{2}|j_{1}}|=1$ and $\boldsymbol{\psi}=e_{d,u}$ where $e_{d,u}:=(0,\ldots,0,\underbrace{1}_{\textit{u-th}},0,\ldots,0)\in\mathbb{N}^{d}$, we receive
$\displaystyle\frac{1}{\mathcal{L}_{1n}}\cdot\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\sum_{j_{2}\in[k^{*}_{2}]:|\mathcal{V}_{j_{2}|j_{1}}|=1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{i_{2}|j_{1}}^{n})\|\boldsymbol{\omega}_{i_{2}|j_{1}}^{n}-\boldsymbol{\omega}_{j_{2}|j_{1}}^{*}\|_{1}\to 0.$
Note that since the norm-1 is equivalent to the norm-2, then we can replace the norm-1 with the norm-2, that is,
$\displaystyle\frac{1}{\mathcal{L}_{1n}}\cdot\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\sum_{j_{2}\in[k^{*}_{2}]:|\mathcal{V}_{j_{2}|j_{1}}|=1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{i_{2}|j_{1}}^{n})\|\boldsymbol{\omega}_{i_{2}|j_{1}}^{n}-\boldsymbol{\omega}_{j_{2}|j_{1}}^{*}\|\to 0.$
(22)
- •
$\dfrac{\exp((\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})^{\top}\boldsymbol{x})\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})\frac{\partial^{\rho_{2}}\pi}{\partial\xi^{\rho_{2}}}(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp((\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}})^{\top}\boldsymbol{x}+\beta^{*}_{j^{\prime}_{2}|j_{1}})}$ in $A_{n}/\mathcal{L}_{1n}$ for $j_{1}\in[k^{*}_{1}],j_{2}\in[k^{*}_{2}]:|\mathcal{V}_{j_{2}|j_{1}}|=1$ and $\rho_{2}=1$, we have that
$\displaystyle\frac{1}{\mathcal{L}_{1n}}\cdot\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\sum_{j_{2}\in[k^{*}_{2}]:|\mathcal{V}_{j_{2}|j_{1}}|=1}\exp(\beta_{j_{2}|j_{1}}^{n})|\tau_{j_{1}j_{2}}^{n}-\tau_{j_{1}j_{2}}^{*}|\to 0.$
(23)
- •
$\dfrac{\boldsymbol{x}^{\boldsymbol{\rho}_{1}}\exp((\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})^{\top}\boldsymbol{x})\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})\frac{\partial^{\rho_{2}}\pi}{\partial\xi^{\rho_{2}}}(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp((\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}})^{\top}\boldsymbol{x}+\beta^{*}_{j^{\prime}_{2}|j_{1}})}$ in $A_{n}/\mathcal{L}_{1n}$ for $j_{1}\in[k^{*}_{1}],j_{2}\in[k^{*}_{2}]:|\mathcal{V}_{j_{2}|j_{1}}|=1$, $\boldsymbol{\rho}_{1}=e_{d,u}$ and $\rho_{2}=1$, we have that
$\displaystyle\frac{1}{\mathcal{L}_{1n}}\cdot\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\sum_{j_{2}\in[k^{*}_{2}]:|\mathcal{V}_{j_{2}|j_{1}}|=1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{j_{2}|j_{1}}^{n})\|\boldsymbol{\eta}_{j_{1}i_{2}}^{n}-\boldsymbol{\eta}_{j_{1}j_{2}}^{*}\|\to 0.$
(24)
- •
$\dfrac{\exp((\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})^{\top}\boldsymbol{x})\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})\frac{\partial^{\rho_{2}}\pi}{\partial\xi^{\rho_{2}}}(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp((\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}})^{\top}\boldsymbol{x}+\beta^{*}_{j^{\prime}_{2}|j_{1}})}$ in $A_{n}/\mathcal{L}_{1n}$ for $j_{1}\in[k^{*}_{1}],j_{2}\in[k^{*}_{2}]:|\mathcal{V}_{j_{2}|j_{1}}|=1$ and $\rho_{2}=2$, we have that
$\displaystyle\frac{1}{\mathcal{L}_{1n}}\cdot\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\sum_{j_{2}\in[k^{*}_{2}]:|\mathcal{V}_{j_{2}|j_{1}}|=1}\exp(\beta_{j_{2}|j_{1}}^{n})|\nu_{j_{1}j_{2}}^{n}-\nu_{j_{1}j_{2}}^{*}|\to 0.$
(25)
- •
$\boldsymbol{x}^{\boldsymbol{\gamma}}\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})p^{SS}_{G_{n}}(y|\boldsymbol{x})$ in $B_{n}/\mathcal{L}_{1n}$ for $j_{1}\in[k^{*}_{1}]$ and $\boldsymbol{\gamma}=e_{d,u}$, we obtain
$\displaystyle\frac{1}{\mathcal{L}_{1n}}\cdot\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\|\boldsymbol{a}_{j_{1}}^{n}-\boldsymbol{a}_{j_{1}}^{*}\|\to 0.$
(26)
- •
$\dfrac{\boldsymbol{x}^{\boldsymbol{\psi}}\exp((\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})^{\top}\boldsymbol{x})\exp((\boldsymbol{a}_{j_{1}}^{n})^{\top}\boldsymbol{x})p^{SS,n}_{j_{1}}(y|\boldsymbol{x})}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp((\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}})^{\top}\boldsymbol{x}+\beta^{*}_{j^{\prime}_{2}|j_{1}})}$ in $A_{n}/\mathcal{L}_{1n}$ for $j_{1}\in[k^{*}_{1}],j_{2}\in[k^{*}_{2}]:|\mathcal{V}_{j_{2}|j_{1}}|>1$ and $\boldsymbol{\psi}=2e_{d,u}$, we receive that
$\displaystyle\frac{1}{\mathcal{L}_{1n}}\cdot\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\sum_{j_{2}\in[k^{*}_{2}]:|\mathcal{V}_{j_{2}|j_{1}}|>1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{i_{2}|j_{1}}^{n})\|\boldsymbol{\omega}_{i_{2}|j_{1}}^{n}-\boldsymbol{\omega}_{j_{2}|j_{1}}^{*}\|^{2}\to 0.$
(27)

Combine the above limits together with the loss $\mathcal{L}_{1n}$ in equation ([16](#A5.E16)), it yields that

$$ $\displaystyle\frac{1}{\mathcal{L}_{1n}}\cdot\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\Bigg{[}\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|>1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta^{n}_{i_{2}|j_{1}})\Big{(}\|\Delta\boldsymbol{\eta}_{j_{1}i_{2}j_{2}}^{n}\|^{\frac{r^{SS}_{j_{2}|j_{1}}}{2}}+|\Delta\tau_{j_{1}i_{2}j_{2}}^{n}|^{r^{SS}_{j_{2}|j_{1}}}$ $\displaystyle\hskip 256.0748pt+|\Delta\nu_{j_{1}i_{2}j_{2}}^{n}|^{\frac{r^{SS}_{j_{2}|j_{1}}}{2}}\Big{)}\Bigg{]}\not\to 0,$ $$

which indicates that

$$ $\displaystyle\frac{1}{\mathcal{L}_{1n}}\cdot\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\Bigg{[}\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|>1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta^{n}_{i_{2}|j_{1}})\Big{(}\|\Delta\boldsymbol{\omega}_{i_{2}j_{2}|j_{1}}^{n}\|^{r^{SS}_{j_{2}|j_{1}}}+\|\Delta\boldsymbol{a}_{j_{1}}^{n}\|^{r^{SS}_{j_{2}|j_{1}}}$ $\displaystyle\hskip 85.35826pt+\|\Delta\boldsymbol{\eta}_{j_{1}i_{2}j_{2}}^{n}\|^{\frac{r^{SS}_{j_{2}|j_{1}}}{2}}+|\Delta\tau_{j_{1}i_{2}j_{2}}^{n}|^{r^{SS}_{j_{2}|j_{1}}}+|\Delta\nu_{j_{1}i_{2}j_{2}}^{n}|^{\frac{r^{SS}_{j_{2}|j_{1}}}{2}}\Big{)}\Bigg{]}\not\to 0,$ $$

as $n\to\infty$. Therefore, there exist indices $j^{*}_{1}\in[k^{*}_{1}]$ and $j^{*}_{2}\in[k^{*}_{2}]:|\mathcal{V}_{j^{*}_{2}|j^{*}_{1}}|>1$ such that

$$ $\displaystyle\frac{1}{\mathcal{L}_{1n}}\cdot\sum_{i_{2}\in\mathcal{V}_{j^{*}_{2}|j^{*}_{1}}}\exp(\beta^{n}_{i_{2}|j^{*}_{1}})\Big{(}\|\boldsymbol{\omega}^{n}_{i_{2}|j^{*}_{1}}-\boldsymbol{\omega}^{*}_{j^{*}_{2}|j^{*}_{1}}\|^{r^{SS}_{j^{*}_{2}|j^{*}_{1}}}+\|\boldsymbol{a}^{n}_{j^{*}_{1}}-\boldsymbol{a}^{*}_{j^{*}_{1}}\|^{r^{SS}_{j^{*}_{2}|j^{*}_{1}}}+\|\boldsymbol{\eta}^{n}_{j^{*}_{1}i_{2}}-\boldsymbol{\eta}^{*}_{j^{*}_{1}j^{*}_{2}}\|^{\frac{r^{SS}_{j^{*}_{2}|j^{*}_{1}}}{2}}$ $\displaystyle\hskip 113.81102pt+|\tau^{n}_{j^{*}_{1}i_{2}}-\tau^{*}_{j^{*}_{1}j^{*}_{2}}|^{r^{SS}_{j^{*}_{2}|j^{*}_{1}}}+|\nu^{n}_{j^{*}_{1}i_{2}}-\nu^{*}_{j^{*}_{1}j^{*}_{2}}|^{\frac{r^{SS}_{j^{*}_{2}|j^{*}_{1}}}{2}}\Big{)}\not\to 0.$ (28) $$

WLOG, we may assume that $j^{*}_{1}=j^{*}_{2}=1$. By examining the coefficients of the terms

$$ $\dfrac{\boldsymbol{x}^{\boldsymbol{\rho}_{1}}\exp((\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})^{\top}\boldsymbol{x})\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})\frac{\partial^{\rho_{2}}\pi}{\partial\xi^{\rho_{2}}}(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp((\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}})^{\top}\boldsymbol{x}+\beta^{*}_{j^{\prime}_{2}|j_{1}})}$ $$

in $A_{n}/\mathcal{L}_{1n}$ for $j_{1}=j_{2}=1$, we have $\exp(b^{n}_{1})S_{n,1|1,\boldsymbol{0}_{d},\boldsymbol{\rho}_{1},\rho_{2}}/\mathcal{L}_{1n}\to 0$, or equivalently,

$$ $\displaystyle\frac{1}{\mathcal{L}_{1n}}\cdot\sum_{i_{2}\in\mathcal{V}_{1|1}}\sum_{(\boldsymbol{\alpha}_{1},\boldsymbol{\alpha}_{2},\boldsymbol{\alpha}_{3},\alpha_{4},\alpha_{5})\in\mathcal{I}^{SS}_{\boldsymbol{\rho}_{1},\rho_{2}}}\frac{\exp(\beta^{n}_{i_{2}|1})}{2^{\alpha_{5}}\boldsymbol{\alpha}!}\cdot(\Delta\boldsymbol{\omega}^{n}_{1i_{2}1})^{\boldsymbol{\alpha}_{1}}(\Delta\boldsymbol{a}^{n}_{1})^{\boldsymbol{\alpha}_{2}}(\Delta\boldsymbol{\eta}^{n}_{1i_{2}1})^{\boldsymbol{\alpha}_{3}}$ $\displaystyle\hskip 199.16928pt\times(\Delta\tau^{n}_{1i_{2}1})^{\alpha_{4}}(\Delta\nu^{n}_{1i_{2}1})^{\alpha_{5}}\to 0.$ (29) $$

By dividing the left hand side of equation ([29](#A5.E29)) by that of equation ([28](#A5.E28)), we get

$$ $\displaystyle\dfrac{\sum_{i_{2}\in\mathcal{V}_{1|1}}\sum_{(\boldsymbol{\alpha}_{1},\boldsymbol{\alpha}_{2},\boldsymbol{\alpha}_{3},\alpha_{4},\alpha_{5})\in\mathcal{I}^{SS}_{\boldsymbol{\rho}_{1},\rho_{2}}}\frac{\exp(\beta^{n}_{i_{2}|1})}{2^{\alpha_{5}}\boldsymbol{\alpha}!}\cdot(\Delta\boldsymbol{\omega}^{n}_{1i_{2}1})^{\boldsymbol{\alpha}_{1}}(\Delta\boldsymbol{a}^{n}_{1})^{\boldsymbol{\alpha}_{2}}(\Delta\boldsymbol{\eta}^{n}_{1i_{2}1})^{\boldsymbol{\alpha}_{3}}(\Delta\tau^{n}_{1i_{2}1})^{\alpha_{4}}(\Delta\nu^{n}_{1i_{2}1})^{\alpha_{5}}}{\sum_{i_{2}\in\mathcal{V}_{1|1}}\exp(\beta^{n}_{i_{2}|1})\Big{(}\|\Delta\boldsymbol{\omega}^{n}_{1i_{2}1}\|^{r^{SS}_{1|1}}+\|\Delta\boldsymbol{a}^{n}_{1}\|^{r^{SS}_{1|1}}+\|\Delta\boldsymbol{\eta}^{n}_{1i_{2}i}\|^{\frac{r^{SS}_{1|1}}{2}}+|\Delta\tau^{n}_{1i_{2}1}|^{r^{SS}_{1|1}}+|\Delta\nu^{n}_{1i_{2}1}|^{\frac{r^{SS}_{1|1}}{2}}\Big{)}}\to 0.$ (30) $$

Let us define $\overline{M}_{n}:=\max\{\|\Delta\boldsymbol{\omega}^{n}_{1i_{2}1}\|,\|\Delta\boldsymbol{a}^{n}_{1}\|,\|\Delta\boldsymbol{\eta}^{n}_{1i_{2}1}\|^{1/2},\|\Delta\tau^{n}_{1i_{2}1}\|,\|\Delta\nu^{n}_{1i_{2}1}\|^{1/2}:i_{2}\in\mathcal{V}_{1|1}\}$, and $\overline{\beta}_{n}:=\max_{i_{2}\in\mathcal{V}_{1|1}}\exp(\beta^{n}_{i_{2}|1})$. Since the sequence $\exp(\beta^{n}_{i_{2}|1})/\overline{\beta}_{n}$ is bounded, we can replace it by its subsequence which has a positive limit $p^{2}_{i_{2}}:=\lim_{n\to\infty}\exp(\beta^{n}_{i_{2}|1})/\overline{\beta}_{n}$. Note that at least one among the limits $p^{2}_{i_{2}}$ must be equal to one. Next, let us define

$$ $\displaystyle(\Delta\boldsymbol{\omega}^{n}_{1i_{2}1})/\overline{M}\to\boldsymbol{q}_{1i_{2}}$ $\displaystyle\quad(\Delta\boldsymbol{a}^{n}_{1})/\overline{M}_{n}\to\boldsymbol{q}_{2},$ $\displaystyle(\Delta\boldsymbol{\eta}^{n}_{1i_{2}1})/\overline{M}_{n}\to\boldsymbol{q}_{3i_{2}},$ $\displaystyle(\Delta\tau^{n}_{1i_{2}1})/\overline{M}_{n}\to q_{4i_{2}},$ $\displaystyle\quad(\Delta\nu^{n}_{1i_{2}1})/2\overline{M}_{n}\to q_{5i_{2}}$ $\displaystyle\quad.$ $$

Note that at least one among $\boldsymbol{q}_{1i_{2}},\boldsymbol{q}_{2},\boldsymbol{q}_{3i_{2}},q_{4i_{2}},q_{5i_{2}}$ must be equal to either 1 or $-1$.

By dividing both the numerator and the denominator of the term in equation ([30](#A5.E30)) by $\overline{\beta}_{n}\overline{M}^{|\boldsymbol{\rho}_{1}|+\rho_{2}}_{n}$, we obtain the system of polynomial equations:

$$ $\displaystyle\sum_{i_{2}\in\mathcal{V}_{1|1}}\sum_{(\boldsymbol{\alpha}_{1},\boldsymbol{\alpha}_{2},\boldsymbol{\alpha}_{3},\alpha_{4},\alpha_{5})\in\mathcal{I}^{SS}_{\boldsymbol{\rho}_{1},\rho_{2}}}\frac{1}{\boldsymbol{\alpha}!}\cdot p^{2}_{i_{2}}\boldsymbol{q}_{1i_{2}}^{\boldsymbol{\alpha}_{1}}\boldsymbol{q}_{2}^{\boldsymbol{\alpha}_{2}}\boldsymbol{q}_{3i_{2}}^{\boldsymbol{\alpha}_{3}}q_{4i_{2}}^{\alpha_{4}}q_{5i_{2}}^{\alpha_{5}}=0,\quad 1\leq|\boldsymbol{\rho}_{1}|+\rho_{2}\leq r^{SS}_{1|1}.$ $$

According to the definition of the term $r^{SS}_{1|1}$, the above system does not have any non-trivial solutions, which is a contradiction. Consequently, at least one among the coefficients in the representation of $A_{n}/\mathcal{L}_{1n}$, $B_{n}/\mathcal{L}_{1n}$ and $C_{n}/\mathcal{L}_{1n}$ must not converge to zero as $n\to\infty$.

Step 3 - Application of the Fatou’s lemma. In this stage, we show that all the coefficients in the formulations of $A_{n}/\mathcal{L}_{1n}$, $B_{n}/\mathcal{L}_{1n}$ and $C_{n}/\mathcal{L}_{1n}$ go to zero as $n\to\infty$. Denote by $m_{n}$ the maximum of the absolute values of those coefficients, the result from Step 2 induces that $1/m_{n}\not\to\infty$. By employing the Fatou’s lemma, we have

$$ $\displaystyle 0=\lim_{n\to\infty}\dfrac{\mathbb{E}_{\boldsymbol{X}}[V(p^{SS}_{G_{n}}(\cdot|\boldsymbol{X}),p^{SS}_{G_{*}}(\cdot|\boldsymbol{X}))]}{m_{n}\mathcal{L}_{1n}}\geq\int\liminf_{n\to\infty}\dfrac{|p^{SS}_{G_{n}}(y|\boldsymbol{x})-p^{SS}_{G_{*}}(y|\boldsymbol{x})|}{2m_{n}\mathcal{L}_{1n}}\mathrm{d}(\boldsymbol{x},y).$ $$

Thus, we deduce that

$$ $\displaystyle\dfrac{|p^{SS}_{G_{n}}(y|\boldsymbol{x})-p^{SS}_{G_{*}}(y|\boldsymbol{x})|}{2m_{n}\mathcal{L}_{1n}}\to 0,$ $$

which results in $Q_{n}/[m_{n}\mathcal{L}_{1n}]\to 0$ as $n\to\infty$ for almost surely $(\boldsymbol{x},y)$.

Next, we denote

$$ $\displaystyle\frac{\exp(b_{j_{1}}^{n})S_{n,j_{2}|j_{1},\boldsymbol{\rho}_{1},\rho_{2}}}{m_{n}\mathcal{L}_{1n}}\to\phi_{j_{2}|j_{1},\boldsymbol{\rho}_{1},\rho_{2}},\quad$ $\displaystyle\frac{\exp(b_{j_{1}}^{n})T_{n,j_{2}|j_{1},\boldsymbol{\psi}}}{m_{n}\mathcal{L}_{1n}}\to\varphi_{j_{2}|j_{1},\boldsymbol{\psi}},$ $\displaystyle\frac{\exp(b_{j_{1}}^{n})(\Delta\boldsymbol{a}_{j_{1}}^{n})^{\boldsymbol{\gamma}}}{m_{n}\mathcal{L}_{1n}}\to\lambda_{j_{1},\boldsymbol{\gamma}},\quad$ $\displaystyle\frac{\exp(b_{j_{1}}^{n})-\exp(b_{j_{1}}^{*})}{m_{n}\mathcal{L}_{1n}}\to\chi_{j_{1}}$ $$

with a note that at least one among them is non-zero. Then, the decomposition of $Q_{n}$ in equation ([17](#A5.E17)) indicates that

$$ $\displaystyle\lim_{n\to\infty}\frac{Q_{n}}{m_{n}\mathcal{L}_{1n}}=\lim_{n\to\infty}\frac{A_{n}}{m_{n}\mathcal{L}_{1n}}-\lim_{n\to\infty}\frac{B_{n}}{m_{n}\mathcal{L}_{1n}}+\lim_{n\to\infty}\frac{C_{n}}{m_{n}\mathcal{L}_{1n}},$ $$

in which

$$ $\displaystyle\lim_{n\to\infty}\frac{A_{n}}{m_{n}\mathcal{L}_{1n}}=\sum_{j_{1}=1}^{k^{*}_{1}}\sum_{j_{2}=1}^{k^{*}_{2}}\Bigg{[}\sum_{|\boldsymbol{\rho}_{1}|+\boldsymbol{\rho}_{2}=0}^{2r^{SS}_{j_{2}|j_{1}}}S_{n,j_{2}|j_{1},\boldsymbol{\rho}_{1},\rho_{2}}\cdot\boldsymbol{x}^{\boldsymbol{\rho}_{1}}\exp((\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})^{\top}\boldsymbol{x})$ $\displaystyle\times\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})\frac{\partial^{\rho_{2}}\pi}{\partial\xi^{\rho_{2}}}(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})-\sum_{|\boldsymbol{\psi}|=0}^{2}\varphi_{j_{2}|j_{1},\boldsymbol{\psi}}\cdot\boldsymbol{x}^{\boldsymbol{\psi}}\exp((\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})^{\top}\boldsymbol{x})$ $\displaystyle\hskip 85.35826pt\times\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})p^{SS,*}_{j_{1}}(y|\boldsymbol{x})\Bigg{]}\frac{1}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp((\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}})^{\top}\boldsymbol{x}+\beta^{*}_{j^{\prime}_{2}|j_{1}})},$ $\displaystyle\lim_{n\to\infty}\frac{B_{n}}{m_{n}\mathcal{L}_{1n}}=\sum_{j_{1}=1}^{k^{*}_{1}}\sum_{|\gamma|=1}\lambda_{j_{1},\boldsymbol{\gamma}}\cdot\boldsymbol{x}^{\boldsymbol{\gamma}}\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})p^{SS}_{G_{*}}(y|\boldsymbol{x}),$ $\displaystyle\lim_{n\to\infty}\frac{C_{n}(\boldsymbol{x})}{m_{n}\mathcal{L}_{1n}}=\sum_{j_{1}=1}^{k^{*}_{1}}\chi_{j_{1}}\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})\left[p^{SS,*}_{j_{1}}(y|\boldsymbol{x})-p^{SS}_{G_{*}}(y|\boldsymbol{x})\right].$ $$

Since the set

$$ $\displaystyle\Bigg{\{}\dfrac{\boldsymbol{x}^{\boldsymbol{\rho}_{1}}\exp((\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})^{\top}\boldsymbol{x})\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})\frac{\partial^{\rho_{2}}\pi}{\partial\xi^{\rho_{2}}}(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp((\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}})^{\top}\boldsymbol{x}+\beta^{*}_{j^{\prime}_{2}|j_{1}})}:j_{1}\in[k^{*}_{1}],j_{2}\in[k^{*}_{2}],$ $\displaystyle\hskip 227.62204pt0\leq|\boldsymbol{\rho}_{1}|+\rho_{2}\leq 2r^{SS}_{j_{2}|j_{1}}\Bigg{\}}$ $\displaystyle\cup$ $\displaystyle~{}\Bigg{\{}\dfrac{\boldsymbol{x}^{\boldsymbol{\psi}}\exp((\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})^{\top}\boldsymbol{x})\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})p^{SS,*}_{j_{1}}(y|\boldsymbol{x})}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp((\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}})^{\top}\boldsymbol{x}+\beta^{*}_{j^{\prime}_{2}|j_{1}})}:j_{1}\in[k^{*}_{1}],j_{2}\in[k^{*}_{2}],0\leq|\boldsymbol{\psi}|\leq 2\Bigg{\}}$ $\displaystyle\cup$ $\displaystyle~{}\Big{\{}\boldsymbol{x}^{\boldsymbol{\gamma}}\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})p^{SS}_{G_{*}}(y|\boldsymbol{x}),\ \exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})p^{SS,*}_{j_{1}}(y|\boldsymbol{x}),\ \exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})p^{SS}_{G_{*}}(y|\boldsymbol{x})$ $\displaystyle\hskip 227.62204pt:j_{1}\in[k^{*}_{1}],0\leq|\boldsymbol{\gamma}|\leq 2\Big{\}}$ $$

is linearly independent, we obtain that $\phi_{j_{2}|j_{1},\boldsymbol{\rho}_{1},\rho_{2}}=\varphi_{j_{2}|j_{1},\boldsymbol{\psi}}=\lambda_{j_{1},\boldsymbol{\gamma}}=\chi_{j_{1}}=0$ for all $j_{1}\in[k^{*}_{1}]$, $j_{2}\in[k^{*}_{2}]$, $0\leq|\boldsymbol{\rho}_{1}|+\rho_{2}\leq 2r^{SS}_{j_{2}|j_{1}}$, $0\leq|\boldsymbol{\psi}|\leq 2$ and $0\leq|\boldsymbol{\gamma}|\leq 1$, which is a contradiction. As a consequence, we obtain the inequality in equation ([12](#A5.E12)). Hence, the proof is completed.

### E.2 When t ​ y ​ p ​ e = S ​ L 𝑡 𝑦 𝑝 𝑒 𝑆 𝐿 type=SL

When $type=SL$, the corresponding Voronoi loss function is $\mathcal{L}_{(\frac{1}{2}r^{SL},r^{SL},\frac{1}{2}r^{SL})}(G_{n},G_{*})=\mathcal{L}_{2n}$ where we define

$$ $\displaystyle\mathcal{L}_{2n}:=\sum_{j_{1}=1}^{k^{*}_{1}}\Big{|}\exp(b^{n}_{j_{1}})-\exp(b^{*}_{j_{1}})\Big{|}+\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b^{n}_{j_{1}})\|\Delta\boldsymbol{a}_{j_{1}}^{n}\|+\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b^{n}_{j_{1}})$ $\displaystyle\times\Bigg{[}\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|=1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta^{n}_{i_{2}|j_{1}})\Big{(}\|\Delta\boldsymbol{\omega}_{i_{2}j_{2}|j_{1}}^{n}\|+\|\Delta\boldsymbol{\eta}_{j_{1}i_{2}j_{2}}^{n}\|+|\Delta\tau_{j_{1}i_{2}j_{2}}^{n}|+|\Delta\nu_{j_{1}i_{2}j_{2}}^{n}|\Big{)}$ $\displaystyle+\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|>1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta^{n}_{i_{2}|j_{1}})\Big{(}\|\Delta\boldsymbol{\omega}_{i_{2}j_{2}|j_{1}}^{n}\|^{2}+\|\Delta\boldsymbol{\eta}_{j_{1}i_{2}j_{2}}^{n}\|^{\frac{r^{SL}_{j_{2}|j_{1}}}{2}}+|\Delta\tau_{j_{1}i_{2}j_{2}}^{n}|^{r^{SL}_{j_{2}|j_{1}}}$ $\displaystyle+|\Delta\nu_{j_{1}i_{2}j_{2}}^{n}|^{\frac{r^{SL}_{j_{2}|j_{1}}}{2}}\Big{)}\Bigg{]}+\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b^{n}_{j_{1}})\sum_{j_{2}=1}^{k^{*}_{2}}\Big{|}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta^{n}_{i_{2}|j_{1}})-\exp(\beta^{*}_{j_{2}|j_{1}})\Big{|}.$ (31) $$

Step 1 - Taylor expansion: In this step, we use the Taylor expansion to decompose the term

$$ $\displaystyle Q_{n}:=\left[\sum_{j_{1}=1}^{k^{*}_{1}}\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x}+b_{j_{1}}^{*})\right][p^{SL}_{G_{n}}(y|\boldsymbol{x})-p^{SL}_{G_{*}}(y|\boldsymbol{x})].$ $$

Prior to that, let us denote

$$ $\displaystyle p^{SL,n}_{j_{1}}(y|\boldsymbol{x})$ $\displaystyle:=\sum_{j_{2}=1}^{k^{*}_{2}}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\sigma(-\|\boldsymbol{\omega}_{i_{2}|j_{1}}^{n}-\boldsymbol{x}\|+\beta^{n}_{i_{2}|j_{1}})\pi(y|(\boldsymbol{\eta}_{j_{1}i_{2}}^{n})^{\top}\boldsymbol{x}+\tau_{j_{1}i_{2}}^{n},\nu_{j_{1}i_{2}}^{n}),$ $\displaystyle p^{SL,*}_{j_{1}}(y|\boldsymbol{x})$ $\displaystyle:=\sum_{j_{2}=1}^{k^{*}_{2}}\sigma(-\|\boldsymbol{\omega}_{j_{2}|j_{1}}^{*}-\boldsymbol{x}\|+\beta_{j_{2}|j_{1}}^{*})\pi(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*}).$ $$

Then, the quantity $Q_{n}$ is divided into three terms as

$$ $\displaystyle Q_{n}$ $\displaystyle=\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\left[\exp((\boldsymbol{a}_{j_{1}}^{n})^{\top}\boldsymbol{x})p^{SL,n}_{j_{1}}(y|\boldsymbol{x})-\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})p^{SL,*}_{j_{1}}(y|\boldsymbol{x})\right]$ $\displaystyle-\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\left[\exp((\boldsymbol{a}_{j_{1}}^{n})^{\top}\boldsymbol{x})-\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})\right]p^{SL}_{G_{n}}(y|\boldsymbol{x})$ $\displaystyle+\sum_{j_{1}=1}^{k^{*}_{1}}\left(\exp(b_{j_{1}}^{n})-\exp(b_{j_{1}}^{*})\right)\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})\left[p^{SL,n}_{j_{1}}(y|\boldsymbol{x})-p^{SL}_{G_{n}}(y|\boldsymbol{x})\right]$ $\displaystyle:$ $\displaystyle=A_{n}-B_{n}+C_{n}.$ (32) $$

Step 1A - Decompose $A_{n}$: We continue to decompose $A_{n}$:

$$ $\displaystyle A_{n}$ $\displaystyle:=\sum_{j_{1}=1}^{k^{*}_{1}}\frac{\exp(b_{j_{1}}^{n})}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp(-\|\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}}-\boldsymbol{x}\|+\beta^{*}_{j^{\prime}_{2}|j_{1}})}[A_{n,j_{1},1}+A_{n,j_{1},2}+A_{n,j_{1},3}],$ $$

in which

$$ $\displaystyle A_{n,j_{1},1}$ $\displaystyle:=\sum_{j_{2}=1}^{k^{*}_{2}}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta^{n}_{i_{2}|j_{1}})\Big{[}\exp(-\|\boldsymbol{\omega}_{i_{2}|j_{1}}^{n}-\boldsymbol{x}\|)\exp((\boldsymbol{a}_{j_{1}}^{n})^{\top}\boldsymbol{x})\pi(y|(\boldsymbol{\eta}_{j_{1}i_{2}}^{n}){\top}\boldsymbol{x}+\tau_{j_{1}i_{2}}^{n},\nu_{j_{1}i_{2}}^{n})$ $\displaystyle\hskip 85.35826pt-\exp(-\|\boldsymbol{\omega}_{j_{2}|j_{1}}^{*}-\boldsymbol{x}\|)\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})\pi(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})\Big{]},$ $\displaystyle A_{n,j_{1},2}$ $\displaystyle:=\sum_{j_{2}=1}^{k^{*}_{2}}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta^{n}_{i_{2}|j_{1}})\Big{[}\exp(-\|\boldsymbol{\omega}_{i_{2}|j_{1}}^{n}-\boldsymbol{x}\|)-\exp(-\|\boldsymbol{\omega}_{j_{2}|j_{1}}^{*}-\boldsymbol{x}\|)\Big{]}$ $\displaystyle\hskip 199.16928pt\times\exp((\boldsymbol{a}_{j_{1}}^{n})^{\top}\boldsymbol{x})p^{SL,n}_{j_{1}}(y|\boldsymbol{x}),$ $\displaystyle A_{n,j_{1},3}$ $\displaystyle:=\sum_{j_{2}=1}^{k^{*}_{2}}\Big{(}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{i_{2}|j_{1}}^{n})-\exp(\beta_{j_{2}|j_{1}}^{*})\Big{)}\exp(-\|\boldsymbol{\omega}_{j_{2}|j_{1}}^{*}-\boldsymbol{x}\|)$ $\displaystyle\hskip 28.45274pt\times[\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})\pi(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})-\exp((\boldsymbol{a}_{j_{1}}^{n})^{\top}\boldsymbol{x})p^{SL,n}_{j_{1}}(y|\boldsymbol{x})].$ $$

Based on the cardinality of the Voronoi cells $\mathcal{V}_{j_{2}|j_{1}}$, we proceed to divide the term $A_{n,j_{1},1}$ into two parts as

$$ $\displaystyle A_{n,j_{1},1}$ $\displaystyle=\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|=1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{i_{2}|j_{1}}^{n})\Big{[}\exp(-\|\boldsymbol{\omega}_{i_{2}|j_{1}}^{n}-\boldsymbol{x}\|)\exp((\boldsymbol{a}_{j_{1}}^{n})^{\top}\boldsymbol{x})\pi(y|(\boldsymbol{\eta}_{j_{1}i_{2}}^{n}){\top}\boldsymbol{x}+\tau_{j_{1}i_{2}}^{n},\nu_{j_{1}i_{2}}^{n})$ $\displaystyle\hskip 85.35826pt-\exp(-\|\boldsymbol{\omega}_{j_{2}|j_{1}}^{*}-\boldsymbol{x}\|)\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})\pi(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})\Big{]},$ $\displaystyle+\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|>1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{i_{2}|j_{1}}^{n})\Big{[}\exp(-\|\boldsymbol{\omega}_{i_{2}|j_{1}}^{n}-\boldsymbol{x}\|)\exp((\boldsymbol{a}_{j_{1}}^{n})^{\top}\boldsymbol{x})\pi(y|(\boldsymbol{\eta}_{j_{1}i_{2}}^{n}){\top}\boldsymbol{x}+\tau_{j_{1}i_{2}}^{n},\nu_{j_{1}i_{2}}^{n})$ $\displaystyle\hskip 85.35826pt-\exp(-\|\boldsymbol{\omega}_{j_{2}|j_{1}}^{*}-\boldsymbol{x}\|)\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})\pi(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})\Big{]}$ $\displaystyle:$ $\displaystyle=A_{n,j_{1},1,1}+A_{n,j_{1},1,2}.$ $$

Let us denote $F(\boldsymbol{x};\boldsymbol{\omega}):=\exp(-\|\boldsymbol{\omega}-\boldsymbol{x}\|)$ and $\xi(\boldsymbol{\eta},\tau)=\boldsymbol{\eta}^{\top}\boldsymbol{x}+\tau$. By means of the first-order Taylor expansion, $A_{n,j_{1},1,1}$ can be represented as

$$ $\displaystyle A_{n,j_{1},1,1}=\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|=1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\sum_{|\boldsymbol{\alpha}|=1}\frac{\exp(\beta_{i_{2}|j_{1}}^{n})}{2^{\alpha_{5}}\boldsymbol{\alpha}!}(\Delta\boldsymbol{\omega}_{i_{2}j_{2}|j_{1}}^{n})^{\boldsymbol{\alpha}_{1}}(\Delta\boldsymbol{a}_{j_{1}}^{n})^{\boldsymbol{\alpha}_{2}}(\Delta\boldsymbol{\eta}_{j_{1}i_{2}j_{2}}^{n})^{\boldsymbol{\alpha}_{3}}(\Delta\tau_{j_{1}i_{2}j_{2}}^{n})^{\alpha_{4}}$ $\displaystyle\times(\Delta\nu_{j_{1}i_{2}j_{2}}^{n})^{\alpha_{5}}\boldsymbol{x}^{\boldsymbol{\alpha}_{2}+\boldsymbol{\alpha}_{3}}\frac{\partial^{|\boldsymbol{\alpha}_{1}|}F}{\partial\boldsymbol{\omega}^{\boldsymbol{\alpha}_{1}}}(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})\frac{\partial^{|\boldsymbol{\alpha}_{3}|+\alpha_{4}+2\alpha_{5}}\pi}{\partial\xi^{|\boldsymbol{\alpha}_{3}|+\alpha_{4}+2\alpha_{5}}}(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})+R_{n,1,1}(\boldsymbol{x})$ $\displaystyle=\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|=1}\sum_{|\boldsymbol{\alpha}_{1}|=0}^{1}\sum_{|\boldsymbol{\rho}_{1}|+\boldsymbol{\rho}_{2}=0\vee 1-|\boldsymbol{\alpha}_{1}|}^{2(1-|\boldsymbol{\alpha}_{1}|)}S_{n,j_{2}|j_{1},\boldsymbol{\alpha}_{1},\boldsymbol{\rho}_{1},\rho_{2}}\cdot\boldsymbol{x}^{\boldsymbol{\rho}_{1}}\cdot\frac{\partial^{|\boldsymbol{\alpha}_{1}|}F}{\partial\boldsymbol{\omega}^{\boldsymbol{\alpha}_{1}}}(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})$ $\displaystyle\hskip 199.16928pt\times\frac{\partial^{\rho_{2}}\pi}{\partial\xi^{\rho_{2}}}(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})+R_{n,1,1}(\boldsymbol{x}),$ $$

where $R_{n,1,1}(\boldsymbol{x})$ is a Taylor remainder such that $R_{n,1,1}(\boldsymbol{x})/\mathcal{L}_{2n}\to 0$ as $n\to\infty$, and

$$ $\displaystyle S_{n,j_{2}|j_{1},\boldsymbol{\alpha}_{1},\boldsymbol{\rho}_{1},\rho_{2}}$ $\displaystyle:=\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\sum_{(\boldsymbol{\alpha}_{2},\boldsymbol{\alpha}_{3},\alpha_{4},\alpha_{5})\in\mathcal{I}^{SL}_{\boldsymbol{\rho}_{1},\rho_{2}}}\frac{\exp(\beta_{i_{2}|j_{1}}^{n})}{2^{\alpha_{5}}\boldsymbol{\alpha}!}(\Delta\boldsymbol{\omega}_{i_{2}j_{2}|j_{1}}^{n})^{\boldsymbol{\alpha}_{1}}(\Delta\boldsymbol{a}_{j_{1}}^{n})^{\boldsymbol{\alpha}_{2}}(\Delta\boldsymbol{\eta}_{j_{1}i_{2}j_{2}}^{n})^{\boldsymbol{\alpha}_{3}}$ $\displaystyle\hskip 199.16928pt\times(\Delta\tau_{j_{1}i_{2}j_{2}}^{n})^{\alpha_{4}}(\Delta\nu_{j_{1}i_{2}j_{2}}^{n})^{\alpha_{5}},$ $$

for any $(\boldsymbol{\alpha}_{1},\boldsymbol{\rho}_{1},\rho_{2})\neq(\boldsymbol{0}_{d},\boldsymbol{0}_{d},0)$ and $j_{1}\in[k^{*}_{1}],j_{2}\in[k^{*}_{2}]$ in which

$$ $\displaystyle\mathcal{I}^{SL}_{\boldsymbol{\rho}_{1},\rho_{2}}:=\{(\boldsymbol{\alpha}_{2},\boldsymbol{\alpha}_{3},\alpha_{4},\alpha_{5})\in\mathbb{R}^{d}\times\mathbb{R}^{d}\times\mathbb{R}^{d}\times\mathbb{R}:\boldsymbol{\alpha}_{2}+\boldsymbol{\alpha}_{3}=\boldsymbol{\rho}_{1},|\boldsymbol{\alpha}_{3}|+\alpha_{4}+2\alpha_{5}=\rho_{2}\}.$ $$

For each $(j_{1},j_{2})\in[k^{*}_{1}]\times[k^{*}_{2}]$, by applying the Taylor expansion of order $r^{SL}(|\mathcal{V}_{j_{2}|j_{1}}|):=r^{SL}_{j_{2}|j_{1}}$, the term $A_{n,j_{1},1,2}$ can be rewritten as

$$ $\displaystyle A_{n,j_{1},1,2}=\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|>1}\sum_{|\boldsymbol{\alpha}_{1}|=1}^{r^{SL}_{j_{2}|j_{1}}}\sum_{|\boldsymbol{\rho}_{1}|+\boldsymbol{\rho}_{2}=0\vee 1-|\boldsymbol{\alpha}_{1}|}^{2(r^{SL}_{j_{2}|j_{1}}-|\boldsymbol{\alpha}_{1}|)}S_{n,j_{2}|j_{1},\boldsymbol{\alpha}_{1},\boldsymbol{\rho}_{1},\rho_{2}}\cdot\boldsymbol{x}^{\boldsymbol{\rho}_{1}}\cdot\frac{\partial^{|\boldsymbol{\alpha}_{1}|}F}{\partial\boldsymbol{\omega}^{\boldsymbol{\alpha}_{1}}}(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})$ $\displaystyle\hskip 199.16928pt\times\frac{\partial^{\rho_{2}}\pi}{\partial\xi^{\rho_{2}}}(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})+R_{n,1,2}(\boldsymbol{x}),$ $$

where $R_{n,1,2}(\boldsymbol{x})$ is a Taylor remainder such that $R_{n,1,2}(\boldsymbol{x})/\mathcal{L}_{2n}\to 0$ as $n\to\infty$.

Next, we rewrite the term $A_{n,j_{1},2}$ as follows:

$$ $\displaystyle\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|=1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{i_{2}|j_{1}}^{n})\Big{[}\exp(-\|\boldsymbol{\omega}_{i_{2}|j_{1}}^{n}-\boldsymbol{x}\|)-\exp(-\|\boldsymbol{\omega}_{j_{2}|j_{1}}^{*}-\boldsymbol{x}\|)\Big{]}\exp((\boldsymbol{a}_{j_{1}}^{n})^{\top}\boldsymbol{x})p^{SL,n}_{j_{1}}(y|\boldsymbol{x})$ $\displaystyle+\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|>1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{i_{2}|j_{1}}^{n})\Big{[}\exp(-\|\boldsymbol{\omega}_{i_{2}|j_{1}}^{n}-\boldsymbol{x}\|)-\exp(-\|\boldsymbol{\omega}_{j_{2}|j_{1}}^{*}-\boldsymbol{x}\|)\Big{]}\exp((\boldsymbol{a}_{j_{1}}^{n})^{\top}\boldsymbol{x})p^{SL,n}_{j_{1}}(y|\boldsymbol{x})$ $\displaystyle:$ $\displaystyle=A_{n,j_{1},2,1}+A_{n,j_{1},2,2}.$ $$

By applying the first-order Taylor expansion, we have

$$ $\displaystyle A_{n,j_{1},2,1}=\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|=1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\sum_{|\boldsymbol{\psi}|=1}\frac{\exp(\beta_{i_{2}|j_{1}}^{n})}{\boldsymbol{\psi}!}(\Delta\boldsymbol{\omega}_{i_{2}j_{2}|j_{1}}^{n})^{\boldsymbol{\psi}}$ $\displaystyle\hskip 142.26378pt\times\frac{\partial^{|\boldsymbol{\psi}|}F}{\partial\boldsymbol{\omega}^{\boldsymbol{\psi}}}(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})\exp((\boldsymbol{a}_{j_{1}}^{n})^{\top}\boldsymbol{x})p^{SL,n}_{j_{1}}(y|\boldsymbol{x})+R_{n,2,1}(\boldsymbol{x}),$ $\displaystyle=\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|=1}\sum_{|\boldsymbol{\psi}|=1}T_{n,j_{2}|j_{1},\boldsymbol{\psi}}\cdot\frac{\partial^{|\boldsymbol{\psi}|}F}{\partial\boldsymbol{\omega}^{\boldsymbol{\psi}}}(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})\exp((\boldsymbol{a}_{j_{1}}^{n})^{\top}\boldsymbol{x})p^{SL,n}_{j_{1}}(y|\boldsymbol{x})+R_{n,2,1}(\boldsymbol{x}),$ $$

where $R_{n,2,1}(\boldsymbol{x})$ is a Taylor remainder such that $R_{n,2,1}(\boldsymbol{x})/\mathcal{L}_{2n}\to 0$ as $n\to\infty$, and

$$ $\displaystyle T_{n,j_{2}|j_{1},\boldsymbol{\psi}}:=\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\frac{\exp(\beta_{i_{2}|j_{1}}^{n})}{\boldsymbol{\psi}!}(\Delta\boldsymbol{\omega}_{i_{2}j_{2}|j_{1}}^{n})^{\boldsymbol{\psi}},$ $$

for any $j_{2}\in[k^{*}_{2}]$ and $\boldsymbol{\psi}\neq\boldsymbol{0}_{d}$.

Meanwhile, we employ the second-order Taylor expansion to $A_{n,j_{1},2,2}$:

$$ $\displaystyle A_{n,j_{1},2,2}$ $\displaystyle=\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|>1}\sum_{|\boldsymbol{\psi}|=1}^{2}T_{n,j_{2}|j_{1},\boldsymbol{\psi}}\cdot\frac{\partial^{|\boldsymbol{\psi}|}F}{\partial\boldsymbol{\omega}^{\boldsymbol{\psi}}}(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})\exp((\boldsymbol{a}_{j_{1}}^{n})^{\top}\boldsymbol{x})p^{SL,n}_{j_{1}}(y|\boldsymbol{x})+R_{n,2,2}(\boldsymbol{x}),$ $$

where $R_{n,2,2}(\boldsymbol{x})$ is a Taylor remainder such that $R_{n,2,2}(\boldsymbol{x})/\mathcal{L}_{2n}\to 0$ as $n\to\infty$.

As a result, the term $A_{n}$ can be rewritten as

$$ $\displaystyle A_{n}=\sum_{j_{1}=1}^{k^{*}_{1}}\sum_{j_{2}=1}^{k^{*}_{2}}\frac{\exp(b_{j_{1}}^{n})}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp(-\|\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}}-\boldsymbol{x}\|+\beta^{*}_{j^{\prime}_{2}|j_{1}})}\Bigg{[}\sum_{|\boldsymbol{\alpha}_{1}|=0}^{r^{SL}_{j_{2}|j_{1}}}\sum_{|\boldsymbol{\rho}_{1}|+\boldsymbol{\rho}_{2}=0\vee 1-|\boldsymbol{\alpha}_{1}|}^{2(r^{SL}_{j_{2}|j_{1}}-|\boldsymbol{\alpha}_{1}|)}S_{n,j_{2}|j_{1},\boldsymbol{\alpha}_{1},\boldsymbol{\rho}_{1},\rho_{2}}$ $\displaystyle\times\boldsymbol{x}^{\boldsymbol{\rho}_{1}}\cdot\frac{\partial^{|\boldsymbol{\alpha}_{1}|}F}{\partial\boldsymbol{\omega}^{\boldsymbol{\alpha}_{1}}}(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})\frac{\partial^{\rho_{2}}\pi}{\partial\xi^{\rho_{2}}}(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})+R_{n,1,1}(\boldsymbol{x})+R_{n,1,2}(\boldsymbol{x})$ $\displaystyle-\sum_{|\boldsymbol{\psi}|=0}^{2}T_{n,j_{2}|j_{1},\boldsymbol{\psi}}\cdot\frac{\partial^{|\boldsymbol{\psi}|}F}{\partial\boldsymbol{\omega}^{\boldsymbol{\psi}}}(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})\exp((\boldsymbol{a}_{j_{1}}^{n})^{\top}\boldsymbol{x})p^{SL,n}_{j_{1}}(y|\boldsymbol{x})-R_{n,2,1}(\boldsymbol{x})-R_{n,2,2}(\boldsymbol{x})\Bigg{]},$ (33) $$

where $S_{n,j_{2}|j_{1},\boldsymbol{\alpha}_{1},\boldsymbol{\rho}_{1},\rho_{2}}=T_{n,j_{2}|j_{1},\boldsymbol{\psi}}=\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{i_{2}|j_{1}}^{n})-\exp(\beta_{j_{2}|j_{1}}^{*})$ for any $j_{2}\in[k^{*}_{2}]$ where $(\boldsymbol{\alpha}_{1},\boldsymbol{\rho}_{1},\rho_{2})=(\boldsymbol{0}_{d},\boldsymbol{0}_{d},0)$ and $\boldsymbol{\psi}=\boldsymbol{0}_{d}$.

Step 1B - Decompose $B_{n}$: By invoking the first-order Taylor expansion, we decompose the term $B_{n}$ defined in equation ([32](#A5.E32)) as

$$ $\displaystyle B_{n}$ $\displaystyle=\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\sum_{|\boldsymbol{\gamma}|=1}(\Delta\boldsymbol{a}_{j_{1}}^{n})^{\boldsymbol{\gamma}}\cdot\boldsymbol{x}^{\boldsymbol{\gamma}}\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})p^{SL}_{G_{n}}(y|\boldsymbol{x})+R_{n,3}(\boldsymbol{x}),$ (34) $$

where $R_{n,3}(\boldsymbol{x})$ is a Taylor remainder such that $R_{n,3}(\boldsymbol{x})/\mathcal{L}_{2n}\to 0$ as $n\to\infty$.

It can be seen from the decomposition in equations ([32](#A5.E32)), ([33](#A5.E33)) and ([34](#A5.E34)) that $A_{n}$, $B_{n}$ and $C_{n}$ can be treated as a linear combination of elements from the following set union:

$$ $\displaystyle\Bigg{\{}\boldsymbol{x}^{\boldsymbol{\rho}_{1}}\cdot\frac{\partial^{|\boldsymbol{\alpha}_{1}|}F}{\partial\boldsymbol{\omega}^{\boldsymbol{\alpha}_{1}}}(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})\frac{\partial^{\rho_{2}}\pi}{\partial\xi^{\rho_{2}}}(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*}):j_{1}\in[k^{*}_{1}],\ j_{2}\in[k^{*}_{2}],$ $\displaystyle\hskip 113.81102pt0\leq|\boldsymbol{\alpha}_{1}|\leq r^{SL}_{j_{2}|j_{1}},\ 0\leq|\boldsymbol{\rho}_{1}|+\rho_{2}\leq 2(r^{SL}_{j_{2}|j_{1}}-|\boldsymbol{\alpha}_{1}|)\Bigg{\}}$ $\displaystyle\cup~{}$ $\displaystyle\Bigg{\{}\frac{\frac{\partial^{|\boldsymbol{\psi}|}F}{\partial\boldsymbol{\omega}^{\boldsymbol{\psi}}}(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})\exp((\boldsymbol{a}_{j_{1}}^{n})^{\top}\boldsymbol{x})p^{SL,n}_{j_{1}}(y|\boldsymbol{x})}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp(-\|\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}}-\boldsymbol{x}\|+\beta^{*}_{j^{\prime}_{2}|j_{1}})}:j_{1}\in[k^{*}_{1}],\ j_{2}\in[k^{*}_{2}],\ 0\leq|\boldsymbol{\psi}|\leq 2\Bigg{\}}$ $\displaystyle\cup~{}$ $\displaystyle\left\{\boldsymbol{x}^{\boldsymbol{\gamma}}\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})p^{SL,n}_{j_{1}}(y|\boldsymbol{x}),\ \boldsymbol{x}^{\boldsymbol{\gamma}}\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})p^{SL}_{G_{n}}(y|\boldsymbol{x}):j_{1}\in[k^{*}_{1}],\ 0\leq|\boldsymbol{\gamma}|\leq 1\right\}.$ $$

Step 2 - Non-vanishing coefficients: In this stage, we illustrate that not all the coefficients in the representation of $A_{n}/\mathcal{L}_{2n}$, $B_{n}/\mathcal{L}_{2n}$ and $C_{n}/\mathcal{L}_{2n}$ go to zero as $n\to\infty$. Suppose that all of them approach zero, then we examine the coefficients associated with the term

- •
$\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})p^{SL,n}_{j_{1}}(y|\boldsymbol{x})$ in $C_{n}/\mathcal{L}_{2n}$, we have
$\displaystyle\frac{1}{\mathcal{L}_{2n}}\cdot\sum_{j_{1}=1}^{k^{*}_{1}}\Big{|}\exp(b_{j_{1}}^{n})-\exp(b_{j_{1}}^{*})\Big{|}\to 0.$
(35)
- •
$\dfrac{F(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})\pi(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp(-\|\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}}-\boldsymbol{x}\|+\beta^{*}_{j^{\prime}_{2}|j_{1}})}$ in $A_{n}/\mathcal{L}_{2n}$, we get that
$\displaystyle\frac{1}{\mathcal{L}_{2n}}\cdot\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\sum_{j_{2}=1}^{k^{*}_{2}}\Big{|}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{i_{2}|j_{1}}^{n})-\exp(\beta_{j_{2}|j_{1}}^{*})\Big{|}\to 0.$
(36)
- •
$\dfrac{\frac{\partial^{|\boldsymbol{\alpha}_{1}|}F}{\partial\boldsymbol{\omega}^{\boldsymbol{\alpha}_{1}}}(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})\exp((\boldsymbol{a}_{j_{1}}^{n})^{\top}\boldsymbol{x})\pi(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp(-\|\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}}-\boldsymbol{x}\|+\beta^{*}_{j^{\prime}_{2}|j_{1}})}$ in $A_{n}/\mathcal{L}_{2n}$ for $j_{1}\in[k^{*}_{1}],j_{2}\in[k^{*}_{2}]:|\mathcal{V}_{j_{2}|j_{1}}|=1$ and $\boldsymbol{\alpha}_{1}=e_{d,u}$ where $e_{d,u}:=(0,\ldots,0,\underbrace{1}_{\textit{u-th}},0,\ldots,0)\in\mathbb{N}^{d}$, we receive
$\displaystyle\frac{1}{\mathcal{L}_{2n}}\cdot\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\sum_{j_{2}\in[k^{*}_{2}]:|\mathcal{V}_{j_{2}|j_{1}}|=1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{i_{2}|j_{1}}^{n})\|\boldsymbol{\omega}_{i_{2}|j_{1}}^{n}-\boldsymbol{\omega}_{j_{2}|j_{1}}^{*}\|_{1}\to 0.$
Note that since the norm-1 is equivalent to the norm-2, then we can replace the norm-1 with the norm-2, that is,
$\displaystyle\frac{1}{\mathcal{L}_{2n}}\cdot\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\sum_{j_{2}\in[k^{*}_{2}]:|\mathcal{V}_{j_{2}|j_{1}}|=1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{i_{2}|j_{1}}^{n})\|\boldsymbol{\omega}_{i_{2}|j_{1}}^{n}-\boldsymbol{\omega}_{j_{2}|j_{1}}^{*}\|\to 0.$
(37)
- •
$\dfrac{F(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})\frac{\partial^{\rho_{2}}\pi}{\partial\xi^{\rho_{2}}}(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp(-\|\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}}-\boldsymbol{x}\|+\beta^{*}_{j^{\prime}_{2}|j_{1}})}$ in $A_{n}/\mathcal{L}_{2n}$ for $j_{1}\in[k^{*}_{1}],j_{2}\in[k^{*}_{2}]:|\mathcal{V}_{j_{2}|j_{1}}|=1$ and $\rho_{2}=1$, we have that
$\displaystyle\frac{1}{\mathcal{L}_{2n}}\cdot\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\sum_{j_{2}\in[k^{*}_{2}]:|\mathcal{V}_{j_{2}|j_{1}}|=1}\exp(\beta_{j_{2}|j_{1}}^{n})|\tau_{j_{1}j_{2}}^{n}-\tau_{j_{1}j_{2}}^{*}|\to 0.$
(38)
- •
$\dfrac{\boldsymbol{x}^{\boldsymbol{\rho}_{1}}F(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})\frac{\partial^{\rho_{2}}\pi}{\partial\xi^{\rho_{2}}}(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp(-\|\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}}-\boldsymbol{x}\|+\beta^{*}_{j^{\prime}_{2}|j_{1}})}$ in $A_{n}/\mathcal{L}_{2n}$ for $j_{1}\in[k^{*}_{1}],j_{2}\in[k^{*}_{2}]:|\mathcal{V}_{j_{2}|j_{1}}|=1$, $\boldsymbol{\rho}_{1}=e_{d,u}$ and $\rho_{2}=1$, we have that
$\displaystyle\frac{1}{\mathcal{L}_{2n}}\cdot\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\sum_{j_{2}\in[k^{*}_{2}]:|\mathcal{V}_{j_{2}|j_{1}}|=1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{j_{2}|j_{1}}^{n})\|\boldsymbol{\eta}_{j_{1}i_{2}}^{n}-\boldsymbol{\eta}_{j_{1}j_{2}}^{*}\|\to 0.$
(39)
- •
$\dfrac{F(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})\frac{\partial^{\rho_{2}}\pi}{\partial\xi^{\rho_{2}}}(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp(-\|\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}}-\boldsymbol{x}\|+\beta^{*}_{j^{\prime}_{2}|j_{1}})}$ in $A_{n}/\mathcal{L}_{2n}$ for $j_{1}\in[k^{*}_{1}],j_{2}\in[k^{*}_{2}]:|\mathcal{V}_{j_{2}|j_{1}}|=1$ and $\rho_{2}=2$, we have that
$\displaystyle\frac{1}{\mathcal{L}_{2n}}\cdot\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\sum_{j_{2}\in[k^{*}_{2}]:|\mathcal{V}_{j_{2}|j_{1}}|=1}\exp(\beta_{j_{2}|j_{1}}^{n})|\nu_{j_{1}j_{2}}^{n}-\nu_{j_{1}j_{2}}^{*}|\to 0.$
(40)
- •
$\boldsymbol{x}^{\boldsymbol{\gamma}}\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})p^{SL}_{G_{n}}(y|\boldsymbol{x})$ in $B_{n}/\mathcal{L}_{2n}$ for $j_{1}\in[k^{*}_{1}]$ and $\boldsymbol{\gamma}=e_{d,u}$, we obtain
$\displaystyle\frac{1}{\mathcal{L}_{2n}}\cdot\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\|\boldsymbol{a}_{j_{1}}^{n}-\boldsymbol{a}_{j_{1}}^{*}\|\to 0.$
(41)
- •
$\dfrac{\frac{\partial^{|\boldsymbol{\alpha}_{1}|}F}{\partial\boldsymbol{\omega}^{\boldsymbol{\alpha}_{1}}}(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})\pi(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp(-\|\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}}-\boldsymbol{x}\|+\beta^{*}_{j^{\prime}_{2}|j_{1}})}$ in $A_{n}/\mathcal{L}_{2n}$ for $j_{1}\in[k^{*}_{1}],j_{2}\in[k^{*}_{2}]:|\mathcal{V}_{j_{2}|j_{1}}|>1$ and $\boldsymbol{\alpha}_{1}=2e_{d,u}$, we receive that
$\displaystyle\frac{1}{\mathcal{L}_{2n}}\cdot\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\sum_{j_{2}\in[k^{*}_{2}]:|\mathcal{V}_{j_{2}|j_{1}}|>1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{i_{2}|j_{1}}^{n})\|\boldsymbol{\omega}_{i_{2}|j_{1}}^{n}-\boldsymbol{\omega}_{j_{2}|j_{1}}^{*}\|^{2}\to 0.$
(42)

Putting the above limits together with the formulation of the loss $\mathcal{L}_{2n}$ in equation ([31](#A5.E31)), we deduce that

$$ $\displaystyle\frac{1}{\mathcal{L}_{2n}}\cdot\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\Bigg{[}\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|>1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta^{n}_{i_{2}|j_{1}})\Big{(}\|\Delta\boldsymbol{\eta}_{j_{1}i_{2}j_{2}}^{n}\|^{\frac{r^{SL}_{j_{2}|j_{1}}}{2}}+|\Delta\tau_{j_{1}i_{2}j_{2}}^{n}|^{r^{SL}_{j_{2}|j_{1}}}$ $\displaystyle\hskip 256.0748pt+|\Delta\nu_{j_{1}i_{2}j_{2}}^{n}|^{\frac{r^{SL}_{j_{2}|j_{1}}}{2}}\Big{)}\Bigg{]}\not\to 0,$ $$

which also suggests that

$$ $\displaystyle\frac{1}{\mathcal{L}_{2n}}\cdot\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\Bigg{[}\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|>1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta^{n}_{i_{2}|j_{1}})\Big{(}\|\Delta\boldsymbol{a}_{j_{1}}^{n}\|^{r^{SL}_{j_{2}|j_{1}}}+\|\Delta\boldsymbol{\eta}_{j_{1}i_{2}j_{2}}^{n}\|^{\frac{r^{SL}_{j_{2}|j_{1}}}{2}}$ $\displaystyle\hskip 170.71652pt+|\Delta\tau_{j_{1}i_{2}j_{2}}^{n}|^{r^{SL}_{j_{2}|j_{1}}}+|\Delta\nu_{j_{1}i_{2}j_{2}}^{n}|^{\frac{r^{SL}_{j_{2}|j_{1}}}{2}}\Big{)}\Bigg{]}\not\to 0,$ $$

as $n\to\infty$. Thus, we can find indices $j^{*}_{1}\in[k^{*}_{1}]$ and $j^{*}_{2}\in[k^{*}_{2}]:|\mathcal{V}_{j^{*}_{2}|j^{*}_{1}}|>1$ such that

$$ $\displaystyle\frac{1}{\mathcal{L}_{2n}}\cdot\sum_{i_{2}\in\mathcal{V}_{j^{*}_{2}|j^{*}_{1}}}\exp(\beta^{n}_{i_{2}|j^{*}_{1}})\Big{(}\|\boldsymbol{a}^{n}_{j^{*}_{1}}-\boldsymbol{a}^{*}_{j^{*}_{1}}\|^{r^{SL}_{j^{*}_{2}|j^{*}_{1}}}+\|\boldsymbol{\eta}^{n}_{j^{*}_{1}i_{2}}-\boldsymbol{\eta}^{*}_{j^{*}_{1}j^{*}_{2}}\|^{\frac{r^{SL}_{j^{*}_{2}|j^{*}_{1}}}{2}}$ $\displaystyle\hskip 113.81102pt+|\tau^{n}_{j^{*}_{1}i_{2}}-\tau^{*}_{j^{*}_{1}j^{*}_{2}}|^{r^{SL}_{j^{*}_{2}|j^{*}_{1}}}+|\nu^{n}_{j^{*}_{1}i_{2}}-\nu^{*}_{j^{*}_{1}j^{*}_{2}}|^{\frac{r^{SL}_{j^{*}_{2}|j^{*}_{1}}}{2}}\Big{)}\not\to 0.$ (43) $$

WLOG, we may assume that $j^{*}_{1}=j^{*}_{2}=1$. By considering the coefficients of the terms

$$ $\dfrac{\boldsymbol{x}^{\boldsymbol{\rho}_{1}}F(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})\frac{\partial^{\rho_{2}}\pi}{\partial\xi^{\rho_{2}}}(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp(-\|\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}}-\boldsymbol{x}\|+\beta^{*}_{j^{\prime}_{2}|j_{1}})}$ $$

in $A_{n}/\mathcal{L}_{2n}$ for $j_{1}=j_{2}=1$, we have $\exp(b^{n}_{1})S_{n,1|1,\boldsymbol{0}_{d},\boldsymbol{\rho}_{1},\rho_{2}}/\mathcal{L}_{2n}\to 0$, or equivalently,

$$ $\displaystyle\frac{1}{\mathcal{L}_{2n}}\cdot\sum_{i_{2}\in\mathcal{V}_{1|1}}\sum_{(\boldsymbol{\alpha}_{2},\boldsymbol{\alpha}_{3},\alpha_{4},\alpha_{5})\in\mathcal{I}^{SL}_{\boldsymbol{\rho}_{1},\rho_{2}}}\frac{\exp(\beta^{n}_{i_{2}|1})}{2^{\alpha_{5}}\boldsymbol{\alpha}_{2}!\boldsymbol{\alpha}_{3}!\alpha_{4}!\alpha_{5}!}\cdot(\Delta\boldsymbol{a}^{n}_{1})^{\boldsymbol{\alpha}_{2}}(\Delta\boldsymbol{\eta}^{n}_{1i_{2}1})^{\boldsymbol{\alpha}_{3}}$ $\displaystyle\hskip 199.16928pt\times(\Delta\tau^{n}_{1i_{2}1})^{\alpha_{4}}(\Delta\nu^{n}_{1i_{2}1})^{\alpha_{5}}\to 0.$ (44) $$

By dividing the left hand side of equation ([44](#A5.E44)) by that of equation ([43](#A5.E43)), we get

$$ $\displaystyle\dfrac{\sum_{i_{2}\in\mathcal{V}_{1|1}}\sum_{(\boldsymbol{\alpha}_{2},\boldsymbol{\alpha}_{3},\alpha_{4},\alpha_{5})\in\mathcal{I}^{SL}_{\boldsymbol{\rho}_{1},\rho_{2}}}\frac{\exp(\beta^{n}_{i_{2}|1})}{2^{\alpha_{5}}\boldsymbol{\alpha}_{2}!\boldsymbol{\alpha}_{3}!\alpha_{4}!\alpha_{5}!}\cdot(\Delta\boldsymbol{a}^{n}_{1})^{\boldsymbol{\alpha}_{2}}(\Delta\boldsymbol{\eta}^{n}_{1i_{2}1})^{\boldsymbol{\alpha}_{3}}(\Delta\tau^{n}_{1i_{2}1})^{\alpha_{4}}(\Delta\nu^{n}_{1i_{2}1})^{\alpha_{5}}}{\sum_{i_{2}\in\mathcal{V}_{1|1}}\exp(\beta^{n}_{i_{2}|1})\Big{(}\|\Delta\boldsymbol{a}^{n}_{1}\|^{r^{SL}_{1|1}}+\|\Delta\boldsymbol{\eta}^{n}_{1i_{2}i}\|^{\frac{r^{SL}_{1|1}}{2}}+|\Delta\tau^{n}_{1i_{2}1}|^{r^{SL}_{1|1}}+|\Delta\nu^{n}_{1i_{2}1}|^{\frac{r^{SL}_{1|1}}{2}}\Big{)}}\to 0.$ (45) $$

Let us define $\overline{M}_{n}:=\max\{\|\Delta\boldsymbol{a}^{n}_{1}\|,\|\Delta\boldsymbol{\eta}^{n}_{1i_{2}i}\|^{1/2},\|\Delta\tau^{n}_{1i_{2}1}\|,\|\Delta\nu^{n}_{1i_{2}1}\|^{1/2}:i_{2}\in\mathcal{V}_{1|1}\}$, and $\overline{\beta}_{n}:=\max_{i_{2}\in\mathcal{V}_{1|1}}\exp(\beta^{n}_{i_{2}|1})$. Since the sequence $\exp(\beta^{n}_{i_{2}|1})/\overline{\beta}_{n}$ is bounded, we can replace it by its subsequence which has a positive limit $p^{2}_{i_{2}}:=\lim_{n\to\infty}\exp(\beta^{n}_{i_{2}|1})/\overline{\beta}_{n}$. Note that at least one among the limits $p^{2}_{i_{2}}$ must be equal to one. Next, let us define

$$ $\displaystyle(\Delta\boldsymbol{a}^{n}_{1})/\overline{M}_{n}\to\boldsymbol{q}_{2},$ $\displaystyle\quad(\Delta\boldsymbol{\eta}^{n}_{1i_{2}1})/\overline{M}_{n}\to\boldsymbol{q}_{3i_{2}},$ $\displaystyle(\Delta\tau^{n}_{1i_{2}1})/\overline{M}_{n}\to q_{4i_{2}},$ $\displaystyle\quad(\Delta\nu^{n}_{1i_{2}1})/2\overline{M}_{n}\to q_{5i_{2}}.$ $$

Note that at least one among $q_{2},q_{3i_{2}},q_{4i_{2}},q_{5i_{2}}$ must be equal to either 1 or $-1$.

By dividing both the numerator and the denominator of the term in equation ([45](#A5.E45)) by $\overline{\beta}_{n}\overline{M}^{|\boldsymbol{\rho}_{1}|+\rho_{2}}_{n}$, we obtain the system of polynomial equations:

$$ $\displaystyle\sum_{i_{2}\in\mathcal{V}_{1|1}}\sum_{(\boldsymbol{\alpha}_{2},\boldsymbol{\alpha}_{3},\alpha_{4},\alpha_{5})\in\mathcal{I}^{SL}_{\boldsymbol{\rho}_{1},\rho_{2}}}\frac{1}{\boldsymbol{\alpha}_{2}!\boldsymbol{\alpha}_{3}!\alpha_{4}!\alpha_{5}!}\cdot p^{2}_{i_{2}}\boldsymbol{q}_{2}^{\boldsymbol{\alpha}_{2}}\boldsymbol{q}_{3i_{2}}^{\boldsymbol{\alpha}_{3}}q_{4i_{2}}^{\alpha_{4}}q_{5i_{2}}^{\alpha_{5}}=0,\quad 1\leq|\boldsymbol{\rho}_{1}|+\rho_{2}\leq r^{SL}_{1|1}.$ $$

According to the definition of the term $r^{SL}_{1|1}$, the above system does not have any non-trivial solutions, which is a contradiction. Consequently, at least one among the coefficients in the representation of $A_{n}/\mathcal{L}_{2n}$, $B_{n}/\mathcal{L}_{2n}$ and $C_{n}/\mathcal{L}_{2n}$ must not converge to zero as $n\to\infty$.

Step 3 - Application of the Fatou’s lemma. In this stage, we show that all the coefficients in the formulations of $A_{n}/\mathcal{L}_{2n}$, $B_{n}/\mathcal{L}_{2n}$ and $C_{n}/\mathcal{L}_{2n}$ go to zero as $n\to\infty$. Denote by $m_{n}$ the maximum of the absolute values of those coefficients, the result from Step 2 induces that $1/m_{n}\not\to\infty$. By employing the Fatou’s lemma, we have

$$ $\displaystyle 0=\lim_{n\to\infty}\dfrac{\mathbb{E}_{\boldsymbol{X}}[V(p^{SL}_{G_{n}}(\cdot|\boldsymbol{X}),p^{SL}_{G_{*}}(\cdot|\boldsymbol{X}))]}{m_{n}\mathcal{L}_{2n}}\geq\int\liminf_{n\to\infty}\dfrac{|p^{SL}_{G_{n}}(y|\boldsymbol{x})-p^{SL}_{G_{*}}(y|\boldsymbol{x})|}{2m_{n}\mathcal{L}_{2n}}\mathrm{d}(\boldsymbol{x},y).$ $$

Thus, we deduce that

$$ $\displaystyle\dfrac{|p^{SL}_{G_{n}}(y|\boldsymbol{x})-p^{SL}_{G_{*}}(y|\boldsymbol{x})|}{2m_{n}\mathcal{L}_{2n}}\to 0,$ $$

which results in $Q_{n}/[m_{n}\mathcal{L}_{2n}]\to 0$ as $n\to\infty$ for almost surely $(\boldsymbol{x},y)$.

Next, we denote

$$ $\displaystyle\frac{\exp(b_{j_{1}}^{n})S_{n,j_{2}|j_{1},\boldsymbol{\alpha}_{1},\boldsymbol{\rho}_{1},\rho_{2}}}{m_{n}\mathcal{L}_{2n}}\to\phi_{j_{2}|j_{1},\boldsymbol{\alpha}_{1},\boldsymbol{\rho}_{1},\rho_{2}},\quad$ $\displaystyle\frac{\exp(b_{j_{1}}^{n})T_{n,j_{2}|j_{1},\boldsymbol{\psi}}}{m_{n}\mathcal{L}_{2n}}\to\varphi_{j_{2}|j_{1},\boldsymbol{\psi}},$ $\displaystyle\frac{\exp(b_{j_{1}}^{n})(\Delta\boldsymbol{a}_{j_{1}}^{n})^{\boldsymbol{\gamma}}}{m_{n}\mathcal{L}_{2n}}\to\lambda_{j_{1},\boldsymbol{\gamma}},\quad$ $\displaystyle\frac{\exp(b_{j_{1}}^{n})-\exp(b_{j_{1}}^{*})}{m_{n}\mathcal{L}_{2n}}\to\chi_{j_{1}}$ $$

with a note that at least one among them is non-zero. Then, the decomposition of $Q_{n}$ in equation ([32](#A5.E32)) indicates that

$$ $\displaystyle\lim_{n\to\infty}\frac{Q_{n}}{m_{n}\mathcal{L}_{2n}}=\lim_{n\to\infty}\frac{A_{n}}{m_{n}\mathcal{L}_{2n}}-\lim_{n\to\infty}\frac{B_{n}}{m_{n}\mathcal{L}_{2n}}+\lim_{n\to\infty}\frac{C_{n}}{m_{n}\mathcal{L}_{2n}},$ $$

in which

$$ $\displaystyle\lim_{n\to\infty}\frac{A_{n}}{m_{n}\mathcal{L}_{2n}}=\sum_{j_{1}=1}^{k^{*}_{1}}\sum_{j_{2}=1}^{k^{*}_{2}}\Bigg{[}\sum_{|\boldsymbol{\alpha}_{1}|=1}^{r^{SL}_{j_{2}|j_{1}}}\sum_{|\boldsymbol{\rho}_{1}|+\boldsymbol{\rho}_{2}=0\vee 1-|\boldsymbol{\alpha}_{1}|}^{2(r^{SL}_{j_{2}|j_{1}}-|\boldsymbol{\alpha}_{1}|)}S_{n,j_{2}|j_{1},\boldsymbol{\alpha}_{1},\boldsymbol{\rho}_{1},\rho_{2}}\cdot\boldsymbol{x}^{\boldsymbol{\rho}_{1}}\frac{\partial^{|\boldsymbol{\alpha}_{1}|}F}{\partial\boldsymbol{\omega}^{\boldsymbol{\alpha}_{1}}}(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})$ $\displaystyle\times\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})\frac{\partial^{\rho_{2}}\pi}{\partial\xi^{\rho_{2}}}(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})-\sum_{|\boldsymbol{\psi}|=0}^{2}\varphi_{j_{2}|j_{1},\boldsymbol{\psi}}\cdot\frac{\partial^{|\boldsymbol{\psi}|}F}{\partial\boldsymbol{\omega}^{\boldsymbol{\psi}}}(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})$ $\displaystyle\hskip 85.35826pt\times\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})p^{SL,*}_{j_{1}}(y|\boldsymbol{x})\Bigg{]}\frac{1}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp(-\|\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}}-\boldsymbol{x}\|+\beta^{*}_{j^{\prime}_{2}|j_{1}})},$ $\displaystyle\lim_{n\to\infty}\frac{B_{n}}{m_{n}\mathcal{L}_{2n}}=\sum_{j_{1}=1}^{k^{*}_{1}}\sum_{|\gamma|=1}\lambda_{j_{1},\boldsymbol{\gamma}}\cdot\boldsymbol{x}^{\boldsymbol{\gamma}}\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})p^{SL}_{G_{*}}(y|\boldsymbol{x}),$ $\displaystyle\lim_{n\to\infty}\frac{C_{n}(\boldsymbol{x})}{m_{n}\mathcal{L}_{2n}}=\sum_{j_{1}=1}^{k^{*}_{1}}\chi_{j_{1}}\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})\left[p^{SL,*}_{j_{1}}(y|\boldsymbol{x})-p^{SL}_{G_{*}}(y|\boldsymbol{x})\right].$ $$

Since the set

$$ $\displaystyle\Bigg{\{}\dfrac{\boldsymbol{x}^{\boldsymbol{\rho}_{1}}\frac{\partial^{|\boldsymbol{\alpha}_{1}|}F}{\partial\boldsymbol{\omega}^{\boldsymbol{\alpha}_{1}}}(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})\frac{\partial^{\rho_{2}}\pi}{\partial\xi^{\rho_{2}}}(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp(-\|\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}}-\boldsymbol{x}\|+\beta^{*}_{j^{\prime}_{2}|j_{1}})}:j_{1}\in[k^{*}_{1}],j_{2}\in[k^{*}_{2}],$ $\displaystyle\hskip 170.71652pt0\leq|\boldsymbol{\alpha}_{1}|\leq r^{SL}_{j_{2}|j_{1}},0\leq|\boldsymbol{\rho}_{1}|+\rho_{2}\leq 2(r^{SL}_{j_{2}|j_{1}}-|\boldsymbol{\alpha}_{1}|)\Bigg{\}}$ $\displaystyle\cup$ $\displaystyle~{}\Bigg{\{}\dfrac{\frac{\partial^{|\boldsymbol{\psi}|}F}{\partial\boldsymbol{\omega}^{\boldsymbol{\psi}}}(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})p^{SL,*}_{j_{1}}(y|\boldsymbol{x})}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp(-\|\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}}-\boldsymbol{x}\|+\beta^{*}_{j^{\prime}_{2}|j_{1}})}:j_{1}\in[k^{*}_{1}],j_{2}\in[k^{*}_{2}],0\leq|\boldsymbol{\psi}|\leq 2\Bigg{\}}$ $\displaystyle\cup$ $\displaystyle~{}\Big{\{}\boldsymbol{x}^{\boldsymbol{\gamma}}\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})p^{SL}_{G_{*}}(y|\boldsymbol{x}),\ \exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})p^{SL,*}_{j_{1}}(y|\boldsymbol{x}),\ \exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})p^{SL}_{G_{*}}(y|\boldsymbol{x})$ $\displaystyle\hskip 227.62204pt:j_{1}\in[k^{*}_{1}],0\leq|\boldsymbol{\gamma}|\leq 2\Big{\}}$ $$

is linearly independent, we obtain that $\phi_{j_{2}|j_{1},\boldsymbol{\alpha}_{1},\boldsymbol{\rho}_{1},\rho_{2}}=\varphi_{j_{2}|j_{1},\boldsymbol{\psi}}=\lambda_{j_{1},\boldsymbol{\gamma}}=\chi_{j_{1}}=0$ for all $j_{1}\in[k^{*}_{1}]$, $j_{2}\in[k^{*}_{2}]$, $0\leq|\boldsymbol{\alpha}_{1}|\leq r^{SL}_{j_{2}|j_{1}}$, $0\leq|\boldsymbol{\rho}_{1}|+\rho_{2}\leq 2(r^{SL}_{j_{2}|j_{1}}-|\boldsymbol{\alpha}_{1}|)$, $0\leq|\boldsymbol{\psi}|\leq 2$ and $0\leq|\boldsymbol{\gamma}|\leq 1$, which is a contradiction. As a consequence, we obtain the inequality in equation ([12](#A5.E12)). Hence, the proof is completed.

### E.3 When t ​ y ​ p ​ e = L ​ L 𝑡 𝑦 𝑝 𝑒 𝐿 𝐿 type=LL

When $type=LL$, the corresponding Voronoi loss function is $\mathcal{L}_{(2,r^{LL},\frac{1}{2}r^{LL})}(G_{n},G_{*})=\mathcal{L}_{3n}$ where we define

$$ $\displaystyle\mathcal{L}_{3n}:=\sum_{j_{1}=1}^{k^{*}_{1}}\Big{|}\exp(b^{n}_{j_{1}})-\exp(b^{*}_{j_{1}})\Big{|}+\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b^{n}_{j_{1}})\|\Delta\boldsymbol{a}_{j_{1}}^{n}\|+\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b^{n}_{j_{1}})$ $\displaystyle\times\Bigg{[}\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|=1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta^{n}_{i_{2}|j_{1}})\Big{(}\|\Delta\boldsymbol{\omega}_{i_{2}j_{2}|j_{1}}^{n}\|+\|\Delta\boldsymbol{\eta}_{j_{1}i_{2}j_{2}}^{n}\|+|\Delta\tau_{j_{1}i_{2}j_{2}}^{n}|+|\Delta\nu_{j_{1}i_{2}j_{2}}^{n}|\Big{)}$ $\displaystyle+\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|>1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta^{n}_{i_{2}|j_{1}})\Big{(}\|\Delta\boldsymbol{\omega}_{i_{2}j_{2}|j_{1}}^{n}\|^{2}+\|\Delta\boldsymbol{\eta}_{j_{1}i_{2}j_{2}}^{n}\|^{2}+|\Delta\tau_{j_{1}i_{2}j_{2}}^{n}|^{r^{LL}_{j_{2}|j_{1}}}$ $\displaystyle+|\Delta\nu_{j_{1}i_{2}j_{2}}^{n}|^{\frac{r^{LL}_{j_{2}|j_{1}}}{2}}\Big{)}\Bigg{]}+\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b^{n}_{j_{1}})\sum_{j_{2}=1}^{k^{*}_{2}}\Big{|}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta^{n}_{i_{2}|j_{1}})-\exp(\beta^{*}_{j_{2}|j_{1}})\Big{|}.$ (46) $$

Step 1 - Taylor expansion: In this step, we use the Taylor expansion to decompose the term

$$ $\displaystyle Q_{n}:=\left[\sum_{j_{1}=1}^{k^{*}_{1}}\exp(-\|\boldsymbol{a}_{j_{1}}^{*}-\boldsymbol{x}\|+b_{j_{1}}^{*})\right][p^{LL}_{G_{n}}(y|\boldsymbol{x})-p^{LL}_{G_{*}}(y|\boldsymbol{x})].$ $$

Prior to that, let us denote

$$ $\displaystyle p^{LL,n}_{j_{1}}(y|\boldsymbol{x})$ $\displaystyle:=\sum_{j_{2}=1}^{k^{*}_{2}}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\sigma(-\|\boldsymbol{\omega}_{i_{2}|j_{1}}^{n}-\boldsymbol{x}\|+\beta^{n}_{i_{2}|j_{1}})\pi(y|(\boldsymbol{\eta}_{j_{1}i_{2}}^{n})^{\top}\boldsymbol{x}+\tau_{j_{1}i_{2}}^{n},\nu_{j_{1}i_{2}}^{n}),$ $\displaystyle p^{LL,*}_{j_{1}}(y|\boldsymbol{x})$ $\displaystyle:=\sum_{j_{2}=1}^{k^{*}_{2}}\sigma(-\|\boldsymbol{\omega}_{j_{2}|j_{1}}^{*}-\boldsymbol{x}\|+\beta_{j_{2}|j_{1}}^{*})\pi(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*}).$ $$

Then, the quantity $Q_{n}$ is divided into three terms as

$$ $\displaystyle Q_{n}$ $\displaystyle=\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\left[\exp(-\|\boldsymbol{a}_{j_{1}}^{n}-\boldsymbol{x}\|)p^{LL,n}_{j_{1}}(y|\boldsymbol{x})-\exp(-\|\boldsymbol{a}_{j_{1}}^{*}-\boldsymbol{x}\|)p^{LL,*}_{j_{1}}(y|\boldsymbol{x})\right]$ $\displaystyle-\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\left[\exp(-\|\boldsymbol{a}_{j_{1}}^{n}-\boldsymbol{x}\|)-\exp(-\|\boldsymbol{a}_{j_{1}}^{*}-\boldsymbol{x}\|)\right]p^{LL}_{G_{n}}(y|\boldsymbol{x})$ $\displaystyle+\sum_{j_{1}=1}^{k^{*}_{1}}\left(\exp(b_{j_{1}}^{n})-\exp(b_{j_{1}}^{*})\right)\exp(-\|\boldsymbol{a}_{j_{1}}^{*}-\boldsymbol{x}\|)\left[p^{LL,n}_{j_{1}}(y|\boldsymbol{x})-p^{LL}_{G_{n}}(y|\boldsymbol{x})\right]$ $\displaystyle:$ $\displaystyle=A_{n}-B_{n}+C_{n}.$ (47) $$

Step 1A - Decompose $A_{n}$: We continue to decompose $A_{n}$:

$$ $\displaystyle A_{n}$ $\displaystyle:=\sum_{j_{1}=1}^{k^{*}_{1}}\frac{\exp(b_{j_{1}}^{n})}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp(-\|\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}}-\boldsymbol{x}\|+\beta^{*}_{j^{\prime}_{2}|j_{1}})}[A_{n,j_{1},1}+A_{n,j_{1},2}+A_{n,j_{1},3}],$ $$

in which

$$ $\displaystyle A_{n,j_{1},1}$ $\displaystyle:=\sum_{j_{2}=1}^{k^{*}_{2}}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta^{n}_{i_{2}|j_{1}})\Big{[}\exp(-\|\boldsymbol{\omega}_{i_{2}|j_{1}}^{n}-\boldsymbol{x}\|)\exp(-\|\boldsymbol{a}_{j_{1}}^{n}-\boldsymbol{x}\|)\pi(y|(\boldsymbol{\eta}_{j_{1}i_{2}}^{n}){\top}\boldsymbol{x}+\tau_{j_{1}i_{2}}^{n},\nu_{j_{1}i_{2}}^{n})$ $\displaystyle\hskip 56.9055pt-\exp(-\|\boldsymbol{\omega}_{j_{2}|j_{1}}^{*}-\boldsymbol{x}\|)\exp(-\|\boldsymbol{a}_{j_{1}}^{*}-\boldsymbol{x}\|)\pi(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})\Big{]},$ $\displaystyle A_{n,j_{1},2}$ $\displaystyle:=\sum_{j_{2}=1}^{k^{*}_{2}}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta^{n}_{i_{2}|j_{1}})\Big{[}\exp(-\|\boldsymbol{\omega}_{i_{2}|j_{1}}^{n}-\boldsymbol{x}\|)-\exp(-\|\boldsymbol{\omega}_{j_{2}|j_{1}}^{*}-\boldsymbol{x}\|)\Big{]}$ $\displaystyle\hskip 199.16928pt\times\exp(-\|\boldsymbol{a}_{j_{1}}^{n}-\boldsymbol{x}\|)p^{LL,n}_{j_{1}}(y|\boldsymbol{x}),$ $\displaystyle A_{n,j_{1},3}$ $\displaystyle:=\sum_{j_{2}=1}^{k^{*}_{2}}\Big{(}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{i_{2}|j_{1}}^{n})-\exp(\beta_{j_{2}|j_{1}}^{*})\Big{)}\exp(-\|\boldsymbol{\omega}_{j_{2}|j_{1}}^{*}-\boldsymbol{x}\|)$ $\displaystyle\hskip 28.45274pt\times[\exp(-\|\boldsymbol{a}_{j_{1}}^{*}-\boldsymbol{x}\|)\pi(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})-\exp(-\|\boldsymbol{a}_{j_{1}}^{n}-\boldsymbol{x}\|)p^{LL,n}_{j_{1}}(y|\boldsymbol{x})].$ $$

Firstly, we separate the term $A_{n,j_{1},1}$ into two parts based on the cardinality of the Voronoi cells $\mathcal{V}_{j_{2}|j_{1}}$ as

$$ $\displaystyle A_{n,j_{1},1}$ $\displaystyle=\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|=1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{i_{2}|j_{1}}^{n})\Big{[}\exp(-\|\boldsymbol{\omega}_{i_{2}|j_{1}}^{n}-\boldsymbol{x}\|)\exp(-\|\boldsymbol{a}_{j_{1}}^{n}-\boldsymbol{x}\|)\pi(y|(\boldsymbol{\eta}_{j_{1}i_{2}}^{n})^{\top}\boldsymbol{x}+\tau_{j_{1}i_{2}}^{n},\nu_{j_{1}i_{2}}^{n})$ $\displaystyle\hskip 56.9055pt-\exp(-\|\boldsymbol{\omega}_{j_{2}|j_{1}}^{*}-\boldsymbol{x}\|)\exp(-\|\boldsymbol{a}_{j_{1}}^{*}-\boldsymbol{x}\|)\pi(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})\Big{]},$ $\displaystyle+\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|>1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{i_{2}|j_{1}}^{n})\Big{[}\exp(-\|\boldsymbol{\omega}_{i_{2}|j_{1}}^{n}-\boldsymbol{x}\|)\exp(-\|\boldsymbol{a}_{j_{1}}^{n}-\boldsymbol{x}\|)\pi(y|(\boldsymbol{\eta}_{j_{1}i_{2}}^{n})^{\top}\boldsymbol{x}+\tau_{j_{1}i_{2}}^{n},\nu_{j_{1}i_{2}}^{n})$ $\displaystyle\hskip 56.9055pt-\exp(-\|\boldsymbol{\omega}_{j_{2}|j_{1}}^{*}-\boldsymbol{x}\|)\exp(-\|\boldsymbol{a}_{j_{1}}^{*}-\boldsymbol{x}\|)\pi(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})\Big{]}$ $\displaystyle:$ $\displaystyle=A_{n,j_{1},1,1}+A_{n,j_{1},1,2}.$ $$

By denoting $F(\boldsymbol{x};\boldsymbol{\omega}):=\exp(-\|\boldsymbol{\omega}-\boldsymbol{x}\|)$ and employing the first-order Taylor expansion, we can represent $A_{n,j_{1},1,1}$ as

$$ $\displaystyle A_{n,j_{1},1,1}=\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|=1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\sum_{|\boldsymbol{\alpha}|=1}\frac{\exp(\beta_{i_{2}|j_{1}}^{n})}{2^{\alpha_{5}!}\boldsymbol{\alpha}!}(\Delta\boldsymbol{\omega}_{i_{2}j_{2}|j_{1}}^{n})^{\boldsymbol{\alpha}_{1}}(\Delta\boldsymbol{a}_{j_{1}}^{n})^{\boldsymbol{\alpha}_{2}}(\Delta\boldsymbol{\eta}_{j_{1}i_{2}j_{2}}^{n})^{\boldsymbol{\alpha}_{3}}(\Delta\tau_{j_{1}i_{2}j_{2}}^{n})^{\alpha_{4}}$ $\displaystyle\times(\Delta\nu_{j_{1}i_{2}j_{2}}^{n})^{\alpha_{5}}\boldsymbol{x}^{\boldsymbol{\alpha}_{3}}\frac{\partial^{|\boldsymbol{\alpha}_{1}|}F}{\partial\boldsymbol{\omega}^{\boldsymbol{\alpha}_{1}}}(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})\frac{\partial^{|\boldsymbol{\alpha}_{2}|}F}{\partial\boldsymbol{a}^{\boldsymbol{\alpha}_{2}}}(\boldsymbol{x};\boldsymbol{a}_{j_{1}}^{*})\frac{\partial^{|\boldsymbol{\alpha}_{3}|+\alpha_{4}+2\alpha_{5}}\pi}{\partial\xi^{|\boldsymbol{\alpha}_{3}|+\alpha_{4}+2\alpha_{5}}}(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})+R_{n,1,1}(\boldsymbol{x})$ $\displaystyle=\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|=1}\sum_{|\boldsymbol{\alpha}_{1}|+|\boldsymbol{\alpha}_{2}|+|\boldsymbol{\alpha}_{3}|=0}^{1}\sum_{\rho=0\vee 1-|\boldsymbol{\alpha}_{1}|-|\boldsymbol{\alpha}_{2}|-|\boldsymbol{\alpha}_{3}|}^{2(1-|\boldsymbol{\alpha}_{1}|-|\boldsymbol{\alpha}_{2}|-|\boldsymbol{\alpha}_{3}|)}S_{n,j_{2}|j_{1},\boldsymbol{\alpha}_{1},\boldsymbol{\alpha}_{2},\boldsymbol{\alpha}_{3},\rho}\cdot\boldsymbol{x}^{\boldsymbol{\alpha}_{3}}\frac{\partial^{|\boldsymbol{\alpha}_{1}|}F}{\partial\boldsymbol{\omega}^{\boldsymbol{\alpha}_{1}}}(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})$ $\displaystyle\hskip 113.81102pt\times\frac{\partial^{|\boldsymbol{\alpha}_{2}|}F}{\partial\boldsymbol{a}^{\boldsymbol{\alpha}_{2}}}(\boldsymbol{x};\boldsymbol{a}_{j_{1}}^{*})\frac{\partial^{|\boldsymbol{\alpha}_{3}|+\rho}\pi}{\partial\xi^{|\boldsymbol{\alpha}_{3}|+\rho}}(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})+R_{n,1,1}(\boldsymbol{x}),$ $$

where $R_{n,1,1}(\boldsymbol{x},y)$ is a Taylor remainder such that $R_{n,1,1}(\boldsymbol{x},y)/\mathcal{L}_{3n}\to 0$ as $n\to\infty$, and

$$ $\displaystyle S_{n,j_{2}|j_{1},\boldsymbol{\alpha}_{1},\boldsymbol{\alpha}_{2},\boldsymbol{\alpha}_{3},\rho}$ $\displaystyle:=\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\sum_{\alpha_{4}+2\alpha_{5}=\rho}\frac{\exp(\beta_{i_{2}|j_{1}}^{n})}{2^{\alpha_{5}}\boldsymbol{\alpha}!}(\Delta\boldsymbol{\omega}_{i_{2}j_{2}|j_{1}}^{n})^{\boldsymbol{\alpha}_{1}}(\Delta\boldsymbol{a}_{j_{1}}^{n})^{\boldsymbol{\alpha}_{2}}(\Delta\boldsymbol{\eta}_{j_{1}i_{2}j_{2}}^{n})^{\boldsymbol{\alpha}_{3}}$ $\displaystyle\hskip 170.71652pt\times(\Delta\tau_{j_{1}i_{2}j_{2}}^{n})^{\alpha_{4}}(\Delta\nu_{j_{1}i_{2}j_{2}}^{n})^{\alpha_{5}},$ $$

for any $(\boldsymbol{\alpha}_{1},\boldsymbol{\alpha}_{2},\boldsymbol{\alpha}_{3},\rho)\neq(\boldsymbol{0}_{d},\boldsymbol{0}_{d},\boldsymbol{0}_{d},0)$, $j_{1}\in[k^{*}_{1}]$ and $j_{2}\in[k^{*}_{2}]$.

For each $(j_{1},j_{2})\in[k^{*}_{1}]\times[k^{*}_{2}]$, by invoking the Taylor expansion of order $r^{LL}(|\mathcal{V}_{j_{2}|j_{1}}|):=r^{LL}_{j_{2}|j_{1}}$, the term $A_{n,j_{1},1,2}$ can be represented as

$$ $\displaystyle A_{n,j_{1},1,2}=\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|>1}\sum_{|\boldsymbol{\alpha}_{1}|+|\boldsymbol{\alpha}_{2}|+|\boldsymbol{\alpha}_{3}|=0}^{r^{LL}_{j_{2}|j_{1}}}\sum_{\rho=0\vee 1-|\boldsymbol{\alpha}_{1}|-|\boldsymbol{\alpha}_{2}|-|\boldsymbol{\alpha}_{3}|}^{2(r^{LL}_{j_{2}|j_{1}}-|\boldsymbol{\alpha}_{1}|-|\boldsymbol{\alpha}_{2}|-|\boldsymbol{\alpha}_{3}|)}S_{n,j_{2}|j_{1},\boldsymbol{\alpha}_{1},\boldsymbol{\alpha}_{2},\boldsymbol{\alpha}_{3},\rho}\cdot\boldsymbol{x}^{\boldsymbol{\alpha}_{3}}$ $\displaystyle\hskip 28.45274pt\times\frac{\partial^{|\boldsymbol{\alpha}_{1}|}F}{\partial\boldsymbol{\omega}^{\boldsymbol{\alpha}_{1}}}(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})\frac{\partial^{|\boldsymbol{\alpha}_{2}|}F}{\partial\boldsymbol{a}^{\boldsymbol{\alpha}_{2}}}(\boldsymbol{x};\boldsymbol{a}_{j_{1}}^{*})\frac{\partial^{|\boldsymbol{\alpha}_{3}|+\rho}\pi}{\partial\xi^{|\boldsymbol{\alpha}_{3}|+\rho}}(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})+R_{n,1,2}(\boldsymbol{x},y),$ $$

where $R_{n,1,2}(\boldsymbol{x},y)$ is a Taylor remainder such that $R_{n,1,2}(\boldsymbol{x},y)/\mathcal{L}_{3n}\to 0$ as $n\to\infty$.

Secondly, we rewrite the term $A_{n,j_{1},2}$ as follows:

$$ $\displaystyle\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|=1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{i_{2}|j_{1}}^{n})\Big{[}\exp(-\|\boldsymbol{\omega}_{i_{2}|j_{1}}^{n}-\boldsymbol{x}\|)-\exp(-\|\boldsymbol{\omega}_{j_{2}|j_{1}}^{*}-\boldsymbol{x}\|)\Big{]}\exp(-\|\boldsymbol{a}_{j_{1}}^{n}-\boldsymbol{x}\|)p^{LL,n}_{j_{1}}(y|\boldsymbol{x})$ $\displaystyle+\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|>1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{i_{2}|j_{1}}^{n})\Big{[}\exp(-\|\boldsymbol{\omega}_{i_{2}|j_{1}}^{n}-\boldsymbol{x}\|)-\exp(-\|\boldsymbol{\omega}_{j_{2}|j_{1}}^{*}-\boldsymbol{x}\|)\Big{]}\exp(-\|\boldsymbol{a}_{j_{1}}^{n}-\boldsymbol{x}\|)p^{LL,n}_{j_{1}}(y|\boldsymbol{x})$ $\displaystyle:$ $\displaystyle=A_{n,j_{1},2,1}+A_{n,j_{1},2,2}.$ $$

According to the first-order Taylor expansion, we have

$$ $\displaystyle A_{n,j_{1},2,1}=\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|=1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\sum_{|\boldsymbol{\psi}|=1}\frac{\exp(\beta_{i_{2}|j_{1}}^{n})}{\boldsymbol{\psi}!}(\Delta\boldsymbol{\omega}_{i_{2}j_{2}|j_{1}}^{n})^{\boldsymbol{\psi}}$ $\displaystyle\hskip 142.26378pt\times\frac{\partial^{|\boldsymbol{\psi}|}F}{\partial\boldsymbol{\omega}^{\boldsymbol{\psi}}}(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})\exp(-\|\boldsymbol{a}_{j_{1}}^{n}-\boldsymbol{x}\|)p^{LL,n}_{j_{1}}(y|\boldsymbol{x})+R_{n,2,1}(\boldsymbol{x},y),$ $\displaystyle=\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|=1}\sum_{|\boldsymbol{\psi}|=1}T_{n,j_{2}|j_{1},\boldsymbol{\psi}}\cdot\frac{\partial^{|\boldsymbol{\psi}|}F}{\partial\boldsymbol{\omega}^{\boldsymbol{\psi}}}(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})\exp(-\|\boldsymbol{a}_{j_{1}}^{n}-\boldsymbol{x}\|)p^{LL,n}_{j_{1}}(y|\boldsymbol{x})+R_{n,2,1}(\boldsymbol{x},y),$ $$

where $R_{n,2,1}(\boldsymbol{x},y)$ is a Taylor remainder such that $R_{n,2,1}(\boldsymbol{x},y)/\mathcal{L}_{3n}\to 0$ as $n\to\infty$, and

$$ $\displaystyle T_{n,j_{2}|j_{1},\boldsymbol{\psi}}:=\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\frac{\exp(\beta_{i_{2}|j_{1}}^{n})}{\boldsymbol{\psi}!}(\Delta\boldsymbol{\omega}_{i_{2}j_{2}|j_{1}}^{n})^{\boldsymbol{\psi}},$ $$

for any $j_{2}\in[k^{*}_{2}]$ and $\boldsymbol{\psi}\neq\boldsymbol{0}_{d}$.

Meanwhile, we apply the second-order Taylor expansion to $A_{n,j_{1},2,2}$:

$$ $\displaystyle A_{n,j_{1},2,2}$ $\displaystyle=\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|>1}\sum_{|\boldsymbol{\psi}|=1}^{2}T_{n,j_{2}|j_{1},\boldsymbol{\psi}}\cdot\frac{\partial^{|\boldsymbol{\psi}|}F}{\partial\boldsymbol{\omega}^{\boldsymbol{\psi}}}(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})\exp(-\|\boldsymbol{a}_{j_{1}}^{n}-\boldsymbol{x}\|)p^{LL,n}_{j_{1}}(y|\boldsymbol{x})+R_{n,2,2}(\boldsymbol{x},y),$ $$

where $R_{n,2,2}(\boldsymbol{x},y)$ is a Taylor remainder such that $R_{n,2,2}(\boldsymbol{x},y)/\mathcal{L}_{3n}\to 0$ as $n\to\infty$.

Combine the above results together, we can illustrate the term $A_{n}$ as

$$ $\displaystyle A_{n}=\sum_{j_{1}=1}^{k^{*}_{1}}\sum_{j_{2}=1}^{k^{*}_{2}}\frac{\exp(b_{j_{1}}^{n})}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp(-\|\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}}-\boldsymbol{x}\|+\beta^{*}_{j^{\prime}_{2}|j_{1}})}\Bigg{[}\sum_{|\boldsymbol{\alpha}_{1}|+|\boldsymbol{\alpha}_{2}|+|\boldsymbol{\alpha}_{3}|=0}^{r^{LL}_{j_{2}|j_{1}}}\sum_{\rho=0\vee 1-|\boldsymbol{\alpha}_{1}|-|\boldsymbol{\alpha}_{2}|-|\boldsymbol{\alpha}_{3}|}^{2(r^{LL}_{j_{2}|j_{1}}-|\boldsymbol{\alpha}_{1}|-|\boldsymbol{\alpha}_{2}|-|\boldsymbol{\alpha}_{3}|)}S_{n,j_{2}|j_{1},\boldsymbol{\alpha}_{1},\boldsymbol{\alpha}_{2},\boldsymbol{\alpha}_{3},\rho}$ $\displaystyle\times\boldsymbol{x}^{\boldsymbol{\alpha}_{3}}\frac{\partial^{|\boldsymbol{\alpha}_{1}|}F}{\partial\boldsymbol{\omega}^{\boldsymbol{\alpha}_{1}}}(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})\frac{\partial^{|\boldsymbol{\alpha}_{2}|}F}{\partial\boldsymbol{a}^{\boldsymbol{\alpha}_{2}}}(\boldsymbol{x};\boldsymbol{a}_{j_{1}}^{*})\frac{\partial^{|\boldsymbol{\alpha}_{3}|+\rho}\pi}{\partial\xi^{|\boldsymbol{\alpha}_{3}|+\rho}}(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})+R_{n,1,1}(\boldsymbol{x},y)+R_{n,1,2}(\boldsymbol{x},y)$ $\displaystyle-\sum_{|\boldsymbol{\psi}|=0}^{2}T_{n,j_{2}|j_{1},\boldsymbol{\psi}}\cdot\frac{\partial^{|\boldsymbol{\psi}|}F}{\partial\boldsymbol{\omega}^{\boldsymbol{\psi}}}(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})\exp(-\|\boldsymbol{a}_{j_{1}}^{n}-\boldsymbol{x}\|)p^{LL,n}_{j_{1}}(y|\boldsymbol{x})-R_{n,2,1}(\boldsymbol{x},y)-R_{n,2,2}(\boldsymbol{x},y)\Bigg{]},$ (48) $$

where $S_{n,j_{2}|j_{1},\boldsymbol{\alpha}_{1},\boldsymbol{\alpha}_{2},\boldsymbol{\alpha}_{3},\rho}=T_{n,j_{2}|j_{1},\boldsymbol{\psi}}=\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{i_{2}|j_{1}}^{n})-\exp(\beta_{j_{2}|j_{1}}^{*})$ for any $j_{1}\in[k^{*}_{1}]$, $j_{2}\in[k^{*}_{2}]$, $(\boldsymbol{\alpha}_{1},\boldsymbol{\alpha}_{2},\boldsymbol{\alpha}_{3},\rho)=(\boldsymbol{0}_{d},\boldsymbol{0}_{d},\boldsymbol{0}_{d},0)$ and $\boldsymbol{\psi}=\boldsymbol{0}_{d}$.

Step 1B - Decompose $B_{n}$: By invoking the first-order Taylor expansion, we decompose the term $B_{n}$ defined in equation ([47](#A5.E47)) as

$$ $\displaystyle B_{n}$ $\displaystyle=\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\sum_{|\boldsymbol{\gamma}|=1}(\Delta\boldsymbol{a}_{j_{1}}^{n})^{\boldsymbol{\gamma}}\cdot\frac{\partial^{|\boldsymbol{\gamma}|}F}{\partial\boldsymbol{a}^{\boldsymbol{\gamma}}}(\boldsymbol{x};\boldsymbol{a}_{j_{1}}^{*})p^{LL}_{G_{n}}(y|\boldsymbol{x})+R_{n,3}(\boldsymbol{x},y)$ (49) $$

where $R_{n,3}(\boldsymbol{x},y)$ is a Taylor remainder such that $R_{n,3}(\boldsymbol{x},y)/\mathcal{L}_{3n}\to 0$ as $n\to\infty$.

Putting the decomposition in equations ([47](#A5.E47)), ([48](#A5.E48)) and ([49](#A5.E49)) together, we realize that $A_{n}$, $B_{n}$ and $C_{n}$ can be treated as a linear combination of elements from the following set union:

$$ $\displaystyle\Bigg{\{}\frac{\boldsymbol{x}^{\boldsymbol{\alpha}_{3}}\frac{\partial^{|\boldsymbol{\alpha}_{1}|}F}{\partial\boldsymbol{\omega}^{\boldsymbol{\alpha}_{1}}}(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})\frac{\partial^{|\boldsymbol{\alpha}_{2}|}F}{\partial\boldsymbol{a}^{\boldsymbol{\alpha}_{2}}}(\boldsymbol{x};\boldsymbol{a}_{j_{1}}^{*})\frac{\partial^{|\boldsymbol{\alpha}_{3}|+\rho}\pi}{\partial\xi^{|\boldsymbol{\alpha}_{3}|+\rho}}(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp(-\|\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}}-\boldsymbol{x}\|+\beta^{*}_{j^{\prime}_{2}|j_{1}})}:j_{1}\in[k^{*}_{1}],\ j_{2}\in[k^{*}_{2}],$ $\displaystyle\hskip 56.9055pt0\leq|\boldsymbol{\alpha}_{1}|+|\boldsymbol{\alpha}_{2}|+|\boldsymbol{\alpha}_{3}|\leq 2r^{LL}_{j_{2}|j_{1}},0\leq\rho\leq 2(r^{LL}_{j_{2}|j_{1}}-|\boldsymbol{\alpha}_{1}|-|\boldsymbol{\alpha}_{2}|-|\boldsymbol{\alpha}_{3}|)\Bigg{\}}$ $\displaystyle\cup~{}$ $\displaystyle\Bigg{\{}\frac{\frac{\partial^{|\boldsymbol{\psi}|}F}{\partial\boldsymbol{\omega}^{\boldsymbol{\psi}}}(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})\exp(-\|\boldsymbol{a}_{j_{1}}^{n}-\boldsymbol{x}\|)p^{LL,n}_{j_{1}}(y|\boldsymbol{x})}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp(-\|\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}}-\boldsymbol{x}\|+\beta^{*}_{j^{\prime}_{2}|j_{1}})}:j_{1}\in[k^{*}_{1}],\ j_{2}\in[k^{*}_{2}],\ 0\leq|\boldsymbol{\psi}|\leq 2\Bigg{\}}$ $\displaystyle\cup~{}$ $\displaystyle\left\{\frac{\partial^{|\boldsymbol{\gamma}|}F}{\partial\boldsymbol{a}^{\boldsymbol{\gamma}}}(\boldsymbol{x};\boldsymbol{a}_{j_{1}}^{*})p^{LL,n}_{j_{1}}(y|\boldsymbol{x}),\ \frac{\partial^{|\boldsymbol{\gamma}|}F}{\partial\boldsymbol{a}^{\boldsymbol{\gamma}}}(\boldsymbol{x};\boldsymbol{a}_{j_{1}}^{*})p^{LL}_{G_{n}}(y|\boldsymbol{x}):j_{1}\in[k^{*}_{1}],\ 0\leq|\boldsymbol{\gamma}|\leq 1\right\}.$ $$

Step 2 - Non-vanishing coefficients: In this step, we demonstrate that not all the coefficients in the representation of $A_{n}/\mathcal{L}_{3n}$, $B_{n}/\mathcal{L}_{3n}$ and $C_{n}/\mathcal{L}_{3n}$ converge to zero as $n\to\infty$. Assume by contrary that all of them go to zero. Then, we look into the coefficients associated with the term

- •
$\exp(-\|\boldsymbol{a}_{j_{1}}^{*}-\boldsymbol{x}\|)p^{LL,n}_{j_{1}}(y|\boldsymbol{x})$ in $C_{n}/\mathcal{L}_{3n}$, we have
$\displaystyle\frac{1}{\mathcal{L}_{3n}}\cdot\sum_{j_{1}=1}^{k^{*}_{1}}\Big{|}\exp(b_{j_{1}}^{n})-\exp(b_{j_{1}}^{*})\Big{|}\to 0.$
(50)
- •
$\dfrac{F(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})F(\boldsymbol{x};\boldsymbol{a}_{j_{1}}^{*})\pi(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp(-\|\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}}-\boldsymbol{x}\|+\beta^{*}_{j^{\prime}_{2}|j_{1}})}$ in $A_{n}/\mathcal{L}_{3n}$, we get that
$\displaystyle\frac{1}{\mathcal{L}_{3n}}\cdot\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\sum_{j_{2}=1}^{k^{*}_{2}}\Big{|}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{i_{2}|j_{1}}^{n})-\exp(\beta_{j_{2}|j_{1}}^{*})\Big{|}\to 0.$
(51)
- •
$\dfrac{\frac{\partial^{|\boldsymbol{\alpha}_{1}|}F}{\partial\boldsymbol{\omega}^{\boldsymbol{\alpha}_{1}}}(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})F(\boldsymbol{x};\boldsymbol{a}_{j_{1}}^{*})\pi(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp(-\|\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}}-\boldsymbol{x}\|+\beta^{*}_{j^{\prime}_{2}|j_{1}})}$ in $A_{n}/\mathcal{L}_{3n}$ for $j_{1}\in[k^{*}_{1}],j_{2}\in[k^{*}_{2}]:|\mathcal{V}_{j_{2}|j_{1}}|=1$ and $\boldsymbol{\alpha}_{1}=e_{d,u}$ where $e_{d,u}:=(0,\ldots,0,\underbrace{1}_{\textit{u-th}},0,\ldots,0)\in\mathbb{N}^{d}$, we receive that
$\displaystyle\frac{1}{\mathcal{L}_{3n}}\cdot\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\sum_{j_{2}\in[k^{*}_{2}]:|\mathcal{V}_{j_{2}|j_{1}}|=1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{i_{2}|j_{1}}^{n})\|\boldsymbol{\omega}_{i_{2}|j_{1}}^{n}-\boldsymbol{\omega}_{j_{2}|j_{1}}^{*}\|_{1}\to 0.$
Note that since the norm-1 is equivalent to the norm-2, then we can replace the norm-1 with the norm-2, that is,
$\displaystyle\frac{1}{\mathcal{L}_{3n}}\cdot\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\sum_{j_{2}\in[k^{*}_{2}]:|\mathcal{V}_{j_{2}|j_{1}}|=1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{i_{2}|j_{1}}^{n})\|\boldsymbol{\omega}_{i_{2}|j_{1}}^{n}-\boldsymbol{\omega}_{j_{2}|j_{1}}^{*}\|\to 0.$
(52)
- •
$\boldsymbol{x}^{\boldsymbol{\alpha}_{3}}\dfrac{F(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})F(\boldsymbol{x};\boldsymbol{a}_{j_{1}}^{*})\frac{\partial^{|\boldsymbol{\alpha}_{3}|}\pi}{\partial\xi^{|\boldsymbol{\alpha}_{3}|}}(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp(-\|\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}}-\boldsymbol{x}\|+\beta^{*}_{j^{\prime}_{2}|j_{1}})}$ in $A_{n}/\mathcal{L}_{3n}$ for $j_{1}\in[k^{*}_{1}],j_{2}\in[k^{*}_{2}]:|\mathcal{V}_{j_{2}|j_{1}}|=1$ and $\boldsymbol{\alpha}_{3}=e_{d,u}$, we have that
$\displaystyle\frac{1}{\mathcal{L}_{3n}}\cdot\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\sum_{j_{2}\in[k^{*}_{2}]:|\mathcal{V}_{j_{2}|j_{1}}|=1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{j_{2}|j_{1}}^{n})\|\boldsymbol{\eta}_{j_{1}i_{2}}^{n}-\boldsymbol{\eta}_{j_{1}j_{2}}^{*}\|\to 0.$
(53)
- •
$\frac{\partial^{|\boldsymbol{\gamma}|}F}{\partial\boldsymbol{a}^{\boldsymbol{\gamma}}}(\boldsymbol{x};\boldsymbol{a}_{j_{1}}^{*})p^{LL}_{G_{n}}(y|\boldsymbol{x})$ in $B_{n}/\mathcal{L}_{3n}$ for $j_{1}\in[k^{*}_{1}]$ and $\boldsymbol{\gamma}=e_{d,u}$, we obtain
$\displaystyle\frac{1}{\mathcal{L}_{3n}}\cdot\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\|\boldsymbol{a}_{j_{1}}^{n}-\boldsymbol{a}_{j_{1}}^{*}\|\to 0.$
(54)
- •
$\dfrac{\frac{\partial^{|\boldsymbol{\alpha}_{1}|}F}{\partial\boldsymbol{\omega}^{\boldsymbol{\alpha}_{1}}}(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})F(\boldsymbol{x};\boldsymbol{a}_{j_{1}}^{*})\pi(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp(-\|\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}}-\boldsymbol{x}\|+\beta^{*}_{j^{\prime}_{2}|j_{1}})}$ in $A_{n}/\mathcal{L}_{3n}$ for $j_{1}\in[k^{*}_{1}],j_{2}\in[k^{*}_{2}]:|\mathcal{V}_{j_{2}|j_{1}}|>1$ and $\boldsymbol{\alpha}_{1}=2e_{d,u}$, we receive that
$\displaystyle\frac{1}{\mathcal{L}_{3n}}\cdot\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\sum_{j_{2}\in[k^{*}_{2}]:|\mathcal{V}_{j_{2}|j_{1}}|>1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{i_{2}|j_{1}}^{n})\|\boldsymbol{\omega}_{i_{2}|j_{1}}^{n}-\boldsymbol{\omega}_{j_{2}|j_{1}}^{*}\|^{2}\to 0.$
(55)
- •
$\dfrac{\boldsymbol{x}^{\boldsymbol{\alpha}_{3}}F(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})F(\boldsymbol{x};\boldsymbol{a}_{j_{1}}^{*})\frac{\partial^{|\boldsymbol{\alpha}_{3}|}\pi}{\partial\xi^{|\boldsymbol{\alpha}_{3}|}}(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp(-\|\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}}-\boldsymbol{x}\|+\beta^{*}_{j^{\prime}_{2}|j_{1}})}$ in $A_{n}/\mathcal{L}_{3n}$ for $j_{1}\in[k^{*}_{1}],j_{2}\in[k^{*}_{2}]:|\mathcal{V}_{j_{2}|j_{1}}|>1$ and $\boldsymbol{\alpha}_{3}=2e_{d,u}$, we have that
$\displaystyle\frac{1}{\mathcal{L}_{3n}}\cdot\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\sum_{j_{2}\in[k^{*}_{2}]:|\mathcal{V}_{j_{2}|j_{1}}|>1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta_{i_{2}|j_{1}}^{n})\|\boldsymbol{\eta}_{j_{1}i_{2}}^{n}-\boldsymbol{\eta}_{j_{1}j_{2}}^{*}\|^{2}\to 0.$
(56)

Combine the above limits and the formulation of the loss $\mathcal{L}_{3n}$ in equation ([46](#A5.E46)), we deduce that

$$ $\displaystyle\frac{1}{\mathcal{L}_{3n}}\cdot\sum_{j_{1}=1}^{k^{*}_{1}}\exp(b_{j_{1}}^{n})\sum_{j_{2}:|\mathcal{V}_{j_{2}|j_{1}}|>1}\sum_{i_{2}\in\mathcal{V}_{j_{2}|j_{1}}}\exp(\beta^{n}_{i_{2}|j_{1}})\Big{(}|\Delta\tau_{j_{1}i_{2}j_{2}}^{n}|^{r^{LL}_{j_{2}|j_{1}}}+|\Delta\nu_{j_{1}i_{2}j_{2}}^{n}|^{\frac{r^{LL}_{j_{2}|j_{1}}}{2}}\Big{)}\not\to 0.$ $$

This indicates that there exist indices $j^{*}_{1}\in[k^{*}_{1}]$ and $j^{*}_{2}\in[k^{*}_{2}]:|\mathcal{V}_{j^{*}_{2}|j^{*}_{1}}|>1$ such that

$$ $\displaystyle\frac{1}{\mathcal{L}_{3n}}\cdot\sum_{i_{2}\in\mathcal{V}_{j^{*}_{2}|j^{*}_{1}}}\exp(\beta^{n}_{i_{2}|j^{*}_{1}})\Big{(}|\Delta\tau^{n}_{j^{*}_{1}i_{2}j^{*}_{2}}|^{r^{LL}_{j^{*}_{2}|j^{*}_{1}}}+|\Delta\nu^{n}_{j^{*}_{1}i_{2}j^{*}_{2}}|^{\frac{r^{LL}_{j^{*}_{2}|j^{*}_{1}}}{2}}\Big{)}\not\to 0.$ (57) $$

WLOG, we may assume that $j^{*}_{1}=j^{*}_{2}=1$. Then, considering the coefficients of the term $F(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})F(\boldsymbol{x};\boldsymbol{a}_{j_{1}}^{*})\frac{\partial^{\rho}\pi}{\partial\xi^{\rho}}(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})$ in $A_{n}/\mathcal{L}_{3n}$ where $j_{1}=j_{2}=1$, we get

$$ $\exp(b^{n}_{1})S_{n,1|1,\boldsymbol{0}_{d},\boldsymbol{0}_{d},\boldsymbol{0}_{d},\rho}/\mathcal{L}_{3n}\to 0$ $$

or equivalently,

$$ $\displaystyle\frac{1}{\mathcal{L}_{3n}}\cdot\sum_{i_{2}\in\mathcal{V}_{1|1}}\sum_{\alpha_{4}+2\alpha_{5}=\rho}\frac{\exp(\beta^{n}_{i_{2}|1})}{2^{\alpha_{5}}\alpha_{4}!\alpha_{5}!}\cdot(\Delta\tau^{n}_{1i_{2}1})^{\alpha_{4}}(\Delta\nu^{n}_{1i_{2}1})^{\alpha_{5}}\to 0.$ (58) $$

Next, we divide the left hand side of equation ([57](#A5.E57)) by that of equation ([58](#A5.E58)), and get that

$$ $\displaystyle\dfrac{\sum_{i_{2}\in\mathcal{V}_{1|1}}\sum_{\alpha_{4}+2\alpha_{5}=\rho}\frac{\exp(\beta^{n}_{i_{2}|1})}{2^{\alpha_{5}}\alpha_{4}!\alpha_{5}!}\cdot(\Delta\tau^{n}_{1i_{2}1})^{\alpha_{4}}(\Delta\nu^{n}_{1i_{2}1})^{\alpha_{5}}}{\sum_{i_{2}\in\mathcal{V}_{1|1}}\exp(\beta^{n}_{i_{2}|1})\Big{(}|\Delta\tau^{n}_{1i_{2}1}|^{r^{LL}_{1|1}}+|\Delta\nu^{n}_{1i_{2}1}|^{\frac{r^{LL}_{1|1}}{2}}\Big{)}}\to 0.$ (59) $$

Let us define $\overline{M}_{n}:=\max\{\|\Delta\tau^{n}_{1i_{2}1}\|,\|\Delta\nu^{n}_{1i_{2}1}\|^{1/2}:i_{2}\in\mathcal{V}_{1|1}\}$, and $\overline{\beta}_{n}:=\max_{i_{2}\in\mathcal{V}_{1|1}}\exp(\beta^{n}_{i_{2}|1})$. Since the sequence $\exp(\beta^{n}_{i_{2}|1})/\overline{\beta}_{n}$ is bounded, we can replace it by its subsequence which has a positive limit $p^{2}_{i_{2}}:=\lim_{n\to\infty}\exp(\beta^{n}_{i_{2}|1})/\overline{\beta}_{n}$. Note that at least one among the limits $p^{2}_{i_{2}}$ must be equal to one. Next, let us define

$$ $\displaystyle(\Delta\tau^{n}_{1i_{2}1})/\overline{M}_{n}\to q_{4i_{2}},$ $\displaystyle\quad(\Delta\nu^{n}_{1i_{2}1})/2\overline{M}_{n}\to q_{5i_{2}}.$ $$

Note that at least one among $q_{4i_{2}},q_{5i_{2}}$ must be equal to either 1 or $-1$.

By dividing both the numerator and the denominator of the term in equation ([45](#A5.E45)) by $\overline{\beta}_{n}\overline{M}^{\rho}_{n}$, we obtain the system of polynomial equations:

$$ $\displaystyle\sum_{i_{2}\in\mathcal{V}_{1|1}}\sum_{\alpha_{4}+2\alpha_{5}=\rho}\frac{1}{\alpha_{4}!\alpha_{5}!}\cdot p^{2}_{i_{2}}q_{4i_{2}}^{\alpha_{4}}q_{5i_{2}}^{\alpha_{5}}=0,\quad 1\leq\rho\leq r^{LL}_{1|1}.$ $$

According to the definition of the term $r^{LL}_{1|1}$, the above system does not have any non-trivial solutions, which is a contradiction. Consequently, at least one among the coefficients in the representation of $A_{n}/\mathcal{L}_{3n}$, $B_{n}/\mathcal{L}_{3n}$ and $C_{n}/\mathcal{L}_{3n}$ must not approach zero as $n\to\infty$.

Step 3 - Application of the Fatou’s lemma. In this stage, we show that all the coefficients in the formulations of $A_{n}/\mathcal{L}_{3n}$, $B_{n}/\mathcal{L}_{3n}$ and $C_{n}/\mathcal{L}_{3n}$ go to zero as $n\to\infty$. Denote by $m_{n}$ the maximum of the absolute values of those coefficients, the result from Step 2 induces that $1/m_{n}\not\to\infty$.

By employing the Fatou’s lemma, we have

$$ $\displaystyle 0=\lim_{n\to\infty}\dfrac{\mathbb{E}_{\boldsymbol{X}}[V(p^{LL}_{G_{n}}(\cdot|\boldsymbol{X}),p^{LL}_{G_{*}}(\cdot|\boldsymbol{X}))]}{m_{n}\mathcal{L}_{3n}}\geq\int\liminf_{n\to\infty}\dfrac{|p^{LL}_{G_{n}}(y|\boldsymbol{x})-p^{LL}_{G_{*}}(y|\boldsymbol{x})|}{2m_{n}\mathcal{L}_{3n}}\mathrm{d}(\boldsymbol{x},y).$ $$

Thus, we deduce that

$$ $\displaystyle\dfrac{|p^{LL}_{G_{n}}(y|\boldsymbol{x})-p^{LL}_{G_{*}}(y|\boldsymbol{x})|}{2m_{n}\mathcal{L}_{3n}}\to 0,$ $$

which results in $Q_{n}/[m_{n}\mathcal{L}_{3n}]\to 0$ as $n\to\infty$ for almost surely $(\boldsymbol{x},y)$.

Next, we denote

$$ $\displaystyle\frac{\exp(b_{j_{1}}^{n})S_{n,j_{2}|j_{1},\boldsymbol{\alpha}_{1},\boldsymbol{\alpha}_{2},\boldsymbol{\alpha}_{3},\rho}}{m_{n}\mathcal{L}_{3n}}\to\phi_{j_{2}|j_{1},\boldsymbol{\alpha}_{1},\boldsymbol{\alpha}_{2},\boldsymbol{\alpha}_{3},\rho},\quad$ $\displaystyle\frac{\exp(b_{j_{1}}^{n})T_{n,j_{2}|j_{1},\boldsymbol{\psi}}}{m_{n}\mathcal{L}_{3n}}\to\varphi_{j_{2}|j_{1},\boldsymbol{\psi}},$ $\displaystyle\frac{\exp(b_{j_{1}}^{n})(\Delta\boldsymbol{a}_{j_{1}}^{n})^{\boldsymbol{\gamma}}}{m_{n}\mathcal{L}_{3n}}\to\lambda_{j_{1},\boldsymbol{\gamma}},\quad$ $\displaystyle\frac{\exp(b_{j_{1}}^{n})-\exp(b_{j_{1}}^{*})}{m_{n}\mathcal{L}_{3n}}\to\chi_{j_{1}}$ $$

with a note that at least one among them is non-zero. Then, the decomposition of $Q_{n}$ in equation ([47](#A5.E47)) indicates that

$$ $\displaystyle\lim_{n\to\infty}\frac{Q_{n}}{m_{n}\mathcal{L}_{3n}}=\lim_{n\to\infty}\frac{A_{n}}{m_{n}\mathcal{L}_{3n}}-\lim_{n\to\infty}\frac{B_{n}}{m_{n}\mathcal{L}_{3n}}+\lim_{n\to\infty}\frac{C_{n}}{m_{n}\mathcal{L}_{3n}},$ $$

in which

$$ $\displaystyle\lim_{n\to\infty}\frac{A_{n}}{m_{n}\mathcal{L}_{3n}}=\sum_{j_{1}=1}^{k^{*}_{1}}\sum_{j_{2}=1}^{k^{*}_{2}}\Bigg{[}\sum_{|\boldsymbol{\alpha}|=0}^{2}\phi_{j_{2}|j_{1},\boldsymbol{\alpha}_{1},\boldsymbol{\alpha}_{2},\boldsymbol{\alpha}_{3},\rho}\cdot\boldsymbol{x}^{\boldsymbol{\alpha}_{3}}\frac{\partial^{|\boldsymbol{\alpha}_{1}|}F}{\partial\boldsymbol{\omega}^{\boldsymbol{\alpha}_{1}}}(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})\frac{\partial^{|\boldsymbol{\alpha}_{2}|}F}{\partial\boldsymbol{a}^{\boldsymbol{\alpha}_{2}}}(\boldsymbol{x};\boldsymbol{a}_{j_{1}}^{*})$ $\displaystyle\hskip 199.16928pt\times\frac{\partial^{|\boldsymbol{\alpha}_{3}|+\rho}\pi}{\partial\xi^{|\boldsymbol{\alpha}_{3}|+\rho}}(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})$ $\displaystyle-\sum_{|\boldsymbol{\psi}|=0}^{2}\varphi_{j_{2}|j_{1},\boldsymbol{\psi}}\cdot\frac{\partial^{|\boldsymbol{\psi}|}F}{\partial\boldsymbol{\omega}^{\boldsymbol{\psi}}}(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})\exp(-\|\boldsymbol{a}_{j_{1}}^{*}-\boldsymbol{x}\|)p^{LL,*}_{j_{1}}(y|\boldsymbol{x})\Bigg{]}\frac{1}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp(-\|\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}}-\boldsymbol{x}\|+\beta^{*}_{j^{\prime}_{2}|j_{1}})},$ $\displaystyle\lim_{n\to\infty}\frac{B_{n}}{m_{n}\mathcal{L}_{3n}}=\sum_{j_{1}=1}^{k^{*}_{1}}\sum_{|\gamma|=1}\lambda_{j_{1},\boldsymbol{\gamma}}\cdot\frac{\partial^{|\boldsymbol{\gamma}|}F}{\partial\boldsymbol{a}^{\boldsymbol{\gamma}}}(\boldsymbol{x};\boldsymbol{a}_{j_{1}}^{*})p^{LL}_{G_{*}}(y|\boldsymbol{x}),$ $\displaystyle\lim_{n\to\infty}\frac{C_{n}}{m_{n}\mathcal{L}_{3n}}=\sum_{j_{1}=1}^{k^{*}_{1}}\chi_{j_{1}}\exp(-\|\boldsymbol{a}_{j_{1}}^{*}-\boldsymbol{x}\|)\left[p^{LL,*}_{j_{1}}(y|\boldsymbol{x})-p^{LL}_{G_{*}}(y|\boldsymbol{x})\right].$ $$

Since the set

$$ $\displaystyle\Bigg{\{}\dfrac{\boldsymbol{x}^{\boldsymbol{\alpha}_{3}}\frac{\partial^{|\boldsymbol{\alpha}_{1}|}F}{\partial\boldsymbol{\omega}^{\boldsymbol{\alpha}_{1}}}(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})\frac{\partial^{|\boldsymbol{\alpha}_{2}|}F}{\partial\boldsymbol{a}^{\boldsymbol{\alpha}_{2}}}(\boldsymbol{x};\boldsymbol{a}_{j_{1}}^{*})\frac{\partial^{|\boldsymbol{\alpha}_{3}|+\rho}\pi}{\partial\xi^{|\boldsymbol{\alpha}_{3}|+\rho}}(y|(\boldsymbol{\eta}_{j_{1}j_{2}}^{*})^{\top}\boldsymbol{x}+\tau_{j_{1}j_{2}}^{*},\nu_{j_{1}j_{2}}^{*})}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp(-\|\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}}-\boldsymbol{x}\|+\beta^{*}_{j^{\prime}_{2}|j_{1}})}:j_{1}\in[k^{*}_{1}],$ $\displaystyle\quad j_{2}\in[k^{*}_{2}],0\leq|\boldsymbol{\alpha}_{1}|+|\boldsymbol{\alpha}_{2}|+|\boldsymbol{\alpha}_{3}|\leq r^{LL}_{j_{2}|j_{1}},0\leq\rho\leq 2(r^{LL}_{j_{2}|j_{1}}-|\boldsymbol{\alpha}_{1}|-|\boldsymbol{\alpha}_{2}|-|\boldsymbol{\alpha}_{3}|\Bigg{\}}$ $\displaystyle\cup$ $\displaystyle~{}\Bigg{\{}\dfrac{\frac{\partial^{|\boldsymbol{\psi}|}F}{\partial\boldsymbol{\omega}^{\boldsymbol{\psi}}}(\boldsymbol{x};\boldsymbol{\omega}_{j_{2}|j_{1}}^{*})\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})p^{LL,*}_{j_{1}}(y|\boldsymbol{x})}{\sum_{j^{\prime}_{2}=1}^{k^{*}_{2}}\exp(-\|\boldsymbol{\omega}^{*}_{j^{\prime}_{2}|j_{1}}-\boldsymbol{x}\|+\beta^{*}_{j^{\prime}_{2}|j_{1}})}:j_{1}\in[k^{*}_{1}],j_{2}\in[k^{*}_{2}],0\leq|\boldsymbol{\psi}|\leq 2\Bigg{\}}$ $\displaystyle\cup$ $\displaystyle~{}\Big{\{}\boldsymbol{x}^{\boldsymbol{\gamma}}\exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})p^{LL}_{G_{*}}(y|\boldsymbol{x}),\ \exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})p^{LL,*}_{j_{1}}(y|\boldsymbol{x}),\ \exp((\boldsymbol{a}_{j_{1}}^{*})^{\top}\boldsymbol{x})p^{LL}_{G_{*}}(y|\boldsymbol{x})$ $\displaystyle\hskip 227.62204pt:j_{1}\in[k^{*}_{1}],0\leq|\boldsymbol{\gamma}|\leq 2\Big{\}}$ $$

is linearly independent, we obtain that $\phi_{j_{2}|j_{1},\boldsymbol{\alpha}_{1},\boldsymbol{\alpha}_{2},\boldsymbol{\alpha}_{3},\rho}=\varphi_{j_{2}|j_{1},\boldsymbol{\psi}}=\lambda_{j_{1},\boldsymbol{\gamma}}=\chi_{j_{1}}=0$ for all $j_{1}\in[k^{*}_{1}]$, $j_{2}\in[k^{*}_{2}]$, $0\leq|\boldsymbol{\alpha}_{1}|+|\boldsymbol{\alpha}_{2}|+|\boldsymbol{\alpha}_{3}|\leq r^{LL}_{j_{2}|j_{1}}$, $0\leq\rho\leq 2(r^{LL}_{j_{2}|j_{1}}-|\boldsymbol{\alpha}_{1}|-|\boldsymbol{\alpha}_{2}|-|\boldsymbol{\alpha}_{3}|)$, $0\leq|\boldsymbol{\psi}|\leq 2$ and $0\leq|\boldsymbol{\gamma}|\leq 1$, which is a contradiction. As a consequence, we obtain the inequality in equation ([12](#A5.E12)). Hence, the proof is completed.
∎

## Appendix F Proofs for Convergence of Density Estimation

###### Proof of Theorem 1 .

To streamline the arguments for this proof, it is necessary to define some notations that will be used in the sequel. First of all, let $\mathcal{P}^{type}_{k^{*}_{1},k_{2}}(\Theta)$ stand for the set of conditional density functions w.r.t mixing measures in $\mathcal{G}_{k^{*}_{1},k_{2}}(\Theta)$ where $type\in\{SS,SL,LL\}$, that is,

$$ $\displaystyle\mathcal{P}^{type}_{k^{*}_{1},k_{2}}(\Theta):=\{p^{type}_{G}(y|\boldsymbol{x}):G\in\mathcal{G}_{k^{*}_{1},k_{2}}(\Theta)\}.$ $$

Additionally, we also define

$$ $\displaystyle\widetilde{\mathcal{P}}^{type}_{k^{*}_{1},k_{2}}(\Theta)$ $\displaystyle:=\{p^{type}_{(G+G_{*})/2}(y|\boldsymbol{x}):G\in\mathcal{G}_{k^{*}_{1},k_{2}}(\Theta)\},$ $\displaystyle\widetilde{\mathcal{P}}^{type,1/2}_{k^{*}_{1},k_{2}}(\Theta)$ $\displaystyle:=\{(p^{type}_{(G+G_{*})/2})^{1/2}(y|\boldsymbol{x}):G\in\mathcal{G}_{k^{*}_{1},k_{2}}(\Theta)\}.$ $$

Next, for each $\delta>0$, we define the $L^{2}$-ball centered around the regression function $p^{type}_{G_{*}}$ and intersected with the set $\widetilde{\mathcal{P}}^{type,1/2}_{k^{*}_{1},k_{2}}(\Theta)$ as

$$ $\displaystyle\widetilde{\mathcal{P}}^{type,1/2}_{k^{*}_{1},k_{2}}(\Theta,\delta):=\left\{p^{1/2}\in\widetilde{\mathcal{P}}^{type,1/2}_{k^{*}_{1},k_{2}}(\Theta):h(p,p^{type}_{G_{*}})\leq\delta\right\}.$ $$

Following the suggestion from Geer et. al. [], we utilize the following integral to capture the size of the above $L^{2}$-ball:

$$ $\displaystyle\mathcal{J}_{B}(\delta,\widetilde{\mathcal{P}}^{type,1/2}_{k^{*}_{1},k_{2}}(\Theta,\delta)):=\int_{\delta^{2}/2^{13}}^{\delta}H_{B}^{1/2}(t,\widetilde{\mathcal{P}}^{type,1/2}_{k^{*}_{1},k_{2}}(\Theta,t),\|\cdot\|_{L^{2}})~{}\mathrm{d}t\vee\delta,$ (60) $$

where the term $H_{B}(t,\widetilde{\mathcal{P}}^{type,1/2}_{k^{*}_{1},k_{2}}(\Theta,t),\|\cdot\|_{L^{2}})$ denotes the bracketing entropy [] of $\widetilde{\mathcal{P}}^{type,1/2}_{k^{*}_{1},k_{2}}(\Theta,t)$ under the $L^{2}$-norm, and $t\vee\delta:=\max\{t,\delta\}$.

Let us recall the statement of Theorem 7.4 in [] with adapted notations to our paper as follows:

###### Lemma 2 (Theorem 7.4, [ 64 ] ) .

Let $\Psi(\delta)\geq\mathcal{J}_{B}(\delta,\widetilde{\mathcal{P}}^{type,1/2}_{k^{*}_{1},k_{2}}(\Theta,\delta))$ be such that $\Psi(\delta)/\delta^{2}$ is a non-increasing function of $\delta$. Then, for some universal constant $c$ and for some sequence $(\delta_{n})$ such that $\sqrt{n}\delta^{2}_{n}\geq c\Psi(\delta_{n})$, the following inequality holds for all $\delta\geq\delta_{n}$:

$$ $\displaystyle\mathbb{P}\Big{(}\mathbb{E}_{\boldsymbol{X}}[h(p^{type}_{\widehat{G}^{type}_{n}}(\cdot|\boldsymbol{X}),p^{type}_{G_{*}}(\cdot|\boldsymbol{X}))]>\delta\Big{)}\leq c\exp\left(-\frac{n\delta^{2}}{c^{2}}\right).$ $$

Proof overview. Given that the expert functions are Lipschitz continuous, we begin with showing that the following bound holds for any $0<\varepsilon\leq 1/2$:

$$ $\displaystyle H_{B}(\varepsilon,\mathcal{P}^{type}_{k^{*}_{1},k_{2}}(\Theta),h)\lesssim\log(1/\varepsilon),$ (61) $$

which yields that

$$ $\displaystyle\mathcal{J}_{B}(\delta,\widetilde{\mathcal{P}}^{type,1/2}_{k^{*}_{1},k_{2}}(\Theta,\delta))$ $\displaystyle=\int_{\delta^{2}/2^{13}}^{\delta}H_{B}^{1/2}(t,\widetilde{\mathcal{P}}^{type,1/2}_{k^{*}_{1},k_{2}}(\Theta,t),\|\cdot\|_{L^{2}})~{}\mathrm{d}t\vee\delta$ $\displaystyle\leq\int_{\delta^{2}/2^{13}}^{\delta}H_{B}^{1/2}(t,\mathcal{P}^{type}_{k^{*}_{1},k_{2}}(\Theta,t),h)~{}\mathrm{d}t\vee\delta$ $\displaystyle\lesssim\int_{\delta^{2}/2^{13}}^{\delta}\log(1/t)dt\vee\delta.$ (62) $$

Let $\Psi(\delta)=\delta\cdot[\log(1/\delta)]^{1/2}$, then it can be checked that $\Psi(\delta)/\delta^{2}$ is a non-increasing function of $\delta$. Moreover, the result in equation ([F](#A6.Ex230)) implies that $\Psi(\delta)\geq\mathcal{J}_{B}(\delta,\widetilde{\mathcal{P}}^{type,1/2}_{k^{*}_{1},k_{2}}(\Theta,\delta))$. By choosing $\delta_{n}=\sqrt{\log(n)/n}$, we have that $\sqrt{n}\delta^{2}_{n}\geq c\Psi(\delta_{n})$ for some universal constant $c$. Then, the conclusion of this theorem is achieved according to Lemma [2](#Thmlemma2). Consequently, it is sufficient to derive the bracketing entropy bound in equation ([61](#A6.E61)).

Proof for the bound ([61](#A6.E61)). To begin with, we provide an upper bound for the Gaussian density function $\pi(y|\eta^{\top}\boldsymbol{x}+\tau,\nu)$. In particular, since the input space $\mathcal{X}$ and the parameter space $\Theta$ are both bounded, we can find some constant $\kappa,\ell,u>0$ such that $-\kappa\leq\eta^{\top}\boldsymbol{x}+\tau\leq\kappa$ and $\ell\leq\nu\leq u$. Then, it can be validated that

$$ $\displaystyle\pi(y|\eta^{\top}\boldsymbol{x}+\tau,\nu)=\frac{1}{\sqrt{2\pi\nu}}\exp\Big{(}-\frac{(y-(\eta^{\top}\boldsymbol{x}+\tau))^{2}}{2\nu}\Big{)}\leq\frac{1}{\sqrt{2\pi\ell}},$ $$

for any $|y|<2\kappa$. On the other hand, for $|y|\geq 2\kappa$, since $\frac{(y-(\eta^{\top}\boldsymbol{x}+\tau))^{2}}{2\nu}\geq\frac{y^{2}}{8u}$, we have that

$$ $\displaystyle\pi(y|\eta^{\top}\boldsymbol{x}+\tau,\nu)\leq\frac{1}{\sqrt{2\pi\ell}}\exp\Big{(}-\frac{y^{2}}{8u}\Big{)}.$ $$

Therefore, we deduce that $\pi(y|\eta^{\top}\boldsymbol{x}+\tau,\nu)\leq M(y|\boldsymbol{x})$, where

$$ $\displaystyle M(y|\boldsymbol{x})=\begin{cases}\frac{1}{\sqrt{2\pi\ell}}\exp\Big{(}-\frac{y^{2}}{8u}\Big{)},\quad\text{for }|y|\geq 2\kappa,\\ \frac{1}{\sqrt{2\pi\ell}},\hskip 65.44142pt\text{for }|y|<2\kappa.\end{cases}$ $$

Next, let $0<\tau\leq\varepsilon$ and $\{\pi_{1},\ldots,\pi_{N}\}$ be the $\tau$-cover under the $L^{\infty}$-norm of the set $\mathcal{P}^{type}_{k^{*}_{1},k_{2}}(\Theta)$ where $N:={N}(\tau,\mathcal{P}^{type}_{k^{*}_{1},k_{2}}(\Theta),\|\cdot\|_{L^{\infty}})$ stands for the $\tau$-covering number of the norm space $(\mathcal{P}^{type}_{k^{*}_{1},k_{2}}(\Theta),\|\cdot\|_{L^{\infty}})$. Equipped with the brackets of the form $[L_{i},U_{i}]$ where

$$ $\displaystyle L_{i}(y|\boldsymbol{x})$ $\displaystyle:=\max\{\pi_{i}(y|\boldsymbol{x})-\tau,0\},$ $\displaystyle U_{i}(y|\boldsymbol{x})$ $\displaystyle:=\max\{\pi_{i}(y|\boldsymbol{x})+\tau,M(y|\boldsymbol{x})\},$ $$

for all $i\in[N]$, we can validate that $\mathcal{P}^{type}_{k^{*}_{1},k_{2}}(\Theta)\subset\cup_{i=1}^{N}[L_{i},U_{i}]$, and $U_{i}(y|\boldsymbol{x})-L_{i}(y|\boldsymbol{x})\leq\min\{2\tau,M\}$. Those results yield that

$$ $\displaystyle\|U_{i}-L_{i}\|_{L^{1}}=\int(U_{i}(y|\boldsymbol{x})-L_{i}(y|\boldsymbol{x}))\mathrm{d}(\boldsymbol{x},y)\leq\int 2\tau\mathrm{d}(\boldsymbol{x},y)=2\tau,$ $$

From the definition of the bracketing entropy, we have

$$ $\displaystyle H_{B}(2\tau,\mathcal{P}^{type}_{k^{*}_{1},k_{2}}(\Theta),\|\cdot\|_{L^{1}})\leq\log N=\log{N}(\tau,\mathcal{P}^{type}_{k^{*}_{1},k_{2}}(\Theta),\|\cdot\|_{L^{\infty}}).$ (63) $$

Therefore, it suffices to provide an upper bound for the covering number $N$. Indeed, let us denote $\Delta:=\{(b,\boldsymbol{a})\in\mathbb{R}\times\mathbb{R}^{d}:(b,\boldsymbol{a},\beta,\boldsymbol{\omega},\tau,\boldsymbol{\eta},\nu)\in\Theta\}$ and $\Omega:=\{(\beta,\boldsymbol{\omega},\tau,\boldsymbol{\eta},\nu)\in\mathbb{R}\times\mathbb{R}^{d}\times\mathbb{R}\times\mathbb{R}^{d}\times\mathbb{R}_{+}:(b,\boldsymbol{a},\beta,\boldsymbol{\omega},\tau,\boldsymbol{\eta},\nu)\in\Theta\}$. As $\Theta$ is a compact set, so are $\Delta$ and $\Omega$. Thus, we can find $\tau$-covers $\Delta_{\tau}$ and ${\Omega}_{\tau}$ for $\Delta$ and $\Omega$, respectively. Furthermore, it can be validated that

$$ $\displaystyle|\Delta_{\tau}|\leq\mathcal{O}_{P}(\tau^{-(d+1)k^{*}_{1}}),\quad|\Omega_{\tau}|\leq\mathcal{O}_{P}(\tau^{-(2d+3)k^{*}_{1}k_{2}}).$ $$

For each mixing measure $G=\sum_{i_{1}=1}^{k^{*}_{1}}\exp(b_{i_{1}})\sum_{i_{2}=1}^{k_{2}}\exp(\beta_{i_{2}|i_{1}})\delta_{(\boldsymbol{a}_{i_{1}},\boldsymbol{\omega}_{i_{2}|i_{1}},\boldsymbol{\eta}_{i_{1}i_{2}},\tau_{i_{1}i_{2}},\nu_{i_{1}i_{2}})}\in\mathcal{G}_{k^{*}_{1},k_{2}}(\Theta)$, we consider two other mixing measures $G^{\prime}$ and $\overline{G}$ defined as

$$ $\displaystyle G^{\prime}$ $\displaystyle:=\sum_{i_{1}=1}^{k^{*}_{1}}\exp(b_{i_{1}})\sum_{i_{2}=1}^{k_{2}}\exp(\overline{\beta}_{i_{2}|i_{1}})\delta_{(\boldsymbol{a}_{i_{1}},\overline{\boldsymbol{\omega}}_{i_{2}|i_{1}},\overline{\boldsymbol{\eta}}_{i_{1}i_{2}},\overline{\tau}_{i_{1}i_{2}},\overline{\nu}_{i_{1}i_{2}})},$ $\displaystyle\overline{G}$ $\displaystyle:=\sum_{i_{1}=1}^{k^{*}_{1}}\exp(\overline{b}_{i_{1}})\sum_{i_{2}=1}^{k_{2}}\exp(\overline{\beta}_{i_{2}|i_{1}})\delta_{(\overline{\boldsymbol{a}}_{i_{1}},\overline{\boldsymbol{\omega}}_{i_{2}|i_{1}},\overline{\boldsymbol{\eta}}_{i_{1}i_{2}},\overline{\tau}_{i_{1}i_{2}},\overline{\nu}_{i_{1}i_{2}})}.$ $$

Above, $(\overline{\beta}_{i_{2}|i_{1}},\overline{\boldsymbol{\omega}}_{i_{2}|i_{1}},\overline{\boldsymbol{\eta}}_{i_{1}i_{2}},\overline{\tau}_{i_{1}i_{2}},\overline{\nu}_{i_{1}i_{2}})\in{\Omega}_{\tau}$ such that $(\overline{\beta}_{i_{2}|i_{1}},\overline{\boldsymbol{\omega}}_{i_{2}|i_{1}},\overline{\boldsymbol{\eta}}_{i_{1}i_{2}},\overline{\tau}_{i_{1}i_{2}},\overline{\nu}_{i_{1}i_{2}})$ is the closest to $(\beta_{i_{2}|i_{1}},\boldsymbol{\omega}_{i_{2}|i_{1}},\boldsymbol{\eta}_{i_{1}i_{2}},\tau_{i_{1}i_{2}},\nu_{i_{1}i_{2}})$ in that set, while $(\overline{b}_{i_{1}},\overline{\boldsymbol{a}}_{i_{1}})\in\Delta_{\tau}$ is the closest to $(b_{i_{1}},\boldsymbol{\omega}_{i})$ in that set.

Now, we begin bounding the term $\|p^{type}_{G}-p^{type}_{G^{\prime}}\|_{L^{\infty}}$. For brevity, we will consider only the case when $type=SS$, while the other two cases when $type=SL$ and $type=LL$ can be argued in a similar fashion.

When $type=SS$: Let us define

$$ $\displaystyle p^{SS}_{i_{1}}(\boldsymbol{x})$ $\displaystyle:=\sum_{i_{2}=1}^{k_{2}}\sigma((\boldsymbol{\omega}_{i_{2}|i_{1}})^{\top}\boldsymbol{x}+\beta_{i_{2}|i_{1}})\pi(y|(\boldsymbol{\eta}_{i_{1}i_{2}})^{\top}\boldsymbol{x}+\tau_{i_{1}i_{2}},\nu_{i_{1}i_{2}}),$ $\displaystyle\overline{p}^{SS}_{i_{1}}(\boldsymbol{x})$ $\displaystyle:=\sum_{i_{2}=1}^{k_{2}}\sigma((\overline{\boldsymbol{\omega}}_{i_{2}|i_{1}})^{\top}\boldsymbol{x}+\overline{\beta}_{i_{2}|i_{1}})\pi(y|(\overline{\boldsymbol{\eta}}_{i_{1}i_{2}})^{\top}\boldsymbol{x}+\overline{\tau}_{i_{1}i_{2}},\overline{\nu}_{i_{1}i_{2}}).$ $$

Then, we have

$$ $\displaystyle\|p^{SS}_{G}-p^{SS}_{G^{\prime}}\|_{L^{\infty}}$ $\displaystyle=\sum_{i_{1}=1}^{k^{*}_{1}}\sigma\left((\boldsymbol{a}_{i_{1}})^{\top}\boldsymbol{x}+b_{i_{1}}\right)\cdot\|p^{SS}_{i_{1}}-\overline{p}^{SS}_{i_{1}}\|_{L^{\infty}}\leq\sum_{i_{1}=1}^{k^{*}_{1}}\|p^{SS}_{i_{1}}-\overline{p}^{SS}_{i_{1}}\|_{L^{\infty}}.$ (64) $$

Next, we need to bound the terms $p^{SS}_{i_{1}}(\boldsymbol{x})-\overline{p}^{SS}_{i_{1}}(\boldsymbol{x})$ using the triangle inequality

$$ $\displaystyle\|p^{SS}_{i_{1}}-\overline{p}^{SS}_{i_{1}}\|_{L^{\infty}}\leq\|p^{SS}_{i_{1}}-\widetilde{p}^{SS}_{i_{1}}\|_{L^{\infty}}+\|\widetilde{p}^{SS}_{i_{1}}-\overline{p}^{SS}_{i_{1}}\|_{L^{\infty}},$ (65) $$

where we define

$$ $\displaystyle\widetilde{p}^{SS}_{i_{1}}(\boldsymbol{x}):=\sum_{i_{2}=1}^{k_{2}}\sigma((\boldsymbol{\omega}_{i_{2}|i_{1}})^{\top}\boldsymbol{x}+\beta_{i_{2}|i_{1}})\pi(y|(\overline{\boldsymbol{\eta}}_{i_{1}i_{2}})^{\top}\boldsymbol{x}+\overline{\tau}_{i_{1}i_{2}},\overline{\nu}_{i_{1}i_{2}}).$ $$

Firstly, we have

$$ $\displaystyle\|p^{SS}_{i_{1}}-\widetilde{p}^{SS}_{i_{1}}\|_{L^{\infty}}\leq\sum_{i_{2}=1}^{k_{2}}\sigma((\boldsymbol{\omega}_{i_{2}|i_{1}})^{\top}\boldsymbol{x}+\beta_{i_{2}|i_{1}})$ $\displaystyle\hskip 85.35826pt\times\|\pi(y|(\boldsymbol{\eta}_{i_{1}i_{2}})^{\top}\boldsymbol{x}+\tau_{i_{1}i_{2}},\nu_{i_{1}i_{2}})-\pi(y|(\overline{\boldsymbol{\eta}}_{i_{1}i_{2}})^{\top}\boldsymbol{x}+\overline{\tau}_{i_{1}i_{2}},\overline{\nu}_{i_{1}i_{2}})\|_{L^{\infty}}$ $\displaystyle\leq\sum_{i_{2}=1}^{k_{2}}\|\pi(y|(\boldsymbol{\eta}_{i_{1}i_{2}})^{\top}\boldsymbol{x}+\tau_{i_{1}i_{2}},\nu_{i_{1}i_{2}})-\pi(y|(\overline{\boldsymbol{\eta}}_{i_{1}i_{2}})^{\top}\boldsymbol{x}+\overline{\tau}_{i_{1}i_{2}},\overline{\nu}_{i_{1}i_{2}})\|_{L^{\infty}}$ $\displaystyle\lesssim\sum_{i_{2}=1}^{k_{2}}\Big{(}\|\boldsymbol{\eta}_{i_{1}i_{2}}-\overline{\boldsymbol{\eta}}_{i_{1}i_{2}}\|+|\tau_{i_{1}i_{2}}-\overline{\tau}_{i_{1}i_{2}}|+|\nu_{i_{1}i_{2}}-\overline{\nu}_{i_{1}i_{2}}|\Big{)}\lesssim\tau.$ (66) $$

Secondly, since $\mathcal{X}$ is a bounded set, we may assume that $\|\boldsymbol{x}\|\leq B$ for any $\boldsymbol{x}\in\mathcal{X}$. Then, it follows that

$$ $\displaystyle\|\widetilde{p}^{SS}_{i_{1}}-\overline{p}^{SS}_{i_{1}}\|_{L^{\infty}}$ $\displaystyle\leq\sum_{i_{2}=1}^{k_{2}}\Big{|}\sigma((\boldsymbol{\omega}_{i_{2}|i_{1}})^{\top}\boldsymbol{x}+\beta_{i_{2}|i_{1}})-\sigma((\overline{\boldsymbol{\omega}}_{i_{2}|i_{1}})^{\top}\boldsymbol{x}+\overline{\beta}_{i_{2}|i_{1}})\Big{|}$ $\displaystyle\hskip 113.81102pt\times\|\pi(y|(\overline{\boldsymbol{\eta}}_{i_{1}i_{2}})^{\top}\boldsymbol{x}+\overline{\tau}_{i_{1}i_{2}},\overline{\nu}_{i_{1}i_{2}})\|_{L^{\infty}}$ $\displaystyle\lesssim\sum_{i_{2}=1}^{k_{2}}\Big{[}\|\boldsymbol{\omega}_{i_{2}|i_{1}}-\overline{\boldsymbol{\omega}}_{i_{2}|i_{1}}\|\cdot|\boldsymbol{x}\|+|\beta_{i_{2}|i_{1}}-\overline{\beta}_{i_{2}|i_{1}}|\Big{]}$ $\displaystyle\leq\sum_{i_{2}=1}^{k_{2}}\Big{(}\tau B+\tau\Big{)}\lesssim\tau.$ (67) $$

From the results in equations ([64](#A6.E64)), ([65](#A6.E65)), ([F](#A6.Ex244)) and ([F](#A6.Ex247)), we deduce that

$$ $\displaystyle\|p^{SS}_{G}-p^{SS}_{G^{\prime}}\|_{L^{\infty}}\lesssim\tau.$ (68) $$

Furthermore, we have

$$ $\displaystyle\|p^{SS}_{G^{\prime}}-p^{SS}_{\overline{G}}\|_{L^{\infty}}$ $\displaystyle=\sum_{i_{1}=1}^{k^{*}_{1}}|\sigma((\boldsymbol{a}_{i_{1}})^{\top}\boldsymbol{x}+b_{i_{1}})-\sigma((\overline{\boldsymbol{a}}_{i_{1}})^{\top}\boldsymbol{x}+\overline{b}_{i_{1}})|\cdot\|\pi(y|(\overline{\boldsymbol{\eta}}_{i_{1}i_{2}})^{\top}\boldsymbol{x}+\overline{\tau}_{i_{1}i_{2}},\overline{\nu}_{i_{1}i_{2}})\|_{L^{\infty}}$ $\displaystyle\lesssim\sum_{i_{1}=1}^{k^{*}_{1}}\Big{(}\|\boldsymbol{a}_{i_{1}}-\overline{\boldsymbol{a}}_{i_{1}}\|\cdot\|\boldsymbol{x}\|+|b_{i_{1}}-\overline{b}_{i_{1}}|\Big{)}$ $\displaystyle\leq\sum_{i_{1}=1}^{k^{*}_{1}}(\tau B+\tau)\lesssim\tau.$ (69) $$

According to the triangle inequality and the results in equations ([68](#A6.E68)), ([69](#A6.E69)), we have

$$ $\displaystyle\|p^{SS}_{G}-p^{SS}_{\overline{G}}\|_{L^{\infty}}\leq\|p^{SS}_{G}-p^{SS}_{G^{\prime}}\|_{L^{\infty}}+\|p^{SS}_{G^{\prime}}-p^{SS}_{\overline{G}}\|_{L^{\infty}}\lesssim\tau.$ $$

By definition of the covering number, we deduce that

$$ $\displaystyle{N}(\tau,\mathcal{P}^{type}_{k^{*}_{1},k_{2}}(\Theta),\|\cdot\|_{L^{2}(\mu)})$ $\displaystyle\leq|\Delta_{\tau}|\times|\Omega_{\tau}|$ $\displaystyle\leq\mathcal{O}_{P}(\tau^{-(d+1)k^{*}_{1}})\times\mathcal{O}_{P}(\tau^{-(2d+3)k^{*}_{1}k_{2}})$ $\displaystyle\leq\mathcal{O}_{P}(\tau^{-(d+1)k^{*}_{1}-(2d+3)k^{*}_{1}k_{2}}).$ (70) $$

Combine the result in equation ([63](#A6.E63)) with that in ([F](#A6.Ex253)), we arrive at

$$ $\displaystyle H_{B}(2\tau,\mathcal{P}^{type}_{k^{*}_{1},k_{2}}(\Theta),\|\cdot\|_{L^{1}})\lesssim\log(1/\tau).$ $$

Let $\tau=\varepsilon/2$, then it follows that

$$ $\displaystyle H_{B}(\varepsilon,\mathcal{P}^{type}_{k^{*}_{1},k_{2}}(\Theta),\|.\|_{L^{1}})\lesssim\log(1/\varepsilon).$ $$

Finally, due to the inequality between the Hellinger distance and the $L^{1}$-norm $h\leq\|\cdot\|_{L^{1}}$, we achieve the conclusion that

$$ $\displaystyle H_{B}(\varepsilon,\mathcal{P}^{type}_{k^{*}_{1},k_{2}}(\Theta),h)\lesssim\log(1/\varepsilon).$ $$

Hence, the proof is completed.
∎

###### Proof of Theorem 1 .

###### Lemma 2 (Theorem 7.4, [ 64 ] ) .

## Appendix G Proof of Lemma 1

Firstly, let us recall the system of polynomial equations given in equation ([4](#S2.E4)):

$$ $\displaystyle\sum_{i_{2}=1}^{m}\sum_{\alpha\in\mathcal{I}^{SS}_{\boldsymbol{\rho}_{1},\rho_{2}}}\dfrac{p_{i_{2}}^{2}~{}\boldsymbol{q}_{1i_{2}}^{\boldsymbol{\alpha}_{1}}~{}\boldsymbol{q}_{2i_{2}}^{\boldsymbol{\alpha}_{2}}~{}\boldsymbol{q}_{3i_{2}}^{\boldsymbol{\alpha}_{3}}~{}q_{4i_{2}}^{\alpha_{4}}~{}q_{5i_{2}}^{\alpha_{5}}}{\boldsymbol{\alpha}_{1}!~{}\boldsymbol{\alpha}_{2}!~{}\boldsymbol{\alpha}_{3}!~{}\alpha_{4}!\alpha_{5}!}=0,\quad 1\leq|\boldsymbol{\rho}_{1}|+\rho_{2}\leq r,$ (71) $$

where $\mathcal{I}^{SS}_{\boldsymbol{\rho}_{1},\rho_{2}}=\{\alpha=(\boldsymbol{\alpha}_{1},\boldsymbol{\alpha}_{2},\boldsymbol{\alpha}_{3},\alpha_{4},\alpha_{5})\in\mathbb{N}^{d}\times\mathbb{N}^{d}\times\mathbb{N}^{d}\times\mathbb{N}\times\mathbb{N}:\ \boldsymbol{\alpha}_{1}+\boldsymbol{\alpha}_{2}+\boldsymbol{\alpha}_{3}=\boldsymbol{\rho}_{1},\ \alpha_{4}+2\alpha_{5}=\rho_{2}-|\boldsymbol{\alpha}_{3}|\}$.

When $m=2$: By observing a portion of the above system when $\boldsymbol{\rho}_{1}=\mathbf{0}_{d}$, which is given by

$$ $\displaystyle\sum_{i_{2}=1}^{m}\sum_{\alpha_{4}+2\alpha_{5}=\rho_{2}}\dfrac{p_{i_{2}}^{2}~{}q_{4i_{2}}^{\alpha_{4}}~{}q_{5i_{2}}^{\alpha_{5}}}{\alpha_{4}!~{}\alpha_{5}!}=0,\quad\rho_{2}=1,2,\ldots,r.$ (72) $$

Proposition 2.1 in [] shows that the smallest $r\in\mathbb{N}$ such that the system ([72](#A7.E72)) does not admit any non-trivial solutions when $m=2$ is $r=4$. Note that a solution of the system [72](#A7.E72) is called non-trivial in [] if all the values of $p_{i_{2}}$ are different from zero, whereas at least one among $q_{4i_{2}}$ is non-zero. This definition of non-trivial solutions totally aligns with ours for the system ([71](#A7.E71)). Therefore, we have $\bar{r}(m)\leq 4$, and it suffices to prove that $\bar{r}(m)>3$.

Indeed, when $r=3$, we demonstrate that the system ([71](#A7.E71)) admits a non-trivial solution: $p_{i_{2}}=1$, $\boldsymbol{q}_{1i_{2}}=\boldsymbol{q}_{2i_{2}}=\boldsymbol{q}_{3i_{2}}=\boldsymbol{0}_{d}$ for all $i_{2}\in[m]$, $q_{41}=1$, $q_{42}=-1$, $q_{51}=q_{52}=-\frac{1}{2}$. Since $\boldsymbol{q}_{1i_{2}}=\boldsymbol{q}_{2i_{2}}=\boldsymbol{q}_{3i_{2}}=\boldsymbol{0}_{d}$, this solution clearly satisfies the equations associated with $\boldsymbol{\rho}_{1}\neq\boldsymbol{0}_{d}$. Thus, we only need to verify those with $\boldsymbol{\rho}_{1}=\boldsymbol{0}_{d}$, which are given by

$$ $\displaystyle\sum_{j=1}^{m}p_{i_{2}}^{2}q_{4i_{2}}$ $\displaystyle=0,$ $\displaystyle\sum_{i_{2}=1}^{m}p_{i_{2}}^{2}\Big{(}\frac{1}{2}q_{4i_{2}}^{2}+q_{5i_{2}}\Big{)}$ $\displaystyle=0,$ $\displaystyle\sum_{i_{2}=1}^{m}p_{i_{2}}^{2}\Big{(}\frac{1}{3!}q_{4i_{2}}^{3}+q_{4i_{2}}q_{5i_{2}}\Big{)}$ $\displaystyle=0.$ $$

By simple calculations, we can check that $p_{i_{2}}=1$, $q_{41}=1$, $q_{42}=-1$, $q_{51}=q_{52}=-\frac{1}{2}$ satisfies the above equations. Hence, we obtain that $\bar{r}(m)>3$, leading to $\bar{r}(m)=4$.

When $m=3$: Note that $\bar{r}(m)$ is a monotonically increasing function of $m$. Therefore, it follows from the previous result that $\bar{r}(m)>\bar{r}(2)=4$, or equivalently, $\bar{r}(m)\geq 5$. Additionally, according to Proposition 2.1 in [], we deduce that $\bar{r}(m)\leq 6$ based on the reduced system in equation ([72](#A7.E72)). Thus, we only need to show that $\bar{r}(m)>5$.

Indeed, we show that the following is a non-trivial solution of the system ([71](#A7.E71)) when $r=5$:

$$ $\displaystyle p_{i_{2}}=1,\quad\boldsymbol{q}_{1i_{2}}=\boldsymbol{q}_{2i_{2}}=\boldsymbol{q}_{3i_{2}}=\boldsymbol{0}_{d},\quad\forall i_{2}\in[m],$ $\displaystyle q_{41}=\frac{\sqrt{3}}{3},\quad q_{42}=-\frac{\sqrt{3}}{3},\quad q_{43}=0,$ $\displaystyle q_{51}=q_{52}=-\frac{1}{6},\quad q_{53}=0.$ $$

Since $\boldsymbol{q}_{1i_{2}}=\boldsymbol{q}_{2i_{2}}=\boldsymbol{q}_{3i_{2}}=\boldsymbol{0}_{d}$, this solution clearly satisfies the equations associated with $\boldsymbol{\rho}_{1}\neq\boldsymbol{0}_{d}$. Thus, we only need to verify those with $\boldsymbol{\rho}_{1}=\boldsymbol{0}_{d}$, which are given by

$$ $\displaystyle\sum_{j=1}^{m}p_{i_{2}}^{2}q_{4i_{2}}$ $\displaystyle=0,$ $\displaystyle\sum_{i_{2}=1}^{m}p_{i_{2}}^{2}\Big{(}\frac{1}{2}q_{4i_{2}}^{2}+q_{5i_{2}}\Big{)}$ $\displaystyle=0,$ $\displaystyle\sum_{i_{2}=1}^{m}p_{i_{2}}^{2}\Big{(}\frac{1}{3!}q_{4i_{2}}^{3}+q_{4i_{2}}q_{5i_{2}}\Big{)}$ $\displaystyle=0,$ $\displaystyle\sum_{i_{2}=1}^{m}p_{i_{2}}^{2}\Big{(}\frac{1}{4!}q_{4i_{2}}^{4}+\frac{1}{2!}q_{4i_{2}}^{2}q_{5i_{2}}+\frac{1}{2!}q_{5i_{2}}^{2}\Big{)}$ $\displaystyle=0,$ $\displaystyle\sum_{i_{2}=1}^{m}p_{i_{2}}^{2}\Big{(}\frac{1}{5!}q_{4i_{2}}^{5}+\frac{1}{3!}q_{4i_{2}}^{3}q_{5i_{2}}+\frac{1}{2!}q_{4i_{2}}q_{5i_{2}}^{2}\Big{)}$ $\displaystyle=0.$ $$

By simple calculations, it can be validated that $p_{i_{2}}=1$, $q_{41}=\frac{\sqrt{3}}{3}$, $q_{42}=-\frac{\sqrt{3}}{3}$, $q_{43}=0$, $q_{51}=q_{52}=-\frac{1}{6}$, $q_{53}=0$ satisfies the above equations. Hence, we conclude $\bar{r}(m)>5$, meaning that $\bar{r}(m)=6$.

## Appendix H Identifiability of the Gaussian HMoE

###### Proposition 1 .

For each $type\in\{SS,SL,LL\}$, suppose that the equation $p^{type}_{G}(y|\boldsymbol{x})=p^{type}_{G_{*}}(y|\boldsymbol{x})$ holds true for almost surely $(\boldsymbol{x},y)$, then we get that $G\equiv G_{*}$.

###### Proof of Proposition 1 .

In this proof, we will consider only the case when $type=SS$ as other cases can be done similarly.

To start with, let us write the equation $p^{SS}_{G}(y|\boldsymbol{x})=p^{SS}_{G_{*}}(y|\boldsymbol{x})$ explicitly as follows:

$$ $\displaystyle\sum_{i_{1}=1}^{k^{*}_{1}}\sigma\Big{(}(\boldsymbol{a}_{i_{1}})^{\top}\boldsymbol{x}+b_{i_{1}}\Big{)}\sum_{i_{2}=1}^{k_{2}}\sigma\Big{(}(\boldsymbol{\omega}_{i_{2}|i_{1}})^{\top}\boldsymbol{x}+\beta_{i_{2}|i_{1}}\Big{)}\pi(y|(\boldsymbol{\eta}_{i_{1}i_{2}})^{\top}\boldsymbol{x}+\tau_{i_{1}i_{2}},\nu_{i_{1}i_{2}})$ $\displaystyle=\sum_{i_{1}=1}^{k^{*}_{1}}\sigma\Big{(}(\boldsymbol{a}^{*}_{i_{1}})^{\top}\boldsymbol{x}+b^{*}_{i_{1}}\Big{)}\sum_{i_{2}=1}^{k^{*}_{2}}\sigma\Big{(}(\boldsymbol{\omega}^{*}_{i_{2}|i_{1}})^{\top}\boldsymbol{x}+\beta^{*}_{i_{2}|i_{1}}\Big{)}\pi(y|(\boldsymbol{\eta}^{*}_{i_{1}i_{2}})^{\top}\boldsymbol{x}+\tau^{*}_{i_{1}i_{2}},\nu^{*}_{i_{1}i_{2}}).$ (73) $$

Then, it follows from the identifiability of the location-scale Gaussian mixtures [, ] that the number of components and the weight set of the mixing measure $G$ equal to those of its counterpart $G_{*}$, i.e. $k_{2}=k_{2}^{*}$ and

$$ $\displaystyle\Bigg{\{}\sigma\Big{(}(\boldsymbol{a}_{i_{1}})^{\top}\boldsymbol{x}+b_{i_{1}}\Big{)}\cdot\sigma\Big{(}(\boldsymbol{\omega}_{i_{2}|i_{1}})^{\top}\boldsymbol{x}+\beta_{i_{2}|i_{1}}\Big{)}:i_{1}\in[k^{*}_{1}],i_{2}\in[k^{*}_{2}]\Bigg{\}}$ $\displaystyle=\Bigg{\{}\sigma\Big{(}(\boldsymbol{a}^{*}_{i_{1}})^{\top}\boldsymbol{x}+b^{*}_{i_{1}}\Big{)}\cdot\sigma\Big{(}(\boldsymbol{\omega}^{*}_{i_{2}|i_{1}})^{\top}\boldsymbol{x}+\beta^{*}_{i_{2}|i_{1}}\Big{)}:i_{1}\in[k^{*}_{1}],i_{2}\in[k^{*}_{2}]\Bigg{\}},$ $$

for almost every $\boldsymbol{x}$. WLOG, we may assume that

$$ $\displaystyle\sigma\Big{(}(\boldsymbol{a}_{i_{1}})^{\top}\boldsymbol{x}+b_{i_{1}}\Big{)}\cdot\sigma\Big{(}(\boldsymbol{\omega}_{i_{2}|i_{1}})^{\top}\boldsymbol{x}+\beta_{i_{2}|i_{1}}\Big{)}=\sigma\Big{(}(\boldsymbol{a}^{*}_{i_{1}})^{\top}\boldsymbol{x}+b^{*}_{i_{1}}\Big{)}\cdot\sigma\Big{(}(\boldsymbol{\omega}^{*}_{i_{2}|i_{1}})^{\top}\boldsymbol{x}+\beta^{*}_{i_{2}|i_{1}}\Big{)},$ (74) $$

for almost every $\boldsymbol{x}$, for any $i_{1}\in[k^{*}_{1}],i_{2}\in[k^{*}_{2}]$.
Due to the assumptions that $\boldsymbol{\omega}_{k^{*}_{2}|i_{1}}=\boldsymbol{\omega}^{*}_{k^{*}_{2}|i_{1}}=\mathbf{0}_{d}$ and $\beta_{k^{*}_{2}|i_{1}}=\beta^{*}_{k^{*}_{2}|i_{1}}=0$, we have that

$$ $\displaystyle\sigma\Big{(}(\boldsymbol{a}_{i_{1}})^{\top}\boldsymbol{x}+b_{i_{1}}\Big{)}=\sigma\Big{(}(\boldsymbol{a}^{*}_{i_{1}})^{\top}\boldsymbol{x}+b^{*}_{i_{1}}\Big{)},$ (75) $$

for almost every $\boldsymbol{x}$, for any $i_{1}\in$. Since the $\sigma$ function is invariant to translations, then it follows from the equation ([75](#A8.E75)) that

$$ $\displaystyle\boldsymbol{a}_{i_{1}}$ $\displaystyle=\boldsymbol{a}^{*}_{i_{1}}+\boldsymbol{a}$ $\displaystyle b_{i_{1}}$ $\displaystyle=b^{*}_{i_{1}}+b,$ $$

for some $\boldsymbol{a}\in\mathbb{R}^{d}$ and $b\in\mathbb{R}$. Moreover, due to the assumption that $\boldsymbol{a}_{k^{*}_{1}}=\boldsymbol{a}^{*}_{k^{*}_{1}}$ and $b_{k^{*}_{1}}=b^{*}_{k^{*}_{1}}=0$, we get $\boldsymbol{a}=\boldsymbol{0}_{d}$ and $b=0$. This leads to $\boldsymbol{a}_{i_{1}}=\boldsymbol{a}^{*}_{i_{1}}$ and $b_{i_{1}}=b^{*}_{i_{1}}$ for any $i_{1}\in[k^{*}_{1}]$. Those results together with equation ([74](#A8.E74)) yield that

$$ $\displaystyle\sigma\Big{(}(\boldsymbol{\omega}_{i_{2}|i_{1}})^{\top}\boldsymbol{x}+\beta_{i_{2}|i_{1}}\Big{)}=\sigma\Big{(}(\boldsymbol{\omega}^{*}_{i_{2}|i_{1}})^{\top}\boldsymbol{x}+\beta^{*}_{i_{2}|i_{1}}\Big{)},$ $$

for almost every $\boldsymbol{x}$, for any $i_{1}\in[k^{*}_{1}],i_{2}\in[k^{*}_{2}]$. By employing the previous arguments, we also obtain that

$$ $\displaystyle\boldsymbol{\omega}_{i_{2}|i_{1}}$ $\displaystyle=\boldsymbol{\omega}^{*}_{i_{2}|i_{1}},$ $\displaystyle\beta_{i_{2}|i_{1}}$ $\displaystyle=\beta^{*}_{i_{2}|i_{1}}.$ $$

Then, the equation ([H](#A8.Ex269)) can be rewritten as

$$ $\displaystyle\sum_{i_{1}=1}^{k^{*}_{1}}\exp(b_{i_{1}})\sum_{i_{2}=1}^{k^{*}_{2}}\exp(\beta_{i_{2}|i_{1}})\exp\Big{(}(\boldsymbol{a}_{i_{1}}+\boldsymbol{\omega}_{i_{2}|i_{1}})^{\top}\boldsymbol{x}\Big{)}\pi(y|(\boldsymbol{\eta}_{i_{1}i_{2}})^{\top}\boldsymbol{x}+\tau_{i_{1}i_{2}},\nu_{i_{1}i_{2}})$ $\displaystyle=\sum_{i_{1}=1}^{k^{*}_{1}}\exp(b^{*}_{i_{1}})\sum_{i_{2}=1}^{k^{*}_{2}}\exp(c^{*}_{i_{2}|i_{1}})\exp\Big{(}(\boldsymbol{a}^{*}_{i_{1}}+\boldsymbol{\omega}^{*}_{i_{2}|i_{1}})^{\top}\boldsymbol{x}\Big{)}\pi(y|(\boldsymbol{\eta}^{*}_{i_{1}i_{2}})^{\top}\boldsymbol{x}+\tau^{*}_{i_{1}i_{2}},\nu^{*}_{i_{1}i_{2}}).$ (76) $$

for almost every $\boldsymbol{x}\in\mathcal{X}$.

Next, we denote $P_{1},P_{2},\ldots,P_{m_{1}}$ as a partition of the index set $[k^{*}_{1}]$, where $m_{1}\leq k^{*}_{1}$, such that $\exp(b_{i_{1}})=\exp(b^{*}_{i^{\prime}_{1}})$ for any $i_{1},i^{\prime}_{1}\in P_{j}$ and $j_{1}\in[m_{1}]$. On the other hand, when $i_{1}$ and $i^{\prime}_{1}$ do not belong to the same set $P_{j_{1}}$, we let $\exp(b_{i_{1}})\neq\exp(b^{*}_{i^{\prime}_{1}})$.

Similarly, for each $i_{1}\in[k^{*}_{1}]$, we also define $Q_{1|i_{1}},Q_{2|i_{1}},\ldots,Q_{m_{2}|i_{1}}$ as a partition of the index set $[k^{*}_{2}]$, where $m_{2}\leq k^{*}_{2}$, such that $\exp(\beta_{i_{2}|i_{1}})=\exp(\beta^{*}_{i^{\prime}_{2}|i_{1}})$ for any $i_{2},i^{\prime}_{2}\in Q_{j_{2}|i_{1}}$ and $j_{2}\in[m_{2}]$. Conversely, when $i_{2}$ and $i^{\prime}_{2}$ do not belong to the same set $Q_{j_{2}|i_{1}}$, we let $\exp(\beta_{i_{2}|i_{1}})\neq\exp(\beta^{*}_{i^{\prime}_{2}|i_{1}})$.

Thus, we can represent equation ([H](#A8.Ex277)) as

$$ $\displaystyle\sum_{j_{1}=1}^{m_{1}}\sum_{i_{1}\in{P}_{j_{1}}}\exp(b_{i_{1}})\sum_{j_{2}=1}^{m_{2}}\sum_{i_{1}\in{Q}_{j_{2}|i_{1}}}\exp(\beta_{i_{2}|i_{1}})\exp\Big{(}(\boldsymbol{a}_{i_{1}}+\boldsymbol{\omega}_{i_{2}|i_{1}})^{\top}\boldsymbol{x}\Big{)}\pi(y|(\boldsymbol{\eta}_{i_{1}i_{2}})^{\top}\boldsymbol{x}+\tau_{i_{1}i_{2}},\nu_{i_{1}i_{2}})$ $\displaystyle=\sum_{j_{1}=1}^{m_{1}}\sum_{i_{1}\in{P}_{j_{1}}}\exp(b^{*}_{i_{1}})\sum_{j_{2}=1}^{m_{2}}\sum_{i_{1}\in{Q}_{j_{2}|i_{1}}}\exp(\beta^{*}_{i_{2}|i_{1}})\exp\Big{(}(\boldsymbol{a}^{*}_{i_{1}}+\boldsymbol{\omega}^{*}_{i_{2}|i_{1}})^{\top}\boldsymbol{x}\Big{)}\pi(y|(\boldsymbol{\eta}^{*}_{i_{1}i_{2}})^{\top}\boldsymbol{x}+\tau^{*}_{i_{1}i_{2}},\nu^{*}_{i_{1}i_{2}}),$ $$

for almost every $\boldsymbol{x}\in\mathcal{X}$. Recall that we have $b_{i_{1}}=b^{*}_{i_{1}}$, $\boldsymbol{a}_{i_{1}}=\boldsymbol{a}^{*}_{i_{1}}$, $\boldsymbol{\omega}_{i_{2}|i_{1}}=\boldsymbol{\omega}^{*}_{i_{2}|i_{1}}$ and $\beta_{i_{2}|i_{1}}=\beta^{*}_{i_{2}|i_{1}}$, for any $i_{1}\in[k^{*}_{1}]$ and $i_{2}\in[k^{*}_{2}]$, then the above result leads to

$$ $\displaystyle\Big{\{}\Big{(}(\boldsymbol{\eta}_{i_{1}i_{2}})^{\top}\boldsymbol{x}+\tau_{i_{1}i_{2}},\nu_{i_{1}i_{2}}\Big{)}$ $\displaystyle:i_{1}\in P_{j_{1}},i_{2}\in Q_{j_{2}|i_{1}}\Big{\}}$ $\displaystyle\equiv\Big{\{}\Big{(}(\boldsymbol{\eta}^{*}_{i_{1}i_{2}})^{\top}\boldsymbol{x}+\tau^{*}_{i_{1}i_{2}},\nu^{*}_{i_{1}i_{2}}\Big{)}:i_{1}\in P_{j_{1}},i_{2}\in Q_{j_{2}|i_{1}}\Big{\}},$ $$

for any $j_{1}\in[m_{1}]$ and $j_{2}\in[m_{2}]$. Consequently, we obtain that

$$ $\displaystyle G$ $\displaystyle=\sum_{j_{1}=1}^{m_{1}}\sum_{i_{1}\in{P}_{j_{1}}}\exp(b_{i_{1}})\sum_{j_{2}=1}^{m_{2}}\sum_{i_{1}\in{Q}_{j_{2}|i_{1}}}\exp(\beta_{i_{2}|i_{1}})\delta_{(\boldsymbol{a}_{i_{1}},\boldsymbol{\omega}_{i_{2}|i_{1}},\boldsymbol{\eta}_{i_{1}i_{2}},\tau_{i_{1}i_{2}},\nu_{i_{1}i_{2}})}$ $\displaystyle=\sum_{j_{1}=1}^{m_{1}}\sum_{i_{1}\in{P}_{j_{1}}}\exp(b^{*}_{i_{1}})\sum_{j_{2}=1}^{m_{2}}\sum_{i_{1}\in{Q}_{j_{2}|i_{1}}}\exp(\beta^{*}_{i_{2}|i_{1}})\delta_{\boldsymbol{a}^{*}_{i_{1}},\boldsymbol{\omega}^{*}_{i_{2}|i_{1}},\boldsymbol{\eta}^{*}_{i_{1}i_{2}},\tau^{*}_{i_{1}i_{2}},\nu^{*}_{i_{1}i_{2}})}$ $\displaystyle\equiv G_{*}.$ $$

Hence, the proof is totally completed.
∎

###### Proposition 1 .

###### Proof of Proposition 1 .

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 3
presentation: 2
questions: * Line 130: I'm not sure what the citation refers to or if it is even needed ? Maximum Likelihood Estimation has been used before 2000.
soundness: 2
strengths: * Thorough analysis of the two-layers Gaussian Mixture HMoE
* Considers two different types of data (clinical data and images)
summary: The paper investigates hierarchical mixture of experts (HMoE) and in particular the type of routing mechanism used in 2-layers HMoE (Laplace vs Softmax). They first derive an analysis to show that using Laplace gating (in both layers) instead of softmax leads to faster expert specialisation. They then verify this insights by comparing HMoE with different routing strategy on experiments involving complex-structured clinical data and the CIFAR-10/ImageNet classification benchmarks.
weaknesses: * The main weakness of the paper is the writing being very unclear. I'll list some examples below:
    * Jumbled transitions between sections. Example in line 126, where we suddenly talk about the "convergence of expert specialisation"(which is only again mentioned line 260) before transitioning to an entirely different topic. Another example is line 272 "moving from this section" while it is not clear at all what we (the reader) should take away from this section.

    * The mathematical notations are cumbersome and only briefly introduced
      * some notations could be omitted for the sake of readability (e.g. the $\ast$ notations in Equation 1)
      * Some notations could be simplified. For instance introducing random variables as $X$ and $Y$ in lines 108 but then only referring to their density $x$ and $y$
      * Overall, I find that the methods section could be shortened by (i) moving some descriptions in appendix, (ii) only underlying the key take-away and (iii) making notations more lightweight.
 

    * Some statements/motivation are not substantiated:
       * line 71: "there is a growing demand for models that can deliver accurate and individualized predictions for each subgroup. Therefore, it is worthwhile to study. HMoE, which can leverage the intrinsic information within complex input structures and achieve superior performance on corresponding tasks." -> I do not see.a good reason why HMoE are a better solution to that problem than e.g. increasing the number of experts in MoE, or a deeper structure, or any other techniques to make the decision space "more complex"

  * Limitations in experimental setting:
    * Some details are missing: such as how many experts are used, or even better, considering different number of experts. For the image classification benchmarks, it's very unclear how a "one-layer MoE module" are placed in the sparse MoE backbone: e.g. what is being fine-tuned ?
    * Since the propose HMoE has a deeper routing mechanism, the paper should also discuss efficiency of the method vs MoE when integrating the new routers.
    * I find the description of the experimental section in the abstract/intro quite misleading. First most of the experiments are on clinical data, which is different from *diverse real-world scenarios* (line 19 + 535). As for image classification, CIFAR or tiny-ImageNet are interesting benchmarks, but a bit far from real world tasks.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: * Is soft or hard (sparse) assignment used at each level?  Since there are only 4 experts in each group and two high-level experts, top-k would need to be top-1 or 2 if there is sparse assignment.  Was this the case or did this use dense mixing?  And if sparse, how many experts were selected at each level?

* Multimodal Routing Distributions section:  It would be interesting to see the distributions for all blocks, not just the last one.  In particular, is there more separation between domains assigned to each expert in earlier blocks?  Layering can also have a domain factorization effect (eg in https://arxiv.org/abs/1312.4314).

* It also would be interesting to see the Softmax-Softmax models here, as well --- is there a difference in the qualitative behavior in these assignments using Laplace vs Softmax?

* In addition to a breakdown of expert assignment domain, it would be interesting to see the assignments according to the different patient latent domains described in comparison with SLDG at l.419.  Were any of these learned to be separated in the model automatically?

* l.528 "increasing the number of experts has a positive impact" in Fig 4(b)(c):  This looks to be the case for the number of inner experts, but I don't see this in the plot for the number of outer experts, which is flat.  But maybe setting this down to 1 would show a difference at least between 1 and 2?

* I don't see a need for the "mixing measure" $G_*$.  Why not just use best-fit parameter set $\hat \theta$, and say $\hat \theta = \arg\max_{\theta\in\Theta} \frac1n \sum_{i=1}^n \log(p_\theta(Y_i|X_i))$ ?

* I don't understand how the shapes in Eq 3 align.  In the second equation du/da = du/dw, a and w are two different shapes, a is [d k1] with k1 the number of first-level experts, w is [k1 d k2] with k2 the number of second-level experts.

* Eq 5 is very long, and could use an explanation of what it is (what it measures) beyond the fact that it is loss based on the cells.  For example, something like "loss function that measures the total distance from the closest true parameter, of each parameter in the parameter set" (if indeed that is what it is, I'm basing that description on just a reading of the first term, so I'm not sure).  Also, what if the cell for a true param index is empty?
soundness: 2
strengths: The theoretical analysis is very detailed, with some interesting results differing from single-gating MoE.

The use of HMoE appears effective at automatically routing multimodal data automatically among experts, which has the potential to make best use of computational resources compared to fixed subsystems for each mode.

The model obtains good results compared to other systems on the multimodal clinical datasets, and shows interesting behavior in the hierarchical assignments related to the modes in this setting.
summary: This paper studies Hierarchical Mixture of Experts (HMoE).  First, theoretical convergence rates for HMoE using both softmax and lapalce gating functions are found in the case of hierarchical MOG-distributed data, relating number of samples drawn to the discrepancy between MLE parameters and the original true distribution parameters, finding Laplace-Laplace gating to converge faster than Softmax when the number of experts exceeds the true number of mixture components.  HMoE is then applied in practice to multimodal health data estimation problems using MIMIC-IV and eICU datasets, achieving performance similar to current SOA that uses explicit patient domain grouping, but without explicit partitions.  Some of the behavior of the model is illustrated by showing the breakdown of expert assignment according to input modality.
weaknesses: While the theoretical work is very detailed, and has an interesting end result, I wasn't able to follow the arguments in detail, though I am not an expert in this sort of convergence theory.  To me, the main text of the theory section is a little too verbose and overly technical, especially since there is already a large appendix with details.  I've listed a number of questions below, but in particular:
  * I didn't see a clear summary sentence of what the results mean, along the lines of "This shows that n samples are required in order for the best MLE params to be within $O(n^{-1/2})$ of the true distribution's parameters, up to log factors".
  * Some of the equations seem unnecessary for a high-level understanding, e.g. I don't see a need to list the entire vornonoi loss function L in the main text, and also don't see a need for the "mixing measure" $G_*$.  The notation is also pretty busy, with lots of subscripts and hats/tildes/etc.  This makes it more difficult to read, and would be good to simplify as well.

For the experimental application, I didn't see an estimate of model size or computation ops in any of the comparisons, so it's hard to know if the model performed better or is simply larger than the baselines.  In particular, comparing against non-hierarchical MoE, all levels of gating should be included in op and param counts.

I also didn't see a section that corroborates the theoretical convergence rates experimentally on toy gaussian data used in the theorems.  Seeing measurements that converge at the described rates would make the theory more convincing and relatable to a wider audience.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 1
questions: Please see weakness section.
soundness: 3
strengths: - This paper firstly proposes HMoE models by incorporating alternative gating mechanisms, such as Laplace gating, which enable more flexible and efficient expert routing. This technical novelty allows HMoE to handle complex, multi-domain inputs with greater adaptability than traditional MoEs.

- This paper provides a thorough theoretical analysis, showing that varied gating functions, espically, Laplace gating functions can improve expert convergence rates. They establish bounds on parameter estimation, demonstrating that specific gating choices accelerate specialization, which is crucial for managing complex data.

- The experiments span diverse, realistic tasks, including medical prediction using datasets like MIMIC-IV, confirming HMoE’s real-world effectiveness, and show consistent gains over strong baselines, validating the model’s utility in high-dimensional, multi-domain settings.
summary: This paper investigates hierarchical mixture of experts (HMoE) models, focusing on enhancing gating mechanisms beyond the conventional softmax function. The authors propose that varied gating functions, especially Laplace gating, improve HMoE's performance across different hierarchical levels by optimizing expert specialization and convergence rates. They conduct a theoretical analysis and show that Laplace gating functions accelerate expert convergence in comparison to softmax and a combination of softmax and Laplace gatings. Empirical results on multimodal and high-dimensional datasets (e.g., image classification, clinical data) demonstrate that HMoE with tailored gating configurations outperforms baseline mixture-of-expert models, particularly in scenarios requiring domain specialization and complex input handling.
weaknesses: - While the paper focuses on enhancing HMoE’s performance with alternative gating mechanisms, it lacks a detailed discussion on the associated computational complexity. A deeper analysis of the model’s resource demands and potential methods to mitigate these challenges would add valuable context, particularly for large-scale or resource-constrained applications.

---

confidence: 3
contribution: 1
initial_rating: 3
presentation: 2
questions: - The theoretical contribution of the paper uses Gaussian MoEs. In contrast, the empirical evaluation seems to use one of the variants of MoE layers, which became common in deep models in recent years. In comparison to Gaussian MoEs: 1) they use Top-k to achieve sparse computation 2) use plain two-layer neural networks as experts (not constrained to a single Gaussian). How relevant are the contributed convergence proofs to this type of MoE?
soundness: 1
strengths: - Although I have not verified it, the theoretical analysis appears exhaustive.
summary: This paper explores the use of different gating functions in Hierarchical Mixture of Experts. In particular, the authors derive expert convergence bounds for Gaussian HMoE with Softmax-Softmax, Softmax-Laplace, and Laplace-Laplace gating. The authors  empirically evaluate models that employ the proposed HMoE layers on multiple multimodal and computer vision tasks.

While the presented analysis is solid, the main contributions of this work are highly incremental over the recent relevant work. Moreover, evaluation does not seem to entirely reflect the claims and the results of the theoretical analysis. As such, I recommend rejection of this work as of now.
weaknesses: - **Limited novelty when stacked against recent related work** The main contribution of this work - convergence analysis for three HMoE variants - is highly incremental, as the main components used in this analyses (such as Voronoi losses) were already introduced in [1], while the Laplace gating for MoEs was introduced in [2]. Application of apparatus from [1] and [2] to HMoE instead of MoE is the only novel contribution of this work.
- **Empirical evaluation is not entirely aligned with the theoretical contribution of the paper** I find most of the experiments in the empirical part of the paper confusing when placed in context of the theoretical contribution. The theoretical part demonstrates the benefits of using Laplace-Laplace gating over the other variants, and I would expect the empirical part to be focused around supporting the theoretical findings. Instead, it appears the authors focus on showcasing the performence of their final HMoE-based model. Only Figure 2 compares the HMoE SS/LS/LL variants, while the rest of the empirical evaluation compares a single HMoE-based model to different multi-modal methods.
- **Unconvincing evaluation** As such, I am not entirely convinced that Laplace-Laplace gating offers consistent advantages also in the real-world, deep model setting. In line 521 the authors write: "best-performing Softmax-Laplace gating combination of HMoE in latent domain discovery". Does this mean that in some cases Laplace-Laplace has inferior performance? The authors should report the score of every variant also in Table 2 and Table 3.
- **Unclear purpose of empirical analysis** Similarly, the analysis from Figure 3 has the same issue. How does the choice of gating functions affect the token distribution? Would token distribution of a non-hierarchical MoE be significantly different? Without such a comparison the analysis is somewhat informative, but not relevant to the theoretical contribution of the paper.
- **Limited impact** In the introduction the authors mention the rising use of MoEs in modern foundational models. However, none of these use hierarchical routing, and the last work to tackle non-Gaussian HMoE was [3]. Given the community's lack of interest in HMoEs, the impact of this work may be limited.
- **Incomplete description of empirical evaluation** Every training hyperparameter should be provided for each experiment performed in the paper. The appendix describes the datasets used and data preprocessing only. The provided code is missing some of the experiments, e.g. for ImageNet. How many seeds were run for calculating the reported standard deviations?
- **Unfair comparison to MoE models** In some of the experiments the score of HMoE-based model is compared to non-hierarchical/standard MoE-based model. However, two levels of routing means two routers are used, resulting with HMoE having an effectively double the computational and memory resources spend on routing compared to MoE. How can be anyone sure that any gains in performance are due to the hierarchical nature of routing as opposed to additional computation and memory allocated for routing? Some recent works does utilize deeper routers, hinting that increased computation allocated for routing does help performance [4, 5].

***References:***

[1] Nguyen, Huy, TrungTin Nguyen, and Nhat Ho. "Demystifying softmax gating function in Gaussian mixture of experts." Advances in Neural Information Processing Systems 36 (2024).

[2] Han, Xing, et al. "Fusemoe: Mixture-of-experts transformers for fleximodal fusion." arXiv preprint arXiv:2402.03226 (2024).

[3] Shazeer, Noam, et al. "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer." International Conference on Learning Representations. 2016.

[4] Chi, Zewen, et al. "On the representation collapse of sparse mixture of experts." Advances in Neural Information Processing Systems 35 (2022): 34600-34613.

[5] Zhang, Zhengyan, et al. "MoEfication: Transformer Feed-forward Layers are Mixtures of Experts." Findings of the Association for Computational Linguistics: ACL 2022. 2022.