because $\mathbf{L}_0(\theta, t) \le \mathbf{L}_1(\theta, t)$ and $\mathbf{L}_1(\theta^*, t) \ge s^*$ by monotonicity. Similarly to the bound on $\mathbf{L}_0$, we derive

$$|\mathbf{M}_0(\theta, t) - \mathbf{M}_0(\theta^*, t)| \\ \le |\mathbb{P}_{\theta^*}(T(X, \theta) < t) - \mathbb{P}_{\theta^*}(T(X, \theta^*) < t)| + \mathbb{E}_{\theta^*} |U(X, \theta) - U(X, \theta^*)|.$$

Define $\bar{q}(X, \theta) \in \underset{q \in \{1, \dots, Q\}}{\operatorname{argmax}} \ell_q(X, \theta)$. Now, since $U(X, \theta^*) \le U(X, \theta)$ by definition (24), we have

$$\begin{aligned}
\mathbb{E}_{\theta^*} |U(X, \theta) - U(X, \theta^*)| &= \mathbb{E}_{\theta^*} [U(X, \theta) - U(X, \theta^*)] \\
&= \mathbb{E}_{\theta^*} [\ell_{\bar{q}(X, \theta)}(X, \theta^*) - \ell_{\bar{q}(X, \theta^*)}(X, \theta^*)] \\
&= \mathbb{E}_{\theta^*} [\ell_{\bar{q}(X, \theta)}(X, \theta^*) - \ell_{\bar{q}(X, \theta)}(X, \theta) + \ell_{\bar{q}(X, \theta^*)}(X, \theta) - \ell_{\bar{q}(X, \theta^*)}(X, \theta^*)] \\
&\le \mathbb{E}_{\theta^*} [\max_{1 \le q \le Q} |\ell_q(X, \theta^*) - \ell_q(X, \theta)|] + \mathbb{E}_{\theta^*} [\max_{1 \le q \le Q} \ell_q(X, \theta) - \max_{1 \le q \le Q} \ell_q(X, \theta^*)] \\
&\le 2 \mathbb{E}_{\theta^*} [\max_{1 \le q \le Q} |\ell_q(X, \theta^*) - \ell_q(X, \theta)|] \le 2 \Psi(\|\theta^* - \theta\|).
\end{aligned}$$

This proves (36) and leads to (37) by following the reasoning that provided (35). $\square$

Next, we have for $0 \le t' \le t \le 1$, by (34),

$$|\mathbf{L}_0(\theta, t) - \mathbf{L}_0(\theta, t')| \le |\mathbf{L}_0(\theta^*, t) - \mathbf{L}_0(\theta^*, t')| + 4\Psi(\|\theta^* - \theta\|).$$

Moreover,

$$\begin{aligned}
|\mathbf{L}_0(\theta^*, t) - \mathbf{L}_0(\theta^*, t')| &= \mathbf{L}_0(\theta^*, t) - \mathbf{L}_0(\theta^*, t') = \mathbb{E}_{\theta^*} [T(X, \theta^*) 1_{t' \le T(X, \theta^*) < t}] \\
&\le \mathbb{E}_{\theta^*} [1_{t' \le T(X, \theta^*) < t}] = \mathbb{P}_{\theta^*} (T(X, \theta^*) < t) - \mathbb{P}_{\theta^*} (T(X, \theta^*) < t'),
\end{aligned}$$

which is below $W_T(t - t')$ by (17). This leads to (38).

**Lemma 14** (Concentration of $\hat{\mathbf{L}}_0$, $\hat{\mathbf{L}}_1$, and $\hat{\mathbf{M}}_0$). For all $n \ge 1$, $x > 0, t \in [0, 1]$,

$$\mathbb{P}_{\theta^*} \left( \sup_{\theta \in \mathcal{D}} |\hat{\mathbf{L}}_0(\theta, t) - \mathbf{L}_0(\theta, t)| > x \right) \le e^{-nu^2}; \qquad (39)$$

$$\mathbb{P}_{\theta^*} \left( \sup_{\theta \in \mathcal{D}} |\hat{\mathbf{L}}_1(\theta, t) - \mathbf{L}_1(\theta, t)| > x \right) \le e^{-nu^2}; \qquad (40)$$

$$\mathbb{P}_{\theta^*} \left( \sup_{\theta \in \mathcal{D}} |\hat{\mathbf{M}}_0(\theta, t) - \mathbf{M}_0(\theta, t)| > x \right) \le e^{-nu^2}, \qquad (41)$$

where $u > 0$ is s.t. $x \ge \frac{2}{\sqrt{n}} + u$.

*Proof.* The concentration of each is a direct application of Talagrand's inequality. To be more precise, Theorem 5.3. in (Massart, 2007)) states the following. Let $\xi_1, \dots, \xi_n$ independent r.v., $\mathcal{F}$ a countable class of measurable functions s.t. $a \le f \le b$ for every $f \in \mathcal{F}$ for some real numbers $a, b$, and $W = \sup_{f \in \mathcal{F}} |\sum_{i=1}^n f(\xi_i) - \mathbb{E}(f(\xi_i))|$. Then, for any $x > 0$,

$$\mathbb{P}(W - \mathbb{E}(W) \ge x) \le e^{-\frac{2x^2}{n^2(b-a)^2}}.$$

The result then follows by applying Lemma 15, which bounds the expectation term $\mathbb{E}(W)$ in each case by $2\sqrt{n}$. $\square$
so that (9) follows by a direct integration w.r.t. the latent variable Z.

## B4 Proof of Lemma 4

The only nontrivial fact is for $W_{t^*,\alpha}(v)$. Assumption 1 and Lemma 8 provide that $t \mapsto \text{mFSR}_t^*$ is a one-to-one continuous increasing map from $(t^*(\alpha_c), t^*(\bar{\alpha}))$ to $(\alpha_c, \bar{\alpha})$. Hence, for $\alpha \in (\alpha_c, \bar{\alpha})$, $\beta \mapsto t^*(\alpha + \beta)$ is continuous in 0 and $\lim_{v \to 0} W_{t^*,\alpha}(v) = 0$.

## B5 Proof of Lemma 5

For all $\theta \in \Theta$ such that $\|\theta - \theta^*\|_2 \le \epsilon$, we have by the mean value theorem

$$ \sum_{q=1}^{Q} \mathbb{E}_{\theta^*} \left[ |\ell_q(X, \theta^*) - \ell_q(X, \theta)| \right] \le \sum_{q=1}^{Q} \mathbb{E}_{\theta^*} \sup_{\theta: \|\theta - \theta^*\|_2 \le \epsilon} \|\nabla_{\theta} \ell_q(X, \theta)\|_2 \|\theta - \theta^*\|_2 \\ \le C_1 \epsilon, $$

by using Assumption 3 (i). This implies $W_\ell(\epsilon) \le C_1\epsilon$. The fact $W_T(\delta) \le C_2\delta$, $W_{t^*,\alpha}(v) \le C_3v$ are obvious by using Assumption 3 (ii) and (iii). The fact $\Psi(\epsilon) \le \sqrt{C_1(C_2 + 1)}\sqrt{\epsilon}$ easily follows.

## B6 Proof of Lemma 6

By Lemma 8, we have that mFSR($C_t^*$) is monotonous in $t$ and continuous w.r.t. $t$ on $(t^*(\alpha_c), 1]$, thus for $\alpha \in (\alpha_c, \bar{\alpha}]$, mFSR($C_{t^*(\alpha)}^*$) = $\alpha$ which gives (i). For (ii), let $C = (\hat{Z}, S)$ be a procedure such that mFSR($C$) $\le \alpha$. Let us consider the procedure $C'$ with the Bayes clustering $\hat{Z}^*$ and the same selection rule $S$. Since $C'$ is based on a Bayes clustering, by the same reasoning leading to $R(\hat{Z}^*) \le R(\hat{Z})$ in Section 3.1, we have that mFSR($C'$) $\le$ mFSR($C$) $\le \alpha$ with

$$ \text{mFSR}(C') = \frac{\mathbb{E}_{\theta^*} \left( \sum_{i \in S} T_i^* \right)}{\mathbb{E}_{\theta^*} (|S|)}. $$

Hence,

$$ \mathbb{E}_{\theta^*} \left( \sum_{i \in S} T_i^* \right) \le \alpha \mathbb{E}_{\theta^*} (|S|). \qquad (B1) $$

Now we use an argument similar to the proof of Theorem 1 in Cai et al. (2019). By definition of $S_{t^*(\alpha)}^*$, we have that

$$ \sum_{i=1}^{n} \left( \mathbb{1}_{i \in S_{t^*(\alpha)}^*}(\mathbf{x}) - \mathbb{1}_{i \in S(\mathbf{x})} \right) (T_i^* - t^*(\alpha)) \le 0 $$

which we can rewrite as

$$ \sum_{i=1}^{n} \left( \mathbb{1}_{i \in S_{t^*(\alpha)}^*}(\mathbf{x}) - \mathbb{1}_{i \in S(\mathbf{x})} \right) (T_i^* - t^*(\alpha) + \alpha - \alpha) \le 0 $$

and so
**Lemma 15.** Let $\xi_1, \dots, \xi_n$ be independent r.v. and $\mathcal{F}$ be a countable set of measurable functions s.t. $\|f\|_{\infty} \le 1$ for $f \in \mathcal{F}$. Then, one has

$$ \mathbb{E} \left( \sup_{f \in \mathcal{F}} \left| \sum_{i=1}^{n} f(\xi_i) - \mathbb{E}(f(\xi_i)) \right| \right) \le 2\sqrt{n}. $$

*Proof.* We have that

$$ \mathbb{E} \left( \sup_{f \in \mathcal{F}} \left| \sum_{i=1}^{n} f(\xi_i) - \mathbb{E} \left( \sum_{i=1}^{n} f(\xi_i) \right) \right| \right) \le \mathbb{E} \left( \sup_{f \in \mathcal{F}} \left| \sum_{i=1}^{n} f(\xi_i) - f(\xi_i') \right| \right), $$

where the $\xi_i'$s are independent r.v. with the same distribution as the $\xi_i$'s. Let $(\epsilon_i)_{1 \le i \le n}$ be independent Rademacher variables. Then, we have

$$ \sum_{i=1}^{n} f(\xi_i) - f(\xi_i') \stackrel{\mathcal{D}}{=} \sum_{i=1}^{n} \epsilon_i(f(\xi_i) - f(\xi_i')) $$

and

$$ \sum_{i=1}^{n} f(\xi_i) - f(\xi_i') = \mathbb{E} \left( \left| \sum_{i=1}^{n} \epsilon_i(f(\xi_i) - f(\xi_i)) \right| \middle| \xi_1, \dots, \xi_n, \xi_1', \dots, \xi_n' \right). $$

Thus,

$$ \begin{aligned} \mathbb{E} \left( \sup_{f \in \mathcal{F}} \left| \sum_{i=1}^{n} f(\xi_i) - \mathbb{E} \left( \sum_{i=1}^{n} f(\xi_i) \right) \right| \right) &\le 2 \sup_{f \in \mathcal{F}} \mathbb{E} \left( \left| \sum_{i=1}^{n} \epsilon_i f(\xi_i) \right| \middle| \xi_1, \dots, \xi_n \right) \\ &\le 2 \sup_{f \in \mathcal{F}} \sqrt{\sum_{i=1}^{n} \epsilon_i f(\xi_i) \middle| \xi_1, \dots, \xi_n} \\ &= 2 \sup_{f \in \mathcal{F}} \sqrt{\sum_{i=1}^{n} (f(\xi_i))^2} \\ &\le 2\sqrt{n}. \end{aligned} $$

where the second inequality is due to Jensen's inequality. $\square$

# D Auxiliary results for the Gaussian case

## D.1 Convergence rate

The likelihood of Gaussian mixture models is unbounded in general (Chen, 2017). Thus, the convergence of MLEs has been studied over a constrained subspace of the parameter space that may expand as the sample size increases, which is often referred to as a sieve MLE.

**Proposition 16.** Consider the mixture model (Section 2.1) in the d-multivariate Gaussian case where $\phi_q = (\pi_q, \Sigma_q) \in \mathcal{U}$ with constrained parameter space defined as follows: $\mathcal{U} = [-a_n, a_n]^d \times \{\Sigma \in S_d^{++}, \lambda \le \lambda_1(\Sigma) \le \lambda_1(\Sigma) \le \bar{\lambda}\}$ where $a_n \le L(\log n)^\gamma$ for some $L, \gamma > 0$, $S_d^{++}$ denotes the space of positive definite matrices, with $\lambda, \bar{\lambda}$ some strictly positive constants. Then the (constrained) MLE is such that the corresponding quantity $\eta(\cdot, \theta^*)$ defined by (20) satisfies $\eta(\delta_n, \theta^*) \le 1/n$ for $n$ large enough, where $\delta_n$ is a sufficiently large multiple of $\sqrt{\log n/n}$.
$$
\begin{align*}
\mathbb{E}_{\theta^*} & \left( \sum_{i=1}^{n} \left( 1_{i \in S_{t^*(\alpha)}^*}(\mathbf{x}) - 1_{i \in S}(\mathbf{x}) \right) (T_i^* - \alpha) \right) \\
& \leq (t^*(\alpha) - \alpha) \mathbb{E}_{\theta^*} \left( \sum_{i=1}^{n} \left( 1_{i \in S_{t^*(\alpha)}^*}(\mathbf{x}) - 1_{i \in S}(\mathbf{x}) \right) \right) \\
& = (t^*(\alpha) - \alpha) \left( \mathbb{E}_{\theta^*}(|S_{t^*(\alpha)}^*|) - \mathbb{E}_{\theta^*}(|S|) \right).
\end{align*}
$$

On the other hand, mFSR($C_{t^*(\alpha)}^*$) = $\alpha$ together with (B1) implies that

$$
\begin{align*}
\mathbb{E}_{\theta^*} \left( \sum_{i=1}^{n} \left( 1_{i \in S_{t^*(\alpha)}^*}(\mathbf{x}) - 1_{i \in S}(\mathbf{x}) \right) (T_i^* - \alpha) \right) \\
&= \mathbb{E}_{\theta^*} \left( \sum_{i \in S_{t^*(\alpha)}^*} T_i^* - \alpha |S_{t^*(\alpha)}^*| - \sum_{i \in S} T_i^* + \alpha |S| \right) \geq 0.
\end{align*}
$$

Combining, the relations above provides

$$
(t^*(\alpha) - \alpha)(\mathbb{E}_{\theta^*}(|S_{t^*(\alpha)}^*|) - \mathbb{E}_{\theta^*}(|S|)) \geq 0.
$$

Finally, noting that $t^*(\alpha) - \alpha > 0$ since $\alpha = \text{mFSR}(C_{t^*(\alpha)}^*) < t^*(\alpha)$ by (ii) Lemma 8, this gives $\mathbb{E}_{\theta^*}(|S_{t^*(\alpha)}^*|) - \mathbb{E}_{\theta^*}(|S|) \ge 0$ and concludes the proof.

B7 Proof of Lemma 7

First, we have by definition $\ell_q(X_i, \theta^\sigma) = \ell_{\sigma(q)}(X_i, \theta)$ and thus $T(X_i, \hat{\theta}) = T(X_i, \hat{\theta}^\sigma)$ by taking the maximum over $q$. This gives $\hat{S}^\sigma = \hat{S}$ and yields the first equality. Next, we have on $\Omega_\epsilon$,

$$
\begin{align*}
\min_{\sigma' \in [Q]} \mathbb{E}_{\theta^*} \left( \varepsilon_{\hat{S}}(\sigma'(\hat{Z}), \mathbf{Z}) \middle| \mathbf{X} \right) &\le \mathbb{E}_{\theta^*} \left( \varepsilon_{\hat{S}}(\sigma(\hat{Z}), \mathbf{Z}) \middle| \mathbf{X} \right) \\
&\le \mathbb{E}_{\theta^*} \left( \varepsilon_{\hat{S}^\sigma}(\sigma(\hat{Z}), \mathbf{Z}) \middle| \mathbf{X} \right),
\end{align*}
$$

still because $\hat{S}^\sigma = \hat{S}$. Now observe that,

$$
\begin{align*}
\mathbb{E}_{\theta^*} \left( \varepsilon_{\hat{S}^\sigma} (\sigma(\hat{Z}), \mathbf{Z}) \middle| \mathbf{X} \right) &= \frac{1}{n} \sum_{i=1}^{n} \mathbb{P}_{\theta^*} (Z_i \neq \sigma(\bar{q}(X_i, \hat{\theta})) \middle| \mathbf{X}) \mathbb{1}_{T(X_i, \hat{\theta}^\sigma) < \hat{l}(\alpha)} \\
&= \frac{1}{n} \sum_{i=1}^{n} (1 - \ell_{i, \sigma(\bar{q}(X_i, \hat{\theta}))}^* ) \mathbb{1}_{T(X_i, \hat{\theta}^\sigma) < \hat{l}(\alpha)} \\
&= \hat{M}_0(\hat{\theta}^\sigma, \hat{l}(\alpha)),
\end{align*}
$$

because $\sigma(\bar{q}(X_i, \hat{\theta})) = \bar{q}(X_i, \hat{\theta}^\sigma)$. This proves the result.
**Proof.** In the considered model, any mixture can be defined in terms of {$f_u, u \in \mathcal{U}$} and a discrete mixing measure $G = \sum_{q=1}^Q \pi_q \delta_{\phi_q}$ with $Q$ support points, as $\sum_{q=1}^Q \pi_q f_{\phi_q} = \int f_u(x)dG(u)$. As shown by Ho and Nguyen (2016), the convergence of mixture model parameters can be measured in terms of a Wasserstein distance on the space of mixing measures. Let $G_1 = \sum_{q=1}^Q \pi_q^1 \delta_{\phi_q^1}$ and $G_2 = \sum_{q=1}^Q \pi_q^2 \delta_{\phi_q^2}$ be two discrete probability measures on some parameter space, which is equipped with metric $\|\cdot\|$. The Wasserstein distance of order 1 between $G_1$ and $G_2$ is given by

$$W_1(G_1, G_2) = \inf_{p} \sum_{q,l} p_{q,l} \|\phi_q^1 - \phi_l^2\|$$

where the infimum is over all couplings $(p_{q,l})_{1 \le q,l \le Q} \in [0,1]^{Q \times Q}$ such that $\sum_l p_{q,l} = \pi_q^1$ and $\sum_q p_{q,l} = \pi_l^2$. Let $G^*, \hat{G}_n$ denote the true mixing measure and the mixing measure that corresponds to the restricted MLE considered here, respectively. Theorem 4.2. in Ho and Nguyen (2016) implies that, with the notation of Ho and Nguyen (2016), for any $\epsilon_n \ge (\sqrt{C_1}/c)\delta_n$, and $\delta_n \le C\sqrt{\log n/n}$, we have $\mathbb{P}_{\theta^*}(W_1(\hat{G}_n, G^*) \ge (c/C_1)\epsilon_n) \le c e^{-n\epsilon_n^2}$. We apply this relation for $\epsilon_n = \max((\sqrt{C_1}/c)\delta_n, \sqrt{\log cn}/n)$. In that case, we have still $\epsilon_n$ of order $\sqrt{\log n/n}$ and the upper-bound is at most $1/n$. On the other hand, if we have a convergence rate in terms of $W_1$, then we have convergence of the mixture model parameters in terms of $\|\cdot\|$ at the same rate, see Lemma 17. This concludes the proof. $\square$

**Lemma 17.** Let $G_n = \sum_{q=1}^Q \pi_q^n \delta_{\phi_q^n}$ be a sequence of discrete probability measures on $\mathcal{U}$, and let $G^*, W_1$ be defined as in the proof of Proposition 16. There exists a constant $C$ only depending on $G^*$ such that if $W_1(G_n, G^*) \to 0$, then for sufficiently large $n$,

$$W_1(G_n, G^*) \ge C \min_{\sigma \in [Q]} \|\theta_n^\sigma - \theta^*\|.$$

**Proof.** In what follows, we let $\{p_{q,l}^n\}$ denote the corresponding probabilities of the optimal coupling for the pair $(G_n, G^*)$. We start by showing that $(\phi_q^n)_q \to (\phi_q^*)_q$ in $\|\cdot\|$ up to a permutation of the labels. Let $\sigma^n$ the permutation of the labels such that $\|\phi_q^n - \phi_l^*\| \ge \|\phi_{\sigma^n(l)}^n - \phi_l^*\|$ for all $q, l \in \{1, ..., Q\}$. Then, by definition,

$$W_1(G_n, G^*) \ge \sum_{1 \le q,l \le Q} p_{q,l}^n \|\phi_{\sigma(l)}^n - \phi_l^*\| \\ = \sum_l \pi_l^* \|\phi_{\sigma^n(l)}^n - \phi_l^*\|.$$

It follows that each $\|\phi_{\sigma^n(l)}^n - \phi_l^*\|$ must converge to zero. Since $(\phi_q^n)_q \to (\phi_q^*)_q$ up to a permutation of the labels, without loss of generality we can assume that $\phi_q^n \to \phi_q^*$ for all $q$. Let $\Delta\phi_q^n := \phi_q^n - \phi_l^*$ and $\Delta\pi_q^n := \pi_q^n - \pi_l^*$. Write $W_1(G_n, G^*)$ as

$$W_1(G_n, G^*) = \sum_q p_{qq}^n \|\Delta\phi_q^n\| + \sum_{q \ne l} p_{ql}^n \|\phi_q^n - \phi_l^*\|$$

Define $C_{ql} = \|\phi_q^* - \phi_l^*\|$ and $C = \min_{q \ne l} C_{ql} > 0$. It follows from the convergence of $\phi^n$ that for $q \ne l$, $\|\phi_q^n - \phi_l^*\| \ge C/2$ for sufficiently large $n$. Thus,

$$W_1(G_n, G^*) \ge \frac{C}{2} \sum_{q \ne l} p_{ql}^n$$

We deduce that $\sum_{q \ne l} p_{ql}^n \to 0$. As a result, $p_{qq}^n = \pi_q^* - \sum_{l \ne q} p_{lq}^n \to \pi_q^*$, and so, $p_{qq}^n \ge (1/2)\pi_{\min}^* := \min_l \pi_l^*$ for sufficiently large $n$. On the other hand, $\sum_{q \ne l} p_{ql}^n = \sum_q \pi_q^n - p_{qq}^n =$
APPENDIX C. AUXILIARY RESULTS

**Lemma 8.** Let us consider the procedure $C_t^*$ defined in Section A.4, the mFSR criterion defined by (3) and the functional mFSR$_t^*$ defined by (12). Then we have

$$ \text{mFSR}_t^* = \text{mFSR}_{\theta^*}(C_t^*) = \frac{\mathbb{E}_{\theta^*} \left( \sum_{i=1}^n T_i^* \mathbb{1}_{T_i^* < t} \right)}{\mathbb{E}_{\theta^*} \left( \sum_{i=1}^n \mathbb{1}_{T_i^* < t} \right)}, \quad t \in [0, 1]. \quad (C1) $$

Moreover, the following properties for the function $t \in [0, 1] \mapsto \text{mFSR}_{\theta^*}(C_t^*)$:

(i) mFSR$_{\theta^*} (C_t^*)$ is nondecreasing in $t \in [0, 1]$ and, under Assumption 1, it is increasing in $t \in (t^*(\alpha_c), t^*(\bar{\alpha}))$;

(ii) mFSR$_{\theta^*} (C_t^*) < t$ for $t \in (0, 1]$;

(iii) Under Assumption 1, mFSR$_{\theta^*} (C_t^*)$ is continuous w.r.t. $t$ on $(t^*(\alpha_c), 1]$, where $t^*(\alpha_c)$ is given by (14).

*Proof.* Write mFSR$_{\theta^*} (\cdot)$ instead of mFSR($\cdot$) for short. First, (C1) is obtained similarly than (8). For proving (i), let $t_1, t_2 \in [0, 1]$ such that $t_1 < t_2$. We show that mFSR$(C_t^*) \le \text{mFSR}(C_{t_2}^*)$. Remember here the convention $0/0 = 0$ and that mFSR$(C_t^*) = \mathbb{E}_{\theta^*} (T(X, \theta^*) | \bar{T}(X, \theta^*) < t)$. First, if $\mathbb{P}_{\theta^*} (T(X, \theta^*) < t_1) = 0$ then the result is immediate. Otherwise, we have that

$$ \begin{aligned} & \text{mFSR}(C_{t_1}^*) - \text{mFSR}(C_{t_2}^*) \\ &= (\mathbb{P}_{\theta^*} (T(X, \theta^*) < t_1))^{-1} \cdot \mathbb{E}_{\theta^*} \left( T(X, \theta^*) \left\{ \mathbb{1}_{T(X, \theta^*) < t_1} - \frac{\mathbb{P}_{\theta^*} (T(X, \theta^*) < t_1)}{\mathbb{P}_{\theta^*} (T(X, \theta^*) < t_2)} \mathbb{1}_{T(X, \theta^*) < t_2} \right\} \right), \end{aligned} $$

where, given that $t_1 < t_2$, the quantity in the brackets is positive when $T(X, \theta^*) < t_1$ and is negative or zero otherwise. Hence,

$$ \begin{aligned} & T(X, \theta^*) \left\{ \mathbb{1}_{T(X, \theta^*) < t_1} - \frac{\mathbb{P}_{\theta^*} (T(X, \theta^*) < t_1)}{\mathbb{P}_{\theta^*} (T(X, \theta^*) < t_2)} \mathbb{1}_{T(X, \theta^*) < t_2} \right\} \\ & \le t_1 \left\{ \mathbb{1}_{T(X, \theta^*) < t_1} - \frac{\mathbb{P}_{\theta^*} (T(X, \theta^*) < t_1)}{\mathbb{P}_{\theta^*} (T(X, \theta^*) < t_2)} \mathbb{1}_{T(X, \theta^*) < t_2} \right\}. \end{aligned} $$

Taking the expectation makes the right-hand-side equal to zero, from which the result follows. Now, to show the increasingness, if mFSR$(C_{t_1}^*) = \text{mFSR}(C_{t_2}^*)$ for $t^*(\alpha_c) < t_1 < t_2 < t^*(\bar{\alpha})$, then the above reasoning shows that

$$ (T(X, \theta^*) - t_1) \left\{ \mathbb{1}_{T(X, \theta^*) < t_1} - \frac{\mathbb{P}_{\theta^*} (T(X, \theta^*) < t_1)}{\mathbb{P}_{\theta^*} (T(X, \theta^*) < t_2)} \mathbb{1}_{T(X, \theta^*) < t_2} \right\} \le 0 $$

and has an expectation equal to 0. Hence, given that $T(X, \theta^*)$ is continuous, we derive that almost surely

$$ \mathbb{P}_{\theta^*} (T(X, \theta^*) < t_2) \mathbb{1}_{T(X, \theta^*) < t_1} = \mathbb{P}_{\theta^*} (T(X, \theta^*) < t_1) \mathbb{1}_{T(X, \theta^*) < t_2}, $$
$\sum_q \pi_q^* - p_{qq}^n$ where $p_{qq}^n \le \min(\pi_q^n, \pi_q^*)$. Thus, $\sum_{q \ne l} p_{ql}^n \ge \sum_q \pi_q^n - \min(\pi_q^n, \pi_q^*) = \sum_{q, \pi_q^n \ge \pi_q^*} \pi_q^n - \pi_q^* = \sum_{q, \pi_q^n \ge \pi_q^*} |\pi_q^n - \pi_q^*|$ and similarly we have that $\sum_{q \ne l} p_{ql}^n \ge \sum_{q, \pi_q^n \ge \pi_q^n} |\pi_q^n - \pi_q^*|$. It follows that $2 \sum_{q \ne l} p_{ql}^n \ge \sum_q |\pi_q^n - \pi_q^*|$. Therefore, for sufficiently large $n$,

$$W_1(G_n, G^*) \ge \frac{1}{2} \pi_{\min}^* \sum_q \|\Delta\phi_q^n\| + \frac{C}{4} \sum_q |\Delta\pi_q^n|.$$

This gives the result.

<figure><img src="image_13.png" /></figure>

## D.2 Gaussian computations

The following lemma holds.

**Lemma 18.** Let us consider the multivariate Gaussian case where $\phi_q = (\mu_q, \Sigma_q)$, $1 \le q \le Q$, with $Q = 2$, $\Sigma_1 = \Sigma_2$ is an invertible covariance matrix and $\mu_1$ and $\mu_2$ are two different vectors of $\mathbb{R}^d$. Then Assumptions 1, 2 and 3 hold true for $\alpha_c = 0$ and for a level $\alpha \in (0, \bar{\alpha}) \setminus \mathcal{E}$ for $\mathcal{E}$ a set of Lebesgue measure 0.

*Proof.* Let us first prove that $T(X, \theta)$ is a continuous random variable under $\mathbb{P}_{\theta^*}$ (this is established below without assuming $\Sigma_1 = \Sigma_2$ for the sake of generality). We have

$$ \begin{aligned} \mathbb{P}_{\theta^*}(T(X, \theta) = t) &= \mathbb{P}_{\theta^*} \left( 1 - \max_{1 \le q \le Q} \left\{ \frac{\pi_q f_{\phi_q}(X)}{\sum_{\ell=1}^Q \pi_\ell f_{\phi_\ell}(X)} \right\} = t \right) \\ &= \mathbb{P}_{\theta^*} \left( \max_{1 \le q \le Q} \{ \pi_q f_{\phi_q}(X) \} = (1-t) \sum_{\ell=1}^Q \pi_\ell f_{\phi_\ell}(X) \right) \\ &\le \sum_{q=1}^Q \mathbb{P}_{\theta^*} \left( \pi_q f_{\phi_q}(X) = (1-t) \sum_{\ell=1}^Q \pi_\ell f_{\phi_\ell}(X) \right). \end{aligned} $$

Since $Q = 2$, the latter probability is (for say $q = 1$), equal to

$$ \begin{aligned} \mathbb{P}_{\theta^*} (\pi_1 f_{\phi_1}(X) = (1/t - 1)\pi_2 f_{\phi_2}(X)) \\ = \mathbb{P}_{\theta^*} (f_{\phi_1}(X)/f_{\phi_2}(X) = (1/t - 1)\pi_2/\pi_1) \\ = \mathbb{P}_{\theta^*} ((X - \mu_1)^t \Sigma_1^{-1} (X - \mu_1) - (X - \mu_2)^t \Sigma_2^{-1} (X - \mu_2) = -2 \log((1/t - 1)\pi_2/\pi_1) - \log(|\Sigma_1|/|\Sigma_2|)). \end{aligned} $$

Now,

$$ \begin{aligned} & (X - \mu_1)^t \Sigma_1^{-1} (X - \mu_1) - (X - \mu_2)^t \Sigma_2^{-1} (X - \mu_2) \\ &= (X - \mu_1)^t \Sigma_1^{-1} (X - \mu_1) - (X - \mu_1)^t \Sigma_2^{-1} (X - \mu_2) - (\mu_1 - \mu_2)^t \Sigma_2^{-1} (X - \mu_2) \\ &= (X - \mu_1)^t (\Sigma_1^{-1} - \Sigma_2^{-1}) (X - \mu_1) - (X - \mu_1)^t \Sigma_2^{-1} (\mu_1 - \mu_2) - (\mu_1 - \mu_2)^t \Sigma_2^{-1} (X - \mu_2) \\ &= (X - \mu_1)^t (\Sigma_1^{-1} - \Sigma_2^{-1}) (X - \mu_1) - (\mu_1 - \mu_2)^t \Sigma_2^{-1} (2X - \mu_2 - \mu_1). \end{aligned} $$

Since the real matrix $\Sigma_1^{-1} - \Sigma_2^{-1}$ is symmetric, we can diagonalize it and we end up with a subset of $\mathbb{R}^d$ of the form

$$ \left\{ y \in \mathbb{R}^d : \sum_{j=1}^{d} (\alpha_j y_j^2 + \beta_j y_j) + \gamma = 0 \right\}, $$

for some real parameters $\alpha_j, \beta_j, \gamma$. The result follows because this set has a Lebesgue measure equal to 0 in any case.
that is, $\mathbb{P}_{\theta^*}(t_1 \le T_i^* < t_2) = 0$, which is excluded by Assumption 1. This entails $\text{mFSR}(C_{t_1}^*) < \text{mFSR}(C_{t_2}^*)$.

For proving (ii), let $t > 0$. If $\mathbb{P}_{\theta^*}(T(X, \theta^*) < t) = 0$ then the result is immediate. Otherwise, we have that $\text{mFSR}(C_t^*) - t = (\mathbb{P}_{\theta^*}(T(X, \theta^*) < t))^{-1}\mathbb{E}_{\theta^*}((T(X, \theta^*) - t)\mathbb{1}\{T(X, \theta^*) < t\})$. The latter is clearly not positive, and is moreover negative because $(T(X, \theta^*) - t)\mathbb{1}\{T(X, \theta^*) < t\} \le 0$ and $\mathbb{P}_{\theta^*}(T(X, \theta^*) = t) = 0$ by Assumption 1.

For proving (iii), let $\psi_0(t) = \mathbb{E}_{\theta^*}(T(X, \theta^*)\mathbb{1}\{T(X, \theta^*) < t\})$ and $\psi_1(t) = \mathbb{P}_{\theta^*}(T(X, \theta^*) < t)$, the numerator and denominator of $\text{mFSR}(C_t^*) = \text{mFSR}_t^*$, respectively. $\psi_1(t)$ is nondecreasing in $t$, with $\psi_1(0) = 0$ and $\psi_1(1) > 0$. Moreover, $\psi_0$ and $\psi_1$ are both continuous under Assumption 1. Then denote by $t_c$ the largest $t$ s.t. $\psi_1(t) = 0$. $\psi_1$ is zero on $[0, t_c]$ then strictly positive and nondecreasing on $(t_c, 1]$, and we have that $t_c = t^*(\alpha_c)$. Hence, $\text{mFSR}(C_t^*)$ is zero on $[0, t_c]$ then strictly positive and continuous on $(t_c, 1]$. ■

**Remark 4.** With the notation of the above proof, $t \mapsto \text{mFSR}(C_t^*)$ may have a discontinuity point at $t_c$ since for $t_n \xrightarrow{t_n \to t_c} t_c$, as $\psi_1(t_n) \to 0$, one does not necessarily have that $\text{mFSR}(C_t^*) \to 0$.

**Lemma 9** (Expression of plug-in procedure as a thresholding rule). For any $\alpha \in (0, 1)$, let us consider the plug-in procedure $\hat{C}_\alpha^{\text{PI}} = (\hat{Z}^{\text{PI}}, \hat{S}_\alpha^{\text{PI}})$ defined by Algorithm 2 and denote $K = |\hat{S}_\alpha^{\text{PI}}|$ the maximum of the $k \in \{0, \dots, n\}$ such that $\max(k, 1)^{-1}\sum_{j=1}^k \hat{T}_j \le \alpha$ for $\hat{T}_i = 1 - \max_q \ell_q(X_i, \hat{\theta})$, $1 \le i \le n$. Consider also $\hat{t}(\alpha)$ defined by (A12). Let Assumption 1 be true and consider an estimator $\hat{\theta}$ satisfying Assumption 5. Then it holds that $\hat{t}(\alpha) = \hat{T}_{(K+1)}$ and

$$ \hat{S}_\alpha^{\text{PI}} = \{i \in \{1, \dots, n\} : \hat{T}_i < \hat{t}(\alpha)\}. $$

*Proof.* If $\hat{T}_{(K)} < \hat{T}_{(K+1)}$ then the result is immediate. Thus it suffices to show that $\hat{T}_{(K)} = \hat{T}_{(K+1)}$ occurs with probability 0. From Assumption 5 (with the countable set $D$ defined therein), we have

$$ \mathbb{P}_{\theta^*}(\hat{T}_{(K)} = \hat{T}_{(K+1)}) \le \mathbb{P}_{\theta^*}\left(\bigcup_{i \ne j}\{\hat{T}_i = \hat{T}_j\}\right) \le \mathbb{P}_{\theta^*}\left(\bigcup_{\theta \in D}\bigcup_{i \ne j}\{T(X_i, \theta) = T(X_j, \theta)\}\right). $$

Now, the right term is a countable union of events which are all of null probability under Assumption 1. The result follows. ■

**Lemma 10.** We have for all $\theta \in \Theta$,

$$ \sup_{t \in [0,1]} |\mathbf{L}_1(\theta, t) - \mathbf{L}_1(\theta^*, t)| \le \Psi(\|\theta^* - \theta\|); \quad (C2) $$

$$ \sup_{t \in [0,1]} |\mathbf{L}_0(\theta, t) - \mathbf{L}_0(\theta^*, t)| \le 2\Psi(\|\theta^* - \theta\|); \quad (C3) $$

$$ \sup_{t \in [t^*((\alpha+\alpha_c)/2),1]} |\mathbf{L}(\theta, t) - \mathbf{L}(\theta^*, t)| \le 3\Psi(\|\theta^* - \theta\|)/s^*; \quad (C4) $$

$$ \sup_{t \in [0,1]} |\mathbf{M}_0(\theta, t) - \mathbf{M}_0(\theta^*, t)| \le 3\Psi(\|\theta^* - \theta\|); \quad (C5) $$

$$ \sup_{t \in [t^*((\alpha+\alpha_c)/2),1]} |\mathbf{M}(\theta, t) - \mathbf{M}(\theta^*, t)| \le 4\Psi(\|\theta^* - \theta\|)/s^*; \quad (C6) $$
Now, since $\Sigma_1 = \Sigma_2 = \Sigma$, we have for all $t \in (0, 1)$,

$$
\begin{aligned}
\{T(X, \theta) > t\} &= \left\{ \forall q \in \{1, \dots, Q\}, \pi_q f_{\phi_q}(X) < (1-t) \sum_{\ell=1}^{Q} \pi_{\ell} f_{\phi_{\ell}}(X) \right\} \\
&= \{\pi_1 f_{\phi_1}(X) < (1/t - 1)\pi_2 f_{\phi_2}(X)\} \cap \{\pi_2 f_{\phi_2}(X) < (1/t - 1)\pi_1 f_{\phi_1}(X)\} \\
&= \left\{ (1/t - 1)^{-1} < \frac{\pi_1 f_{\phi_1}(X)}{\pi_2 f_{\phi_2}(X)} < (1/t - 1) \right\}.
\end{aligned}
$$

Applying $2\log(\cdot)$ on each part of the relation, we obtain

$$ \{T(X, \theta) > t\} = \{-2\log(1/t - 1) < a^t X + b < 2\log(1/t - 1)\}, $$

for

$$
\begin{aligned}
a &= a(\theta) = 2\Sigma^{-1}(\mu_1 - \mu_2) \in \mathbb{R}^d \setminus \{0\} \\
b &= b(\theta) = -(\mu_1 - \mu_2)^t \Sigma^{-1}(\mu_1 + \mu_2) + 2\log(\pi_1/\pi_2) \in \mathbb{R}^d.
\end{aligned}
$$

Since under $P_{\theta^*}$ we have $X \sim \pi_1^*\mathcal{N}(\mu_1^*, \Sigma^*) + \pi_2^*\mathcal{N}(\mu_2^*, \Sigma^*)$, we have $a^t X + b \sim \pi_1^*\mathcal{N}(a^t \mu_1^* + b, a^t \Sigma^* a) + \pi_2^*\mathcal{N}(a^t \mu_2^* + b, a^t \Sigma^* a)$. This yields for all $t \in (0, 1)$,

$$
\begin{aligned}
P_{\theta^*}(T(X, \theta) > t) = & \pi_1 \left[ \Phi \left( \frac{2 \log(1/t - 1) - a^t \mu_1^* - b}{(a^t \Sigma^* a)^{1/2}} \right) - \Phi \left( \frac{-2 \log(1/t - 1) - a^t \mu_1^* - b}{(a^t \Sigma^* a)^{1/2}} \right) \right] \\
& + \pi_2 \left[ \Phi \left( \frac{2 \log(1/t - 1) - a^t \mu_2^* - b}{(a^t \Sigma^* a)^{1/2}} \right) - \Phi \left( \frac{-2 \log(1/t - 1) - a^t \mu_2^* - b}{(a^t \Sigma^* a)^{1/2}} \right) \right].
\end{aligned}
\quad (42)
$$

A direct consequence is that for all $t \in (0, 1)$, we have $P_{\theta^*}(T(X, \theta) > t) < 1$, that is, $P_{\theta^*}(T(X, \theta) \le t) = P_{\theta^*}(T(X, \theta) < t) > 0$. Hence, $\alpha_c$ defined in (14) is equal to zero. Moreover, from (42), we clearly have that $t \in (0, 1) \mapsto P_{\theta^*}(T(X, \theta) > t)$ is decreasing, so that $t \in (0, 1) \mapsto P_{\theta^*}(T(X, \theta) \le t)$ is increasing. This proves that Assumption 1 holds in that case.

Let us now check Assumptions 2 and 3. Assumptions 2 and 3 (i) follow from Result 2.1 in Melnykov (2013).

As for Assumption 3 (ii), from (42), we only have to show that the function $t \in (0, 1) \mapsto \frac{\partial}{\partial t} \Phi(\frac{\log(1/t-1)-\alpha^*}{\beta^*})$ is uniformly bounded by some constant $C = C(\alpha^*, \beta^*)$, for any $\alpha^* \in \mathbb{R}$ and $\beta^* > 0$. A straightforward calculation leads to the following: for all $t \in (0, 1)$,

$$ \left| \frac{\partial}{\partial t} \Phi \left( \frac{\log(1/t - 1) - \alpha^*}{\beta^*} \right) \right| = \frac{e^{-\left(\frac{\log(1/t-1)-\alpha^*}{\beta^*}\right)^2/2}}{\beta^* \sqrt{2\pi}} \frac{1}{t(1-t)}. \quad (43) $$

Consider now $t_0 = t_0(\alpha^*, \beta^*) \in (0, 1/2)$ such that $(\frac{\log(1/t-1)-\alpha^*}{\beta^*})^2 \ge 2\log(1/t)$ for all $t \in (0, t_0)$. It is clear that the right-hand-side of (43) is upper-bounded by $\frac{1}{\beta^*\sqrt{2\pi(1-t_0)}}$ on $t \in (0, t_0)$. Similarly, let $t_1 = t_1(\alpha^*, \beta^*) \in (1/2, 1)$ such that $(\frac{\log(1/t-1)-\alpha^*}{\beta^*})^2 \ge 2\log(1/(1-t))$ for all $t \in (t_1, 1)$. It is clear that the right-hand-side of (43) is upper-bounded by $\frac{1}{\beta^*\sqrt{2\pi t_1}}$ on $t \in (t_1, 1)$. Finally, for $t \in [t_0, t_1]$, the upper-bound $\frac{1}{\beta^*\sqrt{2\pi t_0(1-t_1)}}$ is valid. This proves that Assumption 3 (ii) holds.

Let us now finally turn to Assumption 3 (iii). Lemma 11 ensures that $t \in (0, t^*(\bar{\alpha})) \mapsto \text{mFCR}_t^*$ is continuous increasing. Hence, $t^* : \beta \in (0, \bar{\alpha}) \mapsto t^*(\beta)$ defined in (13) is the inverse of this function and is also continuous increasing. It is therefore differentiable almost everywhere in $(0, \bar{\alpha})$, so everywhere in $(0, \bar{\alpha}) \setminus \mathcal{E}$ where $\mathcal{E}$ is a set of Lebesgue measure 0. By taking $\alpha$ in $(0, \bar{\alpha}) \setminus \mathcal{E}$, this ensures that $t^*$ is differentiable in $\alpha$ and thus that Assumption 3 (iii) holds. $\square$
where $\alpha \in (\alpha_c, \bar{\alpha}]$ and $s^* > 0$ is given by (A11). In addition, for all $\theta \in \Theta$ and $t, t' \in [0, 1]$,

$$|\mathbf{L}_0(\theta, t) - \mathbf{L}_0(\theta, t')| \le 4\Psi(\|\theta^* - \theta\|) + W_T(|t - t'|). \quad (C7)$$

*Proof.* Fix $\theta \in \Theta$ and $t \in [0, 1]$. We have for any $\delta > 0$,

$$\begin{aligned} & |\mathbb{P}_{\theta^*}(T(X, \theta) < t) - \mathbb{P}_{\theta^*}(T(X, \theta^*) < t)| \\ & \le (\mathbb{P}_{\theta^*}(T(X, \theta^*) < t + \delta) - \mathbb{P}_{\theta^*}(T(X, \theta^*) < t)) \lor (\mathbb{P}_{\theta^*}(T(X, \theta^*) < t) - \mathbb{P}_{\theta^*}(T(X, \theta^*) < t - \delta)) \\ & \quad + \mathbb{P}_{\theta^*}(|T(X, \theta^*) - T(X, \theta)| > \delta) \\ & \le W_T(\delta) + \mathbb{E}_{\theta^*}(|T(X, \theta^*) - T(X, \theta)|)/\delta. \end{aligned}$$

In addition, by definition (5),

$$\begin{aligned} |T(X, \theta^*) - T(X, \theta)| & \le \max_{1 \le q \le Q} \ell_q(X, \theta^*) - \max_{1 \le q \le Q} \ell_q(X, \theta)| \\ & \le \max_{1 \le q \le Q} |\ell_q(X, \theta^*) - \ell_q(X, \theta)|. \end{aligned}$$

This implies by definition of $W_\ell(\cdot)$ (see A1) that

$$\mathbb{E}_{\theta^*}(|T(X, \theta^*) - T(X, \theta)|) \le W_\ell(\|\theta - \theta^*\|).$$

Hence,

$$|\mathbb{P}_{\theta^*}(T(X, \theta) < t) - \mathbb{P}_{\theta^*}(T(X, \theta^*) < t)| \le \inf_{\delta \in (0,1)} \{W_T(\delta) + W_\ell(\|\theta^* - \theta\|)/\delta\} \le \Psi(\|\theta^* - \theta\|),$$

which establishes (C2).

Next, we have

$$\begin{aligned} \mathbf{L}_0(\theta, t) - \mathbf{L}_0(\theta^*, t) & = \mathbb{E}_{\theta^*} [T(X, \theta)(1_{T(X,\theta)<t} - 1_{T(X,\theta^*)<t}) + 1_{T(X,\theta^*)<t}(T(X,\theta) - T(X,\theta^*)]) \\ & \le t |\mathbb{P}_{\theta^*}(T(X, \theta) < t) - \mathbb{P}_{\theta^*}(T(X, \theta^*) < t)| \\ & \quad + |\mathbb{E}_{\theta^*} [1_{T(X,\theta^*)<t}(T(X, \theta) - T(X, \theta^*))]| \\ & \le |\mathbb{P}_{\theta^*}(T(X, \theta) < t) - \mathbb{P}_{\theta^*}(T(X, \theta^*) < t)| + \mathbb{E}_{\theta^*} |T(X, \theta) - T(X, \theta^*)| \\ & \le 2\Psi(\|\theta^* - \theta\|) \end{aligned}$$

By exchanging the role of $\theta$ and $\theta^*$ in the above reasoning, the same bound holds for $\mathbf{L}_0(\theta^*, t) - \mathbf{L}_0(\theta, t)$, which gives (C3). To prove (C4), we use for any $t \in [t^*(\frac{\alpha+\alpha_c}{2}), 1]$,

$$\left| \frac{\mathbf{L}_0(\theta, t)}{\mathbf{L}_1(\theta, t)} - \frac{\mathbf{L}_0(\theta^*, t)}{\mathbf{L}_1(\theta^*, t)} \right| \le \left| \frac{\mathbf{L}_0(\theta, t) - \mathbf{L}_0(\theta^*, t)}{\mathbf{L}_1(\theta^*, t)} \right| + \mathbf{L}_0(\theta, t) \left| \frac{1}{\mathbf{L}_1(\theta^*, t)} - \frac{1}{\mathbf{L}_1(\theta, t)} \right|$$
<figure><img src="image_14.png" /><figcaption>Figure 9: Same as Figure 5 with dimension $d = 20$.</figcaption></figure>

## E Further experiments
$$ \begin{aligned} & \le 2\Psi(\|\theta^* - \theta\|)/s^* + \frac{1}{\mathbf{L}_1(\theta^*, t)} \frac{\mathbf{L}_0(\theta, t)}{\mathbf{L}_1(\theta, t)} |\mathbb{P}_{\theta^*}(T(X, \theta^*) < t) - \mathbb{P}_{\theta^*}(T(X, \theta) < t)| \\ & \le 3\Psi(\|\theta^* - \theta\|)/s^*, \end{aligned} $$

because $\mathbf{L}_0(\theta, t) \le \mathbf{L}_1(\theta, t)$ and $\mathbf{L}_1(\theta^*, t) \ge s^*$ by monotonicity. Similarly to the bound on $\mathbf{L}_0$, we derive

$$ \begin{aligned} & |\mathbf{M}_0(\theta, t) - \mathbf{M}_0(\theta^*, t)| \\ & \le |\mathbb{P}_{\theta^*}(T(X, \theta) < t) - \mathbb{P}_{\theta^*}(T(X, \theta^*) < t)| + \mathbb{E}_{\theta^*}|U(X, \theta) - U(X, \theta^*)|. \end{aligned} $$

Define $\bar{q}(X, \theta) \in \underset{q \in \{1, \dots, Q\}}{\operatorname{argmax}} \ell_q(X, \theta)$. Now, since $U(X, \theta^*) \le U(X, \theta)$ by definition (A9), we have

$$ \begin{aligned} \mathbb{E}_{\theta^*} |U(X, \theta) - U(X, \theta^*)| &= \mathbb{E}_{\theta^*} [U(X, \theta) - U(X, \theta^*)] \\ &= \mathbb{E}_{\theta^*} [\ell_{\bar{q}(X, \theta)}(X, \theta^*) - \ell_{\bar{q}(X, \theta^*)}(X, \theta^*)] \\ &= \mathbb{E}_{\theta^*} [\ell_{\bar{q}(X, \theta)}(X, \theta^*) - \ell_{\bar{q}(X, \theta)}(X, \theta) \\ & \quad + \ell_{\bar{q}(X, \theta)}(X, \theta) - \ell_{\bar{q}(X, \theta^*)}(X, \theta^*)] \\ & \le \mathbb{E}_{\theta^*} [\max_{1 \le q \le Q} |\ell_q(X, \theta^*) - \ell_q(X, \theta)|] \\ & \quad + \mathbb{E}_{\theta^*} [\max_{1 \le q \le Q} \ell_q(X, \theta) - \max_{1 \le q \le Q} \ell_q(X, \theta^*)] \\ & \le 2\mathbb{E}_{\theta^*} [\max_{1 \le q \le Q} |\ell_q(X, \theta^*) - \ell_q(X, \theta)|] \le 2\Psi(\|\theta^* - \theta\|). \end{aligned} $$

This proves (C5) and leads to (C6) by following the reasoning that provided (C4). Next, we have for $0 \le t' \le t \le 1$, by (C3),

$$ |\mathbf{L}_0(\theta, t) - \mathbf{L}_0(\theta, t')| \le |\mathbf{L}_0(\theta^*, t) - \mathbf{L}_0(\theta^*, t')| + 4\Psi(\|\theta^* - \theta\|). $$

Moreover,

$$ \begin{aligned} |\mathbf{L}_0(\theta^*, t) - \mathbf{L}_0(\theta^*, t')| &= \mathbf{L}_0(\theta^*, t) - \mathbf{L}_0(\theta^*, t') = \mathbb{E}_{\theta^*} [T(X, \theta^*) 1_{t' \le T(X, \theta^*) < t}] \\ & \le \mathbb{E}_{\theta^*} [1_{t' \le T(X, \theta^*) < t}] \\ &= \mathbb{P}_{\theta^*}(T(X, \theta^*) < t) - \mathbb{P}_{\theta^*}(T(X, \theta^*) < t'), \end{aligned} $$

which is below $\mathcal{W}_T(t - t')$ by (A3). This leads to (C7).

■

**Lemma 11** (Concentration of $\hat{\mathbf{L}}_0$ (A6), $\hat{\mathbf{L}}_1$ (A7), and $\hat{\mathbf{M}}_0$ (A10)). Let Assumption 1 be true. Recall $\mathcal{V}$, $\mathcal{V}_-$ defined by (16), (17) respectively, set $c := 14Q\sqrt{\mathcal{V}} + 7Q^2\sqrt{\mathcal{V}_-}$ and consider any countable set $D \subset \Theta$. For all $t \in (0, 1]$ and for $n \ge (2e)^3$, we have

$$ \mathbb{P}_{\theta^*} \left( \sup_{\theta \in D} |\hat{\mathbf{L}}_0(\theta, t) - \mathbf{L}_0(\theta, t)| > x \right) \le n^{-2}; \qquad (C8) $$

$$ \mathbb{P}_{\theta^*} \left( \sup_{\theta \in D} |\hat{\mathbf{L}}_1(\theta, t) - \mathbf{L}_1(\theta, t)| > x \right) \le n^{-2}; \qquad (C9) $$

$$ \mathbb{P}_{\theta^*} \left( \sup_{\theta \in D} |\hat{\mathbf{M}}_0(\theta, t) - \mathbf{M}_0(\theta, t)| > x \right) \le n^{-2}, \qquad (C10) $$
<figure><img src="image_15.png" /><figcaption>Figure 10: FCR and selection frequency as a function of the nominal level $\alpha$. Three-component model $Q = 3$, $\pi_1 = \pi_2 = \pi_3 = 1/3$, $\Sigma_1 = \Sigma_2 = \Sigma_3 = I_2$, $\mu_1 = 0$, $\mu_2 = (\epsilon/\sqrt{2}, \epsilon/\sqrt{2})$, $\mu_3 = (0, 2\epsilon/\sqrt{2})$.</figcaption></figure>
for any $x \ge (1 + 2c)\sqrt{\frac{\log n}{n}}$ and provided that $(1 + 2c)\sqrt{\frac{\log n}{n}} \le 1$.

*Proof.* For a fixed $t \in (0, 1]$, let $\mathcal{F}_{L_0} = \{T(., \theta)1\{T(., \theta) \le t\}, \theta \in D\}$, $\mathcal{F}_{L_1} = \{1\{T(., \theta) \le t\}, \theta \in D\}$, and $\mathcal{F}_{M_0} = \{U(., \theta)1\{T(., \theta) \le t\}, \theta \in D\}$. We apply Lemmas 14 and 15 for $\xi_i = X_i$, $1 \le i \le n$, $b = 1$, $a = 0$ and for each $\mathcal{F} \in \{\mathcal{F}_{L_0}, \mathcal{F}_{L_1}, \mathcal{F}_{M_0}\}$ to get that the corresponding probability in (C8), (C9) and (C10) is at most $n^{-2}$ by taking

$$x \ge \sqrt{\frac{\log n}{n}} + 2\mathbb{E}\mathfrak{R}_n(\mathcal{F}),$$

where $\mathfrak{R}_n(\mathcal{F})$ denotes the Rademacher complexity of $\mathcal{F}$, see (C14). We now bound each $\mathfrak{R}_n(\mathcal{F})$ by using Lemma 12:

$$\begin{aligned} \mathbb{E}\mathfrak{R}_n(\mathcal{F}_{L_0}) &\le \mathbb{E}\mathfrak{R}_n(\mathcal{F}_{L_1}) + \mathbb{E}\mathfrak{R}_n(\{T(., \theta), \theta \in \Theta\}) \\ &\le \mathbb{E}\mathfrak{R}_n(\mathcal{F}_{L_1}) + \sum_{q=1}^{Q} \mathbb{E}\mathfrak{R}_n(\{\ell_q(., \theta), \theta \in \Theta\}); \end{aligned} \quad (C11)$$

$$\mathbb{E}\mathfrak{R}_n(\mathcal{F}_{L_1}) \le \sum_{q=1}^{Q} \mathbb{E}\mathfrak{R}_n(\{1\{\ell_q(., \theta) < 1-t\}, \theta \in \Theta\}); \quad (C12)$$

$$\mathbb{E}\mathfrak{R}_n(\mathcal{F}_{M_0}) \le \mathbb{E}\mathfrak{R}_n(\mathcal{F}_{L_1}) + \mathbb{E}\mathfrak{R}_n(\{U(., \theta), \theta \in \Theta\}),$$

where for (C11) and (C12), we used that $T(., \theta) = 1 - \max_q \ell_q(., \theta)$ and $1\{T(., \theta) \le t\} = 1 - \prod_{q=1}^Q 1\{\ell_q(., \theta) < 1 - t\}$ and the fact that the variables $\ell_q(X_i, \theta)$ are continuous by Assumption 1. Similarly, we have $U(., \theta) = \sum_{q=1}^Q \ell_q(., \theta^*) \prod_{k \ne q} 1\{\ell_q(., \theta) \ge \ell_k(., \theta)\}$. Hence, Lemma 12 once again entails that

$$\begin{aligned} \mathbb{E}\mathfrak{R}_n(\{U(., \theta), \theta \in \Theta\}) &\le \sum_{q=1}^{Q} \sum_{k=1, k \ne q}^{Q} \mathbb{E}\mathfrak{R}_n(\{1\{\ell_q(., \theta) - \ell_k(., \theta) \ge 0\}, \theta \in \Theta\}) \\ &\quad + \sum_{q=1}^{Q} \mathbb{E}\mathfrak{R}_n(\{\ell_q(., \theta), \theta \in \Theta\}). \end{aligned} \quad (C13)$$

To bound both $\mathbb{E}\mathfrak{R}_n(\{\ell_q(., \theta), \theta \in \Theta\})$ and $\mathbb{E}\mathfrak{R}_n(\{1\{\ell_q(., \theta) < 1 - t\}, \theta \in \Theta\})$, we use the results of Baraud (2016) (more specifically the proof of Theorem 1 therein), to obtain that they are bounded by

$$\sqrt{\mathcal{V} \log \frac{2en}{\mathcal{V}'} \frac{\sqrt{2}}{\sqrt{n}}} + 4\mathcal{V}' \log \frac{2en}{\mathcal{V}'} \frac{1}{n} \le 7\sqrt{\mathcal{V}' \frac{\log n}{n}},$$

provided that $\mathcal{V}'(\log n)/n \le 1$ and for $n \ge (2e)^3$. Similarly, $\mathbb{E}\mathfrak{R}_n(\{1\{\ell_q(., \theta) - \ell_k(., \theta) \ge 0\}, \theta \in \Theta\})$ is bounded by

$$\sqrt{\mathcal{V}' - \log \frac{2en}{\mathcal{V}' - \sqrt{2}} \frac{\sqrt{2}}{\sqrt{n}}} + 4\mathcal{V}' - \log \frac{2en}{\mathcal{V}' - n} \le 7\sqrt{\mathcal{V}' - \frac{\log n}{n}},$$

$\mathcal{V}'(\log n)/n \le 1$ and for $n \ge (2e)^3$. Combining this with what is above entails
<figure><img src="image_16.png" /><figcaption>Figure 11: FCR and selection frequency as a function of the nominal level $\alpha$. Unconstrained model with $Q = 2$, $d = 2$, $\epsilon = \sqrt{2}$, $n \in \{200, 1000\}$.</figcaption></figure>
$$
\begin{align*}
\mathbb{E}\mathfrak{R}_n(\mathcal{F}_{L_1}) &\le 7Q\sqrt{\mathcal{V} \frac{\log n}{n}} \\
\mathbb{E}\mathfrak{R}_n(\mathcal{F}_{L_0}) &\le 14Q\sqrt{\mathcal{V} \frac{\log n}{n}} \\
\mathbb{E}\mathfrak{R}_n(\mathcal{F}_{M_0}) &\le 14Q\sqrt{\mathcal{V} \frac{\log n}{n}} + 7Q^2\sqrt{\mathcal{V} - \frac{\log n}{n}}.
\end{align*}
$$

In particular, all expectations are upper-bounded by $c\sqrt{\frac{\log n}{n}}$, which leads to the result. ■

**Lemma 12.** If $\mathcal{F}$ is a class of indicator functions and $\mathcal{G}$ is a class of functions from $\mathbb{R}^d$
to $[0, 1]$, we have

$$
\begin{align*}
\mathbb{E}\mathfrak{R}_n(F \cdot G) &\le \mathbb{E}\mathfrak{R}_n(F) + \mathbb{E}\mathfrak{R}_n(G) \\
\mathbb{E}\mathfrak{R}_n(\max(F, G)) &\le \mathbb{E}\mathfrak{R}_n(F) + \mathbb{E}\mathfrak{R}_n(G),
\end{align*}
$$

where we denoted $F \cdot G = \{fg, f \in \mathcal{F}, g \in \mathcal{G}\}$ and $\max(F, G) = \{f \vee g, f \in \mathcal{F}, g \in \mathcal{G}\}$.

*Proof.* We have

$$
\begin{align*}
\mathbb{E}\mathfrak{R}_n(F \cdot G) &= \mathbb{E}\left( \sup_{f \in \mathcal{F}, g \in \mathcal{G}} \left| \sum_{i=1}^n \varepsilon_i f \cdot g(X_i) \right| \right) \\
&\le \mathbb{E}\left( \sup_{f \in \mathcal{F}, g \in \mathcal{G}} \left| \sum_{i=1}^n \varepsilon_i (f(X_i) + g(X_i)) \right| \right) \\
&\le \mathfrak{R}_n(F + G),
\end{align*}
$$

because $fg = (f + g - 1)_+ = 0.5(f + g - 1 + |f + g - 1|)$ and by applying the contraction lemma of Talagrand (see e.g. Lemma 5.7 in Mohri et al., 2012) with $x \mapsto 0.5(x - 1 + |x - 1|)$ which is 1-Lipchitz. Then we conclude by using the triangular inequality. For the max, we use $\max(f, g) = 0.5(f + g + |f - g|)$. ■

**Lemma 13.** Consider the case where $Q = 2$ and $\{F_u, u \in \mathcal{V}\}$ is an exponential family, i.e. there exists some functions $A, B, C, D$ such that $f(x, u) = \exp(A(u)^t B(x) - C(u) + D(x))$. Let $k$ be the dimension of the sufficient statistic vector $B(x)$. If $k \ge 3$, then $\mathcal{V}, \mathcal{V}_-$ defined by (16), (17) satisfy $\mathcal{V}, \mathcal{V}_- \le Qk(k+1)[3\log(k+1)) + 2(Q-1)]$. In addition, this bound still holds for $\mathcal{V}_-$ in the case $Q \ge 3$.

*Proof.* Let us first bound $\mathcal{V}$. Given that, for $Q = 2$, $\theta = (\pi_1, \pi_2, \phi_1, \phi_2)$, $\ell_1(x, \theta) \ge t$ is equivalent to $\pi_1 f(x, \phi_1)/\pi_2 f(x, \phi_2) \ge g(t)$ for some function $g$, we get that $\ell_1(x, \theta) \ge t$ if and only if $a(\theta)^t B(x) - b(\theta) \ge h(t)$ for some functions $a, b, h$. The set family is a subset of $\{\{x \in \mathbb{R}^d, a^t B(x) + b \ge 0\}, a \in \mathbb{R}^k, b \in \mathbb{R}\}$, whose VC dimension is bounded by $k(k+1)[3\log(k+1)) + 2]$ for $k \ge 3$, see Lemma 10.3 in Shalev-Shwartz and Ben-David (2014). By symmetry, this bound also holds for the VC dimension of $\{\ell_2(\cdot, \theta), \theta \in \Theta\}$. It follows that $\mathcal{V} \le Qk(k+1)[3\log(k+1)) + 2] + 2(Q-1)$ (see,
e.g., Exercice 3.24 in Mohri et al. (2012) on the VC dimension of the union of two classes with bounded VC dimension).

For $\mathcal{V}_-$, we have that for any $q \neq q' \in \{1, \dots, Q\}$, $\ell_q(x, \theta) - \ell_{q'}(x, \theta) \ge 0$ is equivalent to $\pi_q f(x, \phi_q)/\pi_{q'} f(x, \phi_{q'}) \ge 1$. The rest of the proof follows similarly as for $\mathcal{V}$. ■

**Lemma 14** (Talagrand's inequality, Theorem 5.3. in Massart (2007)). Let $\xi_1, \dots, \xi_n$ independent r.v., $\mathcal{F}$ a countable class of measurable functions s.t. $a \le f \le b$ for every $f \in \mathcal{F}$ for some real numbers $a \le b$, and $W = \sup_{f \in \mathcal{F}} |\sum_{i=1}^n f(\xi_i) - \mathbb{E}(f(\xi_i))|$. Then, for any $x > 0$,

$$ \mathbb{P}(W - \mathbb{E}(W) \ge x) \le e^{-\frac{2x^2}{n(b-a)^2}}. $$

**Lemma 15** (Rademacher complexity bound, see, e.g., Lemma 1 in Baraud (2016)). In the setting of Lemma 14 (and with the notation therein), we have

$$ \mathbb{E}(W) \le 2\mathfrak{R}_n(\mathcal{F}), $$

where

$$ \mathfrak{R}_n(\mathcal{F}) = \sup_{f \in \mathcal{F}} \left| \sum_{i=1}^{n} \varepsilon_i f(\xi_i) \right| \quad (C14) $$

is the Rademacher complexity of the class $\mathcal{F}$ (with $\varepsilon_1, \dots, \varepsilon_n$ being i.i.d. random signs).

# APPENDIX D. AUXILIARY RESULTS FOR THE GAUSSIAN CASE

## D1 Convergence rate for parameter estimation

The following result presents two situations where the parameter of a Gaussian mixture model can be consistently estimated, with an explicit rate.

**Proposition 1.** Consider the mixture model (Section 2.1) in the d-multivariate Gaussian case with true parameter $\theta^* = (\pi^*, \phi^*)$, where $\phi_q^* = (\mu_q^*, \Sigma_q^*)$, $1 \le q \le Q$. Then $\eta(\epsilon, \theta^*)$ defined by (18) is such that $\eta(\epsilon_n, \theta^*) \le 1/n$ for $\epsilon_n \ge C\sqrt{\log n/n}$, where $C > 0$ is a sufficiently large constant, in two following situations:

(i) $\hat{\theta}$ is the constrained MLE, that is, computed for $\phi_q = (\mu_q, \Sigma_q) \in \mathcal{U}$ with constrained parameter space $\mathcal{U} = [-a_n, a_n]^d \times \{\Sigma \in S_d^{++}, \underline{\lambda} \le \lambda_1(\Sigma) \le \lambda_d(\Sigma) \le \bar{\lambda}\}^1$ where $a_n \le L(\log n)^\gamma$ for some $L, \gamma > 0$ and $S_d^{++}$ denotes the space of positive definite matrices, with $\underline{\lambda}, \bar{\lambda} > 0$. In that case, $C$ only depends on $\theta^*$ and $L, \gamma, \underline{\lambda}, \bar{\lambda}$.

(ii) $\hat{\theta}$ is the estimator coming from EM algorithm (when the iteration number is infinite) for an initialization $\mu_1^{(0)}, \mu_2^{(0)}$ such that $\| (\mu_1^{(0)} - \mu_2^{(0)}) - (\mu_1 - \mu_2) \| \le \Delta/4$, where $\Delta = \|\mu_1 - \mu_2\|_2$ is the separation between the true means. Here, we consider an homoscedastic model with $\Sigma_1 = \Sigma_2 = \Sigma = vI_d$ with known $v$. The conclusion applies if the signal-to-noise ratio $\Delta/v$ is large enough, and for a constant $C$ of the form $c(v, \Delta)\sqrt{d}$.
Proof. Since case (ii) is a direct application of Balakrishnan et al. (2017), we focus in what follows on proving case (i), by revisiting the result of Ho and Nguyen (2016). First, in the considered model, any mixture can be defined in terms of {$f_u, u \in \mathcal{U}$} and a discrete mixing measure $G = \sum_{q=1}^{Q} \pi_q \delta_{\phi_q}$ with Q support points, as $\sum_{q=1}^{Q} \pi_q f_{\phi_q} = \int f_u(x) dG(u)$. As shown by Ho and Nguyen (2016), the convergence of mixture model parameters can be measured in terms of a Wasserstein distance on the space of mixing measures. Let $G_1 = \sum_{q=1}^{Q} \pi_q^1 \delta_{\phi_q^1}$ and $G_2 = \sum_{q=1}^{Q} \pi_q^2 \delta_{\phi_q^2}$ be two discrete probability measures on some parameter space, which is equipped with metric $||.||$. The Wasserstein distance of order 1 between $G_1$ and $G_2$ is given by

$$W_1(G_1, G_2) = \inf_{p} \sum_{q,l} p_{q,l} ||\phi_q^1 - \phi_l^2||$$

where the infimum is over all couplings ($p_{q,l}$) $1 \le q,l \le Q \in [0, 1]^{Q \times Q}$ such that $\sum_l p_{q,l} = \pi_q^1$ and $\sum_q p_{q,l} = \pi_l^2$. Let $G^*$, $\hat{G}_n$ denote the true mixing measure and the mixing measure that corresponds to the restricted MLE considered here, respectively. Theorem 4.2. in Ho and Nguyen (2016) implies that, with the notation of Ho and Nguyen (2016), for any $\epsilon_n \ge (\sqrt{C_1/c})\delta_n$, and $\delta_n \le C\sqrt{\log n/n}$, we have $\mathbb{P}_{\theta^*}(W_1(\hat{G}_n, G^*) \ge (c/C_1)\epsilon_n) \le ce^{-n\epsilon_n^2}$. We apply this relation for $\epsilon_n = \max((\sqrt{C_1/c})\delta_n, \sqrt{\log cn}/n)$. In that case, we have still $\epsilon_n$ of order $\sqrt{\log n/n}$ and the upper-bound is at most $1/n$. On the other hand, if we have a convergence rate in terms of $W_1$, then we have convergence of the mixture model parameters in terms of $||.||$ at the same rate, see Lemma 16. This concludes the proof. ■

**Lemma 16.** Let $G_n = \sum_{q=1}^{Q} \pi_q^n \delta_{\phi_q^n}$ be a sequence of discrete probability measures on $\mathcal{U}$, and let $G^*$, $W_1$ be defined as in the proof of Proposition 1. There exists a constant $C$ only depending on $G^*$ such that if $W_1(G_n, G^*) \to 0$, then for sufficiently large $n$,

$$W_1(G_n, G^*) \ge C \min_{\sigma \in [Q]} ||\theta_n^\sigma - \theta^*||.$$

Proof. In what follows, we let {$p_{q,l}^n$} denote the corresponding probabilities of the optimal coupling for the pair $(G_n, G^*)$. We start by showing that $(\phi_q^n)_q \to (\phi_q^*)_q$ in $||.||$ up to a permutation of the labels. Let $\sigma^n$ the permutation of the labels such that $||\phi_q^n - \phi_l^*|| \ge ||\phi_{\sigma^n(l)}^n - \phi_l^*||$ for all $q, l \in \{1, \dots, Q\}$. Then, by definition,

$$W_1(G_n, G^*) \ge \sum_{1 \le q,l \le Q} p_{q,l}^n ||\phi_{\sigma^n(l)}^n - \phi_l^*|| \\ = \sum_l \pi_l^* ||\phi_{\sigma^n(l)}^n - \phi_l^*||.$$

It follows that each $||\phi_{\sigma^n(l)}^n - \phi_l^*||$ must converge to zero. Since $(\phi_q^n)_q \to (\phi_q^*)_q$ up to a permutation of the labels, without loss of generality we can assume that $\phi_q^n \to \phi_q^*$ for all $q$. Let $\Delta\phi_q^n := \phi_q^n - \phi_q^*$ and $\Delta\pi_q^n := \pi_q^n - \pi_q^*$. Write $W_1(G_n, G^*)$ as

$$W_1(G_n, G^*) = \sum_q p_{qq}^n ||\Delta\phi_q^n|| + \sum_{q \ne l} p_{ql}^n ||\phi_q^n - \phi_l^*||$$
Define $C_{ql} = ||\phi_q^* - \phi_l^*||$ and $C = \min_{q \neq l} C_{ql} > 0$. It follows from the convergence of $\phi^n$ that for $q \neq l$, $||\phi_q^n - \phi_l^n|| \ge C/2$ for sufficiently large $n$. Thus,

$$W_1(G_n, G^*) \ge \frac{C}{2} \sum_{q \neq l} p_{ql}^n$$

We deduce that $\sum_{q \neq l} p_{ql}^n \to 0$. As a result, $p_{qq}^n = \pi_q^* - \sum_{l \neq q} p_{lq}^n \to \pi_q^*$, and so, $p_{qq}^n \ge (1/2)\pi_{min}^* := \min_l \pi_l^*$ for sufficiently large $n$. On the other hand, $\sum_{q \neq l} p_{ql}^n = \sum_q \pi_q^n - p_{qq}^n = \sum_q \pi_q^* - p_{qq}^n$ where $p_{qq}^n \le \min(\pi_q^n, \pi_q^*)$. Thus, $\sum_{q \neq l} p_{ql}^n \ge \sum_q \pi_q^n - \min(\pi_q^n, \pi_q^*) = \sum_{q, \pi_q^n \ge \pi_q^*} \pi_q^n - \pi_q^* = \sum_{q, \pi_q^n \ge \pi_q^*} |\pi_q^n - \pi_q^*|$ and similarly we have that $\sum_{q \neq l} p_{ql}^n \ge \sum_{q, \pi_q^n \ge \pi_q^*} |\pi_q^n - \pi_q^*|$. It follows that $2 \sum_{q \neq l} p_{ql}^n \ge \sum_q |\pi_q^n - \pi_q^*|$. Therefore, for sufficiently large $n$,

$$W_1(G_n, G^*) \ge \frac{1}{2} \pi_{min}^* \sum_q ||\Delta\phi_q^n|| + \frac{C}{4} \sum_q |\Delta\pi_q^n|$$

This gives the result.

<figure><img src="image_9.png" /></figure>

## D2 Gaussian computations

The following lemma holds.

**Lemma 17.** Let us consider the multivariate Gaussian case where $\phi_q = (\mu_q, \Sigma_q)$, $1 \le q \le Q$, with $Q = 2$, $\Sigma_1 = \Sigma_2$ is an invertible covariance matrix and $\mu_1$ and $\mu_2$ are two different vectors of $\mathbb{R}^d$. Then Assumptions 1, 2 and 3 hold true for $\alpha_c = 0$ and for a level $\alpha \in (0, \bar{\alpha}) \setminus \mathcal{E}$ for $\mathcal{E}$ a set of Lebesgue measure 0.

*Proof.* Let us first prove that $\ell_q(X, \theta)$ is a continuous random variable under $\mathbb{P}_{\theta^*}$ (this is established below without assuming $\Sigma_1 = \Sigma_2$ for the sake of generality). We have

$$ \begin{aligned} \mathbb{P}_{\theta^*}(\ell_1(X, \theta) = t) \\ &= \mathbb{P}_{\theta^*}(f_{\phi_1}(X)/f_{\phi_2}(X) = t\pi_2/\pi_1) \\ &= \mathbb{P}_{\theta^*}((X - \mu_1)^t \Sigma_1^{-1}(X - \mu_1) - (X - \mu_2)^t \Sigma_2^{-1}(X - \mu_2) = -2 \log(t\pi_2/\pi_1) - \log(|\Sigma_1|/|\Sigma_2|)). \end{aligned} $$

Now,

$$ \begin{aligned} & (X - \mu_1)^t \Sigma_1^{-1} (X - \mu_1) - (X - \mu_2)^t \Sigma_2^{-1} (X - \mu_2) \\ &= (X - \mu_1)^t \Sigma_1^{-1} (X - \mu_1) - (X - \mu_1)^t \Sigma_2^{-1} (X - \mu_2) - (\mu_1 - \mu_2)^t \Sigma_2^{-1} (X - \mu_2) \\ &= (X - \mu_1)^t (\Sigma_1^{-1} - \Sigma_2^{-1}) (X - \mu_1) - (X - \mu_1)^t \Sigma_2^{-1} (\mu_1 - \mu_2) - (\mu_1 - \mu_2)^t \Sigma_2^{-1} (X - \mu_2) \\ &= (X - \mu_1)^t (\Sigma_1^{-1} - \Sigma_2^{-1}) (X - \mu_1) - (\mu_1 - \mu_2)^t \Sigma_2^{-1} (2X - \mu_2 - \mu_1). \end{aligned} $$

Since the real matrix $\Sigma_1^{-1} - \Sigma_2^{-1}$ is symmetric, we can diagonalize it and we end up with a subset of $\mathbb{R}^d$ of the form

$$ \left\{ y \in \mathbb{R}^d : \sum_{j=1}^{d} (\alpha_j y_j^2 + \beta_j y_j) + \gamma = 0 \right\}, $$
for some real parameters $\alpha_j, \beta_j, \gamma$. The result follows because this set has a Lebesgue measure equal to 0 in any case.

Now, since $\Sigma_1 = \Sigma_2 = \Sigma$, we have for all $t \in (0, 1)$,

$$ \begin{aligned} \{T(X, \theta) > t\} &= \left\{ \forall q \in \{1, \dots, Q\}, \pi_q f_{\phi_q}(X) < (1-t) \sum_{\ell=1}^{Q} \pi_{\ell} f_{\phi_{\ell}}(X) \right\} \\ &= \left\{ \pi_1 f_{\phi_1}(X) < (1/t-1) \pi_2 f_{\phi_2}(X) \right\} \cap \left\{ \pi_2 f_{\phi_2}(X) < (1/t-1) \pi_1 f_{\phi_1}(X) \right\} \\ &= \left\{ (1/t-1)^{-1} < \frac{\pi_1 f_{\phi_1}(X)}{\pi_2 f_{\phi_2}(X)} < (1/t-1) \right\}. \end{aligned} $$

Applying $2 \log(\cdot)$ on each part of the relation, we obtain

$$ \{T(X, \theta) > t\} = \{-2 \log(1/t - 1) < a^t X + b < 2 \log(1/t - 1)\}, $$

for

$$ \begin{aligned} a &= a(\theta) = 2\Sigma^{-1}(\mu_1 - \mu_2) \in \mathbb{R}^d \setminus \{0\} \\ b &= b(\theta) = -(\mu_1 - \mu_2)^t \Sigma^{-1}(\mu_1 + \mu_2) + 2 \log(\pi_1 / \pi_2) \in \mathbb{R}^d. \end{aligned} $$

Since under $P_{\theta^*}$ we have $X \sim \pi_1^* \mathcal{N}(\mu_1^*, \Sigma^*) + \pi_2^* \mathcal{N}(\mu_2^*, \Sigma^*)$, we have $a^t X + b \sim \pi_1^* \mathcal{N}(a^t \mu_1^* + b, a^t \Sigma^* a) + \pi_2^* \mathcal{N}(a^t \mu_2^* + b, a^t \Sigma^* a)$. This yields for all $t \in (0, 1)$,

$$ \begin{aligned} \mathbb{P}_{\theta^*}(T(X, \theta) > t) = \pi_1 & \left[ \Phi \left( \frac{2 \log(1/t - 1) - a^t \mu_1^* - b}{(a^t \Sigma^* a)^{1/2}} \right) \right. \\ & \left. - \Phi \left( \frac{-2 \log(1/t - 1) - a^t \mu_1^* - b}{(a^t \Sigma^* a)^{1/2}} \right) \right] \\ & + \pi_2 \left[ \Phi \left( \frac{2 \log(1/t - 1) - a^t \mu_2^* - b}{(a^t \Sigma^* a)^{1/2}} \right) \right. \\ & \left. - \Phi \left( \frac{-2 \log(1/t - 1) - a^t \mu_2^* - b}{(a^t \Sigma^* a)^{1/2}} \right) \right]. \end{aligned} \quad (D1) $$

A direct consequence is that for all $t \in (0, 1)$, we have $\mathbb{P}_{\theta^*}(T(X, \theta) > t) < 1$, that is, $\mathbb{P}_{\theta^*}(T(X, \theta) \le t) = \mathbb{P}_{\theta^*}(T(X, \theta) < t) > 0$. Hence, $\alpha_c$ defined in (14) is equal to zero. Moreover, from (D1), we clearly have that $t \in (0, 1) \mapsto \mathbb{P}_{\theta^*}(T(X, \theta) > t)$ is decreasing, so that $t \in (0, 1) \mapsto \mathbb{P}_{\theta^*}(T(X, \theta) \le t)$ is increasing. This proves that Assumption 1 holds in that case.

Let us now check Assumptions 2 and 3. Assumptions 2 and 3 (i) follow from Result 2.1 in Melnykov (2013).

As for Assumption 3 (ii), from (D1), we only have to show that the function $t \in (0, 1) \mapsto \frac{\partial}{\partial t} \Phi(\frac{\log(1/t-1)-a^t}{\beta^*})$ is uniformly bounded by some constant $C = C(\alpha^*, \beta^*)$, for any $\alpha^* \in \mathbb{R}$ and $\beta^* > 0$. A straightforward calculation leads to the following: for all $t \in (0, 1)$,
$$ \left| \frac{\partial}{\partial t} \Phi \left( \frac{\log(1/t - 1) - \alpha^*}{\beta^*} \right) \right| = \frac{e^{-\left( \frac{\log(1/t-1)-\alpha^*}{\beta^*} \right)^2/2}}{\beta^* \sqrt{2\pi}} \frac{1}{t(1-t)}. \quad (D2) $$

Consider now $t_0 = t_0(\alpha^*, \beta^*) \in (0, 1/2)$ such that $(\frac{\log(1/t-1)-\alpha^*}{\beta^*})^2 \ge 2 \log(1/t)$ for all $t \in (0, t_0)$. It is clear that the right-hand-side of (D2) is upper-bounded by $\frac{1}{\beta^* \sqrt{2\pi}(1-t_0)}$ on $t \in (0, t_0)$. Similarly, let $t_1 = t_1(\alpha^*, \beta^*) \in (1/2, 1)$ such that $(\frac{\log(1/t-1)-\alpha^*}{\beta^*})^2 \ge 2 \log(1/(1-t))$ for all $t \in (t_1, 1)$. It is clear that the right-hand-side of (D2) is upper-bounded by $\frac{1}{\beta^* \sqrt{2\pi t_1}}$ on $t \in (t_1, 1)$. Finally, for $t \in [t_0, t_1]$, the upper-bound $\frac{1}{\beta^* \sqrt{2\pi t_0(1-t_1)}}$ is valid. This proves that Assumption 3 (ii) holds.

Let us now finally turn to Assumption 3 (iii). Lemma 8 ensures that $t \in (0, t^*(\bar{\alpha})) \mapsto mFSR_i^*$ is continuous increasing. Hence, $t^* : \beta \in (0, \bar{\alpha}) \mapsto t^*(\beta)$ defined in (13) is the inverse of this function and is also continuous increasing. It is therefore differentiable almost everywhere in $(0, \bar{\alpha})$, so everywhere in $(0, \bar{\alpha}) \setminus \mathcal{E}$ where $\mathcal{E}$ is a set of Lebesgue measure 0. By taking $\alpha$ in $(0, \bar{\alpha}) \setminus \mathcal{E}$, this ensures that $t^*$ is differentiable in $\alpha$ and thus that Assumption 3 (iii) holds. ■

**Lemma 18.** In the multivariate Gaussian case with $Q = 2$ and $\Sigma_1 = \Sigma_2$, we have that $\mathcal{V} \le 2d + 4$ and $\mathcal{V}_- \le 2d + 4$.

*Proof.* In that case, we have that (see the proof of Lemma 17)

$$ \{ \ell_q(x, \theta) \le u, x \in \mathbb{R}^d \} = \{ a_{\theta}^t x + b_{\theta} \ge g(u), x \in \mathbb{R}^d \}. $$

Since the VC dimension of the vector space of real-valued affine functions is bounded by $d+1$ (see, e.g., Exercice 3.19 in Mohri et al., 2012). We obtain the result by applying the usual bound on the VC dimension of the union of two classes with bounded VC dimension (see, e.g., Exercice 3.24 in Mohri et al., 2012). ■

# APPENDIX E. ADDITIONAL NUMERICAL EXPERIMENTS

## E1 Additional bootstrap procedure

Here we evaluate an additional bootstrap procedure, that is based on the work of O'Hagan et al. (2019). O'Hagan et al. (2019) investigated different sampling methods for estimating standard errors of model parameters in the context of Gaussian mixture models. In particular, they considered the case where some of the clusters are small and/or overlapping and they proposed a so-called weighted likelihood bootstrap approach that is shown to be effective in that case. The method consists of sampling a weight vector $(w_i^b)_{1 \le i \le n}$ (e.g. according to a uniform Dirichlet distribution), to then fit a bootstrap parameter estimate $\hat{\theta}^b$ by using the EM algorithm with the weighted log-likelihood $\sum_{i=1}^n \sum_{q=1}^Q 1_{Z_i=q} w_i^b(\log \pi_q + \log f_{\mu_q, \Sigma_q}(X_i))$. We adapt this approach by replacing $\hat{Z}_i^{\text{PI}}(\mathbf{X}^b)$, $\hat{S}_\alpha^{\text{PI}}(\mathbf{X}^b)$ in (10) with the partition and selection $\hat{Z}_i^{\text{PI}}(\mathbf{X})$, $\hat{S}_\alpha^{\text{PI}}(\mathbf{X})$ of the plug-in procedure computed with $\hat{\theta}^b$ and the original sample $\mathbf{X}$. Formally, by denoting $\hat{S}_\alpha^{\text{PI}}(\mathbf{X}) = \Psi_1(\mathbf{X}, \hat{\theta}(\mathbf{X}))$ and $\hat{Z}_i^{\text{PI}}(\mathbf{X}) = \Psi_2(\mathbf{X}, \hat{\theta}(\mathbf{X}))$ for some functions $\Psi_1, \Psi_2$, the weighted nonparametric bootstrap estimate of the FSR is given by
<figure><img src="image_10.png" /><figcaption>FIGURE E1 FSR (left panel) and selection frequency (right panel) as a function of: (a) the mean separation; (b) the sample size *n*; (c) the nominal level *α*. Diagonal covariances setting with *Q* = 2, *d* = 2. Default settings are: *n* = 200, *α* = 0.1, *ε* = √2.</figcaption></figure>

$$ \widehat{\text{FSR}}_{\alpha}^{w,B} := \frac{1}{B} \sum_{b=1}^{B} \frac{\min_{\sigma \in \{Q\}} \frac{\sum_{i=1}^{n} \{1 - \ell_{\sigma}(\Psi_2(\mathbf{X}, \hat{\theta}^b))(X_i, \hat{\theta}(\mathbf{X}))\} \mathbb{1}\{i \in \Psi_1(\mathbf{X}, \hat{\theta}^b)\}}{\max(|\Psi_1(\mathbf{X}, \hat{\theta}^b)|, 1)}, $$

This is markedly different than the bootstrap approaches in (10) because the sample **X** is fixed once for all when computing the bootstrap estimates of the FSR.

We evaluate the procedure by reproducing the experiments of Figure 3, in which we considered a balanced Gaussian mixture model with *d* = 2, *Q* = 2 and the covariance matrices were assumed to be diagonal in the EM algorithm: Figure E1 displays the FSR and the selection frequency as a function of the mean separation *ε* (Figure E1a) and the nominal level *α* (Figure E1c). We observe that across all settings, the FSR of the weighted nonparametric bootstrap is between that of the plug-in and the parametric bootstrap. In addition, we evaluate the procedure in the presence of a very small component, considering as before a Gaussian mixture model with *d* = 2, *Q* = 2 and diagonal covariance matrices, with mixture proportions set to *π* = (0.1, 0.9): Figure E2 displays the FSR and the selection frequency for varying *α* and *n* ∈ {200, 1000}. In this case the nonparametric bootstrap shows poor performance, which is consistent with the observations of O'Hagan et al. (2019). The parametric bootstrap still displays an FCR level close to the nominal level, which we explain by the accuracy of the specified model. The weighted nonparametric bootstrap gives results that are very close to the plug-in, as in the previous experiment, and so lacks
<figure><img src="image_11.png" /><figcaption>**FIGURE E2** FSR (left panel) and selection frequency (right panel) as a function of the nominal level $\alpha$. Gaussian mixture model with $Q = 2$, $\pi = (0.1, 0.9)$, $d = 2$, $\Sigma_1 = \Sigma_2 = I_2$, $\mu_1 = \mu_2 = 0_2$ and $\mu_2 = (\epsilon/\sqrt{d}, \epsilon/\sqrt{d})$, $\epsilon = 2$.</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>**FIGURE E3** FSR (left panel) and selection frequency (right panel) as a function of the nominal level $\alpha$. Here, $P_0$, is a t-mixture model with $Q = 2$, $\pi_1 = \pi_2 = 1/2$, $d = 2$, $\Sigma_1 = \Sigma_2 = I_2$, $\mu_1 = (0, 0)$ and $\mu_2 = (\sqrt{2}, \sqrt{2})$, and the degrees of freedom of each component is set to 4.</figcaption></figure>

FCR control. We conjecture that this is due to evaluating the plug-in procedure on $\mathbf{X}$ in the computation of the FSR estimate instead of bootstrap samples $\mathbf{X}^b \sim \hat{P}$, resulting in reducing the variance of the FSP estimate and leading to an FSR estimate that is closer to the FSR of the plug-in comparison to the other bootstrap procedures. We conclude that integrating the approach of O'Hagan et al. (2019) with our procedure presents nontrivial challenges that require further consideration and which we leave for future work.

## E2 t-mixture models

In this section, we evaluate our procedures using *t*-mixture models, both on data generated from *t*-mixtures as well as on Gaussian mixtures. The *t*-mixture model is of particular interest in our context as it is appropriate for modeling data containing observations with longer than normal tails or atypical observations leading to overlapping clusters (Peel & McLachlan, 2000). In all experiments, the *t*-mixture is fit via the EM algorithm for *t*-mixtures (Peel & McLachlan, 2000) provided by the Python package `studenttmixture` (Parkinson, 2018) and no constraints are put on the parameters in the estimation procedure. In particular, all parameters are estimated—including degrees of freedom—without any assumption on the covariance structure.

We start with the case where the data is itself generated from a *t*-mixture. We consider $Q = 2$, $d = 2$, $\pi_1 = \pi_2 = 1/2$, $\Sigma_1 = \Sigma_2 = I_2$, $\mu_1 = (0, 0)$ and $\mu_2 = (\sqrt{2}, \sqrt{2})$, and the degrees of freedom of each component is 4. Figure E3 displays the FSR and the selection frequency for varying $\alpha$.
<figure><img src="image_13.png" /><figcaption>**FIGURE E4** FSR (left panel) and selection frequency (right panel) as a function of the nominal level $\alpha$. Here, $P_{\theta^*}$ is a Gaussian mixture model with $Q = 2$, $\pi_1 = \pi_2 = 1/2$, $d = 2$, $\Sigma_1 = \Sigma_2 = I_2$, $\mu_1 = (0, 0)$ and $\mu_2 = (1, 1)$. The sample size is $n = 200$.</figcaption></figure>

and $n \in \{200, 1000\}$. The conclusions are qualitatively the same as in Section 5.1: the plug-in procedure has an FSR that exceeds the nominal level when the sample size is too small, whereas the bootstrap procedures allow for a more robust FSR control, the nonparametric one being the more robust and controlling the FSR in all settings.

Next, we evaluate the procedures with the use of *t*-mixture modeling on Gaussian-generated data. Specifically, $P_{\theta^*}$ is a Gaussian mixture model with $Q = 2$, $\pi_1 = \pi_2 = 1/2$, $\Sigma_1 = \Sigma_2 = I_2$, $\mu_1 = (0, 0)$ and $\mu_2 = (1, 1)$. However, concerning the specification of $P_{\theta^*}$ we choose to model the generated data as a *t*-mixture. The idea is to fit a model that is more robust for parameter estimation in the context of overlapping clusters. Compared to Gaussian mixtures, Student's *t*-distributions are less concentrated and thus produce estimates of the posterior probabilities of class memberships that are less extreme, which is favorable for our selection procedures. Figure E4a displays the FSR and the selection frequency for varying $\alpha$ and $n = 200$. For comparison, Figure E4b displays the results obtained when using Gaussian mixture modeling, fit via the EM algorithm and with no parameter constraints. As expected, the use of *t*-mixture modeling leads to a lower FSR for all procedures. In particular, the FSR of the plug-in procedure and the parametric bootstrap procedure are closer to the nominal level $\alpha$. In the case of the nonparametric bootstrap, since the procedure is already controlling the FSR when using Gaussian mixtures, the use of *t*-mixture modeling makes the procedure more conservative, but only very slightly.