SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \frac {\partial (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) ^ {\top}}{\partial \boldsymbol {v} _ {j , i}} \\ = \left( \begin{array}{c c c c c} 0 & \dots & \frac {\partial (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) _ {1 , j} ^ {\top}}{\partial \boldsymbol {v} _ {j , i}} & \dots & 0 \\ \vdots & \ddots & \vdots & \ddots & \vdots \\ \frac {\partial (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) _ {j , 1} ^ {\top}}{\partial \boldsymbol {v} _ {j , i}} & \dots & \frac {\partial (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) _ {j , j} ^ {\top}}{\partial \boldsymbol {v} _ {j , i}} & \dots & \frac {\partial (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) _ {j , \ell} ^ {\top}}{\partial \boldsymbol {v} _ {j , i}} \\ \vdots & \ddots & \vdots & \ddots & \vdots \\ 0 & \dots & \frac {\partial (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) _ {\ell , j} ^ {\top}}{\partial \boldsymbol {v} _ {j , i}} & \dots & 0 \end{array} \right). \\ \end{array}
$$
Here,  $(\hat{\pmb{x}}_r - \pmb{c}_{\hat{X}})(\hat{\pmb{x}}_r - \pmb{c}_{\hat{X}})_{j,t}^{\top}$  denotes the element in  $j$ -th row and  $t$ -th column, and we have
$$
\begin{array}{l} (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) _ {j, t} ^ {\top} \\ = \left(\frac {m - 1}{m} \kappa_ {t} (\boldsymbol {x} _ {r}, \boldsymbol {v} _ {j}) - \frac {1}{m} \sum_ {s \neq r} ^ {m} \kappa_ {t} (\boldsymbol {x} _ {s}, \boldsymbol {v} _ {j})\right) \left(\frac {m - 1}{m} \kappa_ {t} (\boldsymbol {x} _ {r}, \boldsymbol {v} _ {t}) - \frac {1}{m} \sum_ {s \neq r} ^ {m} \kappa_ {t} (\boldsymbol {x} _ {s}, \boldsymbol {v} _ {t})\right), \\ \end{array}
$$
$$
\begin{array}{l} \frac {\partial (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) _ {j , t} ^ {\top}}{\partial \boldsymbol {v} _ {j , i}} \\ { = } { \left( \frac { m - 1 } { m } \kappa _ { j } ( \pmb { x } _ { r } , \pmb { v } _ { t } ) - \frac { 1 } { m } \sum _ { s \neq r } ^ { m } \kappa _ { j } ( \pmb { x } _ { s } , \pmb { v } _ { t } ) \right) \left( \frac { m - 1 } { m } \frac { \partial \kappa _ { j } ( \pmb { x } _ { r } , \pmb { v } _ { j } ) } { \partial \pmb { v } _ { j , i } } - \frac { 1 } { m } \sum _ { s \neq r } ^ { m } \frac { \partial \kappa _ { j } ( \pmb { x } _ { s } , \pmb { v } _ { j } ) } { \partial \pmb { v } _ { j , i } } \right) . } \\ \end{array}
$$
We similarly have
$$
\begin{array}{l} \frac {\partial (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) _ {j , j} ^ {\top}}{\partial \boldsymbol {v} _ {j , i}} \\ = 2 \left(\frac {m - 1}{m} \kappa_ {j} (\boldsymbol {x} _ {r}, \boldsymbol {v} _ {j}) - \frac {1}{m} \sum_ {s \neq r} ^ {m} \kappa_ {j} (\boldsymbol {x} _ {s}, \boldsymbol {v} _ {j})\right) \left(\frac {m - 1}{m} \frac {\partial \kappa_ {j} (\boldsymbol {x} _ {r} , \boldsymbol {v} _ {j})}{\partial \boldsymbol {v} _ {j , i}} - \frac {1}{m} \sum_ {s = r} ^ {m} \frac {\partial \kappa_ {j} (\boldsymbol {x} _ {s} , \boldsymbol {v} _ {j})}{\partial \boldsymbol {v} _ {j , i}}\right). \\ \end{array}
$$
For gamma parameter  $\gamma_{j}$  with  $j\in [\ell ]$  , we have
$$
\nabla_ {\gamma_ {j}} \mathcal {T} (\hat {X}, \hat {Y}) = \frac {\partial \mathcal {T} (\hat {X} , \hat {Y})}{\partial \boldsymbol {c} _ {\hat {X}}} \frac {\partial \boldsymbol {c} _ {\hat {X}}}{\partial \gamma_ {j}} + \frac {\partial \mathcal {T} (\hat {X} , \hat {Y})}{\partial \boldsymbol {c} _ {\hat {Y}}} \frac {\partial \boldsymbol {c} _ {\hat {Y}}}{\partial \gamma_ {j}} + \operatorname {T r} \left[ \frac {\partial \mathcal {T} (\hat {X} , \hat {Y})}{\partial \Sigma_ {\hat {X} , \hat {Y}}} \frac {\partial \Sigma_ {\hat {X} , \hat {Y}}}{\partial \gamma_ {j}} \right]. \tag {34}
$$
$$
\frac {\partial \boldsymbol {c} _ {\hat {X}}}{\partial \gamma_ {j}} = \frac {1}{m} \sum_ {r = 1} ^ {m} \frac {\partial \hat {\boldsymbol {x}} _ {r}}{\partial \gamma_ {j}} \quad \text {w i t h} \quad \frac {\partial \hat {\boldsymbol {x}} _ {r}}{\partial \gamma_ {j}} = \left(0, \dots , \frac {\partial \kappa_ {j} (\hat {\boldsymbol {x}} _ {r} , \boldsymbol {v} _ {j})}{\partial \gamma_ {j}}, \dots , 0\right) ^ {\top},
$$
$$
\begin{array}{l} \frac {\partial \kappa_ {j} (\pmb {x} _ {r} , \pmb {v} _ {j})}{\partial \gamma_ {j}} = \frac {\partial \exp \left(- (\pmb {x} _ {r} - \pmb {v} _ {j}) ^ {\top} M _ {j} (\pmb {x} _ {r} - \pmb {v} _ {j}) / 2 \gamma_ {j} ^ {2}\right)}{\partial \gamma_ {j}} \\ { = } { \kappa _ { j } ( \hat { \boldsymbol { x } } _ { r } , \boldsymbol { v } _ { j } ) ( \boldsymbol { x } _ { r } - \boldsymbol { v } _ { j } ) ^ { \top } M _ { j } ( \boldsymbol { x } _ { r } - \boldsymbol { v } _ { j } ) \gamma _ { j } ^ { - 3 } . } \\ \end{array}
$$
We similarly calculate
$$
\frac {\partial \Sigma_ {\hat {X} , \hat {Y}}}{\partial \gamma_ {j}} = \frac {1}{m + n - 2} \left(\sum_ {r = 1} ^ {m} \frac {\partial (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) ^ {\top}}{\partial \gamma_ {j}} + \sum_ {r = 1} ^ {n} \frac {\partial (\hat {\boldsymbol {y}} _ {r} - \boldsymbol {c} _ {\hat {Y}}) (\hat {\boldsymbol {y}} _ {r} - \boldsymbol {c} _ {\hat {Y}}) ^ {\top}}{\partial \gamma_ {j}}\right),
$$
$$
\begin{array}{l} \frac {\partial (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) ^ {\top}}{\partial \gamma_ {j}} \\ = \left( \begin{array}{c c c c c} 0 & \dots & \frac {\partial (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\tilde {X}}) (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\tilde {X}}) _ {1 , j} ^ {\top}}{\partial \gamma_ {j}} & \dots & 0 \\ \vdots & \ddots & \vdots & \ddots & \vdots \\ \frac {\partial (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\tilde {X}}) (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\tilde {X}}) _ {j , 1} ^ {\top}}{\partial \gamma_ {j}} & \dots & \frac {\partial (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\tilde {X}}) (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\tilde {X}}) _ {j , j} ^ {\top}}{\partial \gamma_ {j}} & \dots & \frac {\partial (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\tilde {X}}) (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\tilde {X}}) _ {j , \ell} ^ {\top}}{\partial \gamma_ {j}} \\ \vdots & \ddots & \vdots & \ddots & \vdots \\ 0 & \dots & \frac {\partial (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\tilde {X}}) (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\tilde {X}}) _ {\ell , j} ^ {\top}}{\partial \gamma_ {j}} & \dots & 0 \end{array} \right), \\ \end{array}
$$
$$
\begin{array}{l} \frac {\partial (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) _ {j , t} ^ {\top}}{\partial \gamma_ {j}} \\ = \left(\frac {m - 1}{m} \kappa_ {j} (\boldsymbol {x} _ {r}, \boldsymbol {v} _ {t}) - \frac {1}{m} \sum_ {s \neq r} ^ {m} \kappa_ {j} (\boldsymbol {x} _ {s}, \boldsymbol {v} _ {t})\right) \left(\frac {m - 1}{m} \frac {\partial \kappa_ {j} (\boldsymbol {x} _ {r} , \boldsymbol {v} _ {j})}{\partial \gamma_ {j}} - \frac {1}{m} \sum_ {s \neq r} ^ {m} \frac {\partial \kappa_ {j} (\boldsymbol {x} _ {s} , \boldsymbol {v} _ {j})}{\partial \gamma_ {j}}\right). \\ \end{array}
$$
$$
\begin{array}{l} \frac {\partial (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) _ {j , j} ^ {\top}}{\partial \gamma_ {j}} \\ = 2 \left(\frac {m - 1}{m} \kappa_ {j} (\boldsymbol {x} _ {r}, \boldsymbol {v} _ {j}) - \frac {1}{m} \sum_ {s \neq r} ^ {m} \kappa_ {j} (\boldsymbol {x} _ {s}, \boldsymbol {v} _ {j})\right) \left(\frac {m - 1}{m} \frac {\partial \kappa_ {j} (\boldsymbol {x} _ {r} , \boldsymbol {v} _ {j})}{\partial \gamma_ {j}} - \frac {1}{m} \sum_ {s = r} ^ {m} \frac {\partial \kappa_ {j} (\boldsymbol {x} _ {s} , \boldsymbol {v} _ {j})}{\partial \gamma_ {j}}\right). \\ \end{array}
$$
For Mahalanobis matrix  $M_{j}$  with  $j\in [\ell ]$  , we have
$$
\nabla_ {M _ {j}} \mathcal {T} (\hat {X}, \hat {Y}) = \left( \begin{array}{c c c} \frac {\partial \mathcal {T} (\hat {X} , \hat {Y})}{\partial M _ {j , 1 , 1}} & \dots & \frac {\partial \mathcal {T} (\hat {X} , \hat {Y})}{\partial M _ {j , 1 , \ell}} \\ \vdots & \ddots & \vdots \\ \frac {\partial \mathcal {T} (\hat {X} , \hat {Y})}{\partial M _ {j , \ell , 1}} & \dots & \frac {\partial \mathcal {T} (\hat {X} , \hat {Y})}{\partial M _ {j , \ell , \ell}} \end{array} \right), \tag {35}
$$
where we denote by  $M_{j,a,b}$  the element in  $a$ -th row and  $b$ -th column in  $M_j$ .
$$
\frac {\partial \mathcal {T} (\hat {X} , \hat {Y})}{\partial M _ {j , a , b}} = \frac {\partial \mathcal {T} (\hat {X} , \hat {Y})}{\partial \boldsymbol {c} _ {\hat {X}}} \frac {\partial \boldsymbol {c} _ {\hat {X}}}{\partial M _ {j , a , b}} + \frac {\partial \mathcal {T} (\hat {X} , \hat {Y})}{\partial \boldsymbol {c} _ {\hat {Y}}} \frac {\partial \boldsymbol {c} _ {\hat {Y}}}{\partial M _ {j , a , b}} + \mathrm {T r} \left[ \frac {\partial \mathcal {T} (\hat {X} , \hat {Y})}{\partial \Sigma_ {\hat {X} , \hat {Y}}} \frac {\partial \Sigma_ {\hat {X} , \hat {Y}}}{\partial M _ {j , a , b}} \right],
$$
$$
\frac {\partial \boldsymbol {c} _ {\hat {X}}}{\partial M _ {j , a , b}} = \frac {1}{m} \sum_ {r = 1} ^ {m} \frac {\partial \hat {\boldsymbol {x}} _ {r}}{\partial M _ {j , a , b}} \quad \text {a n d} \quad \frac {\partial \hat {\boldsymbol {x}} _ {r}}{\partial M _ {j , a , b}} = \left(0, \dots , \frac {\partial \kappa_ {j} (\hat {\boldsymbol {x}} _ {r} , \boldsymbol {v} _ {j})}{\partial M _ {j , a , b}}, \dots , 0\right) ^ {\top},
$$
$$
\begin{array}{l} \frac {\partial \kappa_ {j} (\boldsymbol {x} _ {r} , \boldsymbol {v} _ {j})}{\partial M _ {j , a , b}} = \frac {\partial \exp \left(- (\boldsymbol {x} _ {r} - \boldsymbol {v} _ {j}) ^ {\top} M _ {j} (\boldsymbol {x} _ {r} - \boldsymbol {v} _ {j}) / 2 \gamma_ {j} ^ {2}\right)}{\partial M _ {j , a , b}} \\ = - \frac {\kappa_ {j} (\boldsymbol {x} _ {r} , \boldsymbol {v} _ {j})}{2 \gamma_ {j} ^ {2}} \operatorname {T r} \left[ \frac {\partial (\boldsymbol {x} _ {r} - \boldsymbol {v} _ {j}) ^ {\top} M _ {j} (\boldsymbol {x} _ {r} - \boldsymbol {v} _ {j})}{\partial M _ {j}} \frac {\partial M _ {j}}{\partial M _ {j , a , b}} \right] \\ = - \frac {\kappa_ {j} (\boldsymbol {x} _ {r} , \boldsymbol {v} _ {j})}{2 \gamma_ {j} ^ {2}} \operatorname {T r} \left[ (\boldsymbol {x} _ {r} - \boldsymbol {v} _ {j}) (\boldsymbol {x} _ {r} - \boldsymbol {v} _ {j}) ^ {\top} \boldsymbol {J} ^ {a, b} \right], \\ \end{array}
$$
where  $J^{a,b}$  is the single-entry matrix (1 at  $(a,b)$  and zero elsewhere). We similarly calculate  $\partial c_{\hat{Y}} / \partial M_{j,a,b}$  and have
$$
\frac {\partial \Sigma_ {\hat {X} , \hat {Y}}}{\partial M _ {j , a , b}} = \frac {1}{m + n - 2} \left(\sum_ {r = 1} ^ {m} \frac {\partial (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) ^ {\top}}{\partial M _ {j , a , b}} + \sum_ {r = 1} ^ {n} \frac {\partial (\hat {\boldsymbol {y}} _ {r} - \boldsymbol {c} _ {\hat {Y}}) (\hat {\boldsymbol {y}} _ {r} - \boldsymbol {c} _ {\hat {Y}}) ^ {\top}}{\partial M _ {j , a , b}}\right),
$$
$$
\begin{array}{l} \frac {\partial (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) ^ {\top}}{\partial M _ {j , a , b}} \\ = \left( \begin{array}{c c c c c} 0 & \dots & \frac {\partial (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\tilde {X}}) (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\tilde {X}}) _ {1 , j} ^ {\top}}{\partial M _ {j , a , b}} & \dots & 0 \\ \vdots & \ddots & \vdots & \ddots & \vdots \\ \frac {\partial (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\tilde {X}}) (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\tilde {X}}) _ {j , 1} ^ {\top}}{\partial M _ {j , a , b}} & \dots & \frac {\partial (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\tilde {X}}) (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\tilde {X}}) _ {j , j} ^ {\top}}{\partial M _ {j , a , b}} & \dots & \frac {\partial (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\tilde {X}}) (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\tilde {X}}) _ {j , \ell} ^ {\top}}{\partial M _ {j , a , b}} \\ \vdots & \ddots & \vdots & \ddots & \vdots \\ 0 & \dots & \frac {\partial (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\tilde {X}}) (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\tilde {X}}) _ {\ell , j} ^ {\top}}{\partial M _ {j , a , b}} & \dots & 0 \end{array} \right). \\ \end{array}
$$
Here, we have
$$
\begin{array}{l} \frac {\partial (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) _ {j , t} ^ {\top}}{\partial M _ {j , a , b}} \\ = \left(\frac {m - 1}{m} \kappa_ {j} (\boldsymbol {x} _ {r}, \boldsymbol {v} _ {t}) - \frac {1}{m} \sum_ {s \neq r} ^ {m} \kappa_ {j} (\boldsymbol {x} _ {s}, \boldsymbol {v} _ {t})\right) \left(\frac {m - 1}{m} \frac {\partial \kappa_ {j} (\boldsymbol {x} _ {r} , \boldsymbol {v} _ {j})}{\partial M _ {j , a , b}} - \frac {1}{m} \sum_ {s \neq r} ^ {m} \frac {\partial \kappa_ {j} (\boldsymbol {x} _ {s} , \boldsymbol {v} _ {j})}{\partial M _ {j , a , b}}\right). \\ \end{array}
$$
We similarly have  $\partial (\hat{\pmb{x}}_r - \pmb{c}_{\hat{X}})(\hat{\pmb{x}}_r - \pmb{c}_{\hat{X}})_{t,j}^{\top} / \partial M_{j,a,b}$  and
$$
\begin{array}{l} \frac {\partial (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) (\hat {\boldsymbol {x}} _ {r} - \boldsymbol {c} _ {\hat {X}}) _ {j , j} ^ {\top}}{\partial M _ {j , a , b}} \\ { = } { 2 \left( \frac { m - 1 } { m } \kappa _ { j } ( \pmb { x } _ { r } , \pmb { v } _ { j } ) - \frac { 1 } { m } \sum _ { s \neq r } ^ { m } \kappa _ { j } ( \pmb { x } _ { s } , \pmb { v } _ { j } ) \right) \left( \frac { m - 1 } { m } \frac { \partial \kappa _ { j } ( \pmb { x } _ { r } , \pmb { v } _ { j } ) } { \partial M _ { j , a , b } } - \frac { 1 } { m } \sum _ { s = r } ^ { m } \frac { \partial \kappa _ { j } ( \pmb { x } _ { s } , \pmb { v } _ { j } ) } { \partial M _ { j , a , b } } \right) . } \\ \end{array}
$$
Project Mahalanobis matrix onto a positive definite cone
We can not guarantee the positive-definiteness of Mahalanobis matrices during the optimization process via gradient ascend. Motivated from [43], we project Mahalanobis matrix onto a positive definite cone as follows:
- Present the spectral (eigenvalue) decomposition of a Mahalanobis matrix  $M$  as
$$
M = \sum_ {i = 1} ^ {d} \lambda_ {i} \boldsymbol {p} _ {i} \boldsymbol {p} _ {i} ^ {T}.
$$
where  $\lambda_1, \lambda_2, \dots, \lambda_d$  are their eigenvalues with corresponding eigenvectors  $p_1, p_2, \dots, p_d$ .
- Project the Mahalanobis matrix  $M$  onto a positive definite cone
$$
M = \sum_ {i = 1} ^ {d} \max  \left\{\lambda_ {i}, \delta \right\} v _ {i} v _ {i} ^ {T} \text {f o r s m a l l p o s i t i v e c o n s t a n t} \delta .
$$
C Datasets and Parameter Setting
We partition datasets into several disjoint subsets, and then randomly draw data elements from each subset based on the sample fraction, i.e., the proportion of samples to be selected from each
subset. We construct two different samples by using two different sample fractions in above stratified sampling process, and construct two samples drawn from one identical distribution by adapting a same sample fraction, as done in [60].
We provide the details of constructing two samples for each dataset as follows:
- blob is constructed as the mixture of nine Gaussian modes. We first write
$$
u _ {1} = [ 0, 0 ], \quad u _ {2} = [ 0, 1 ], \quad u _ {3} = [ 0, 2 ],
$$
$$
u _ {4} = [ 1, 0 ], \quad u _ {5} = [ 1, 1 ], \quad u _ {6} = [ 1, 2 ],
$$
$$
u _ {7} = [ 2, 0 ], \quad u _ {8} = [ 2, 1 ], \quad u _ {9} = [ 2, 2 ],
$$
$$
\Delta_ {i} = \left\{ \begin{array}{l l} - 0. 0 2 - 0. 0 0 2 \times (i - 1) & \text {f o r} i <   5 \\ 0 & \text {f o r} i = 5 \\ 0. 0 2 + 0. 0 0 2 \times (i - 6) & \text {f o r} i > 5  . \end{array} \right.
$$
To construct different distributions  $\mathbb{P}$  and  $\mathbb{Q}$ , we adapt different covariance structures and
$$
\mathbb {P} = \sum_ {i = 1} ^ {9} \frac {1}{9} \mathcal {N} \left(u _ {i}, 0. 0 3 \times I _ {2}\right) \quad \text {a n d} \quad \mathbb {Q} = \sum_ {i = 1} ^ {9} \frac {1}{9} \mathcal {N} \left(u _ {i}, \left[ \begin{array}{c c} 0. 0 3 & \Delta_ {i} \\ \Delta_ {i} & 0. 0 3 \end{array} \right]\right).
$$
To construct identical distribution for two samples, i.e.,  $\mathbb{P} = \mathbb{Q}$ , we have
$$
\mathbb {P} = \sum_ {i = 1} ^ {9} \frac {1}{9} \mathcal {N} \left(u _ {i}, 0. 0 3 \times I _ {2}\right) \quad \text {a n d} \quad \mathbb {Q} = \sum_ {i = 1} ^ {9} \frac {1}{9} \mathcal {N} \left(u _ {i}, 0. 0 3 \times I _ {2}\right).
$$
We set the sample size for training to 900 and for testing to 224.
- dna is a categorical dataset with 3 classes. For constructing two different samples, we set the sample fraction for one sample to  $[0.30, 0.35, 0.35]$  and for the other sample to  $[0.45, 0.25, 0.3]$ ; To construct two samples with one identical distribution, we set the same sample fraction  $[0.30, 0.35, 0.35]$  for two samples. We set the sample size for training to 1000 and for testing to 250.
- agnos (agnostic) is a categorical dataset with 2 classes. For constructing two different samples, we set the sample fraction for one sample to [0.35, 0.65] and for the other sample to [0.65, 0.35]; To construct two samples with one identical distribution, we set the same sample fraction [0.35, 0.65] for two samples. We set the sample size for training to 1000 and for testing to 250.
- topo21 is a regression dataset with continuous target variables. Based on the sorted target variables, we divide the data into 4 equal parts. For constructing two different samples, we set the sample fraction for one sample to  $[0.1, 0.3, 0.2, 0.4]$  and for the other sample to  $[0.5, 0.2, 0.1, 0.2]$ ; To construct two samples with one identical distribution, we set the same sample fraction  $[0.1, 0.3, 0.2, 0.4]$  for two samples. We set the sample size for training to 2200 and for testing to 550.
- har is a categorical dataset with 6 classes. For constructing two different samples, we set the sample fraction for one sample to [0.10, 0.20, 0.10, 0.20, 0.20, 0.10] and for the other sample to [0.15, 0.15, 0.20, 0.15, 0.20, 0.15]; To construct two samples with one identical distribution, we set the same sample fraction [0.10, 0.20, 0.10, 0.20, 0.20, 0.10] for two samples. We set the sample size for training to 2200 and for testing to 550.
- kropt is a categorical dataset with 18 classes, where we only consider categories 13, 14, 15 and 16 which have the majority of the data.. For constructing two different samples, we set the sample fraction for one sample to [0.15, 0.2, 0.3, 0.35] and for the other sample to [0.35, 0.35, 0.15, 0.15]; To construct two samples with one identical distribution, we set the same sample fraction [0.25, 0.25, 0.25, 0.25] for two samples. We set the sample size for training to 2000 and for testing to 500.
- diamond (diamonds) is a regression dataset with continuous target variables. Based on the sorted target variables, we divide the data into 4 equal parts. For constructing two different samples, we set the sample fraction for one sample to  $[0.35, 0.2, 0.2, 0.25]$  and for the other sample to  $[0.2, 0.3, 0.3, 0.2]$ ; To construct two samples with one identical distribution, we set the same sample fraction  $[0.25, 0.25, 0.25, 0.25]$  for two samples. We set the sample size for training to 2000 and for testing to 500.
Table 4: Optimization parameters of our  ${\mathrm{{ME}}}_{\mathrm{{MaBiD}}}$  test for different datasets.
Dataset	# Test Locations	Learning Rate	Optimization Epoch
blob	17	0.007	1000
dna	15	0.0004	1000
agnos	15	0.001	200
topo21	15	0.001	1000
har	15	0.001	100
kropt	1	0.002	1000
diamon	2	0.0013	1000
cifar10	2	0.003	200
mnist	2	0.001	500
santan	10	0.001	1000
codrna	2	0.01	1000
sea50	1	0.01	1000
adult	1	0.001	1000
labor	1	0.001	1000
poker	3	0.002	200
higgs	15	0.001	1000
Algorithm 2  $\mathrm{ME}_{\mathrm{MaBiD}}$  training
Input: Two training samples  $X$  and  $Y$ , number of test locations  $\ell$ , optimization epoch  $K$ , step size  $\eta$   
Output:  $\{v_{i}\}_{i=1}^{\ell}$ ,  $\{M_{i}\}_{i=1}^{\ell}$ ,  $\{\gamma_{i}\}_{i=1}^{\ell}$  and  $\pmb{F}$
1: Initialize  $\{\pmb{v}_i\}_{i=1}^\ell, \{M_i\}_{i=1}^\ell = \pmb{I}_d$  and  $\{\gamma_i\}_{i=1}^\ell$  by Eqn. (36)
2: for  $k = 1,2,\ldots ,K$  do
3: Calculate the embedding training samples  $\hat{X}$  and  $\hat{Y}$  based on Eqn (2)
4: Calculate the statistic  $\mathcal{T}(\hat{X},\hat{Y})$  based on Eqn. (4)
5: Calculate gradient  $\nabla_{\boldsymbol{v}_j}\mathcal{T}(\hat{X},\hat{Y}),\nabla_{\gamma_j}\mathcal{T}(\hat{X},\hat{Y})$  and  $\nabla_{M_j}\mathcal{T}(\hat{X},\hat{Y})$  via Eqns. (32), (34), (35)
6: Gradient ascend based on Adam optimization method with step size  $\eta$
7: end for
8: Calculate the embedding training samples  $\hat{X}$  and  $\hat{Y}$  based on Eqn (2)
9: Calculate the pooled covariance matrix  $\Sigma_{\hat{X},\hat{Y}}$  based on Eqn. (3)
0: Calculate the Schur decomposition of the pooled covariance matrix:  $LL = \Sigma_{\hat{X},\hat{Y}}$ .
11: Calculate the inference direction  $\mathbf{F}$  based on Eqn. (6)
12: return  $\{\pmb{v}_i\}_{i=1}^{\ell}$ ,  $\{M_i\}_{i=1}^{\ell}$ ,  $\{\gamma_i\}_{i=1}^{\ell}$  and  $\pmb{F}$
- higgs is a categorical dataset with 2 classes. For constructing two different samples, we set the sample fraction for one sample to  $[0.0, 1.0]$  and for the other sample to  $[1.0, 0.0]$ ; To construct two samples with one identical distribution, we set same sample fraction  $[1.0, 0.0]$  for two samples. We set the sample size for training to 16000 and for testing to 4000.
In optimization, we adapt Adam optimization method from the pytorch library in python [88, 89]. Table 4 presents the details of the hyperparameter settings for each dataset, including the number of test locations, learning rate.
Experimental settings
At initialization, we usually set the Mahalanobis matrices  $\{M_i\}_{i=1}^{\ell}$  to identity matrices. We further provide an alternative initialization for the Mahalanobis matrices based on the correlation information of two training samples as follows:
$$
M _ {j} = \sum_ {i = 1} ^ {m + n} \left(\boldsymbol {S} _ {j, i} - \boldsymbol {c} _ {\boldsymbol {S} _ {j}}\right) \left(\boldsymbol {S} _ {j, i} - \boldsymbol {c} _ {\boldsymbol {S} _ {j}}\right) ^ {\top} / (m + n - 1) + \delta \mathbf {I} _ {d},
$$
where  $S_{j} = \{x_{1} - v_{j},\dots,x_{m} - v_{j},y_{1} - v_{j},\dots,y_{n} - v_{j}\}$ ,  $S_{j,i}$  is the  $i$ -th element of  $S_{j}$ , and  $c_{S_j} = \sum_{i = 1}^{m + n}S_{j,i} / (m + n)$ .  $\mathbf{I}_d$  is an identity matrix of size  $d\times d$  to guarantee the positive definiteness with small constant  $\delta >0$ . Here, the elements  $x$  and  $y$  are come from training samples  $X$  and  $Y$ , and  $m$  and  $n$  are the numbers of elements in  $X$  and  $Y$  respectively.
For initialization of test locations, we first fit a Gaussian distribution for each sample and draw half of the number of test locations from each distribution. This could be expensive for high dimensional dataset, and we can simplify the process by directly sampling test locations from original dataset.
For initialization of the bandwidth parameters of Mahalanobis kernels, we let  $\gamma_{j} = \gamma$  for every  $j\in [\ell ]$  and then linearly search for  $\gamma$  that maximize the statistic from a candidate list with fixed test locations and Mahalanobis matrices, which can be formalized as follows: