SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We see that when selected  $x_{i}^{t*}$  by GFT is close to  $\mu_{i}$  then RFT and GFT on average share the same performance. (show that if maximal model disagreement occurs at the mean of  $x$  distribution, then such a greedy teacher may share similar performance with a random teacher on average.) This is important to gain an insight on when a greedy teacher is better than a random teacher. Then, we have
$$
\begin{array}{l} \sum_ {i = 1} ^ {d} m _ {i, t} \left(x _ {i} ^ {t ^ {*}}\right) - \sum_ {i = 1} ^ {d} \left(\left\| x _ {i} ^ {t ^ {*}} - \mu_ {i} \right\| _ {2}\right) = \sum_ {i = 1} ^ {d} \left(m _ {i, t} \left(x _ {i} ^ {t ^ {*}}\right) - \left\| x _ {i} ^ {t ^ {*}} - \mu_ {i} \right\| _ {2}\right) \\ \leq \frac {2 \mathbb {E} _ {\boldsymbol {x} \sim [ \mathbb {P} _ {i} \left(x _ {i}\right) ] ^ {d}} \left[ \mathcal {L} \left(\boldsymbol {f} ^ {t}\right) - \mathcal {L} \left(\boldsymbol {f} ^ {t + 1}\right) \right]}{\tilde {\eta} ^ {t}}. \tag {48} \\ \end{array}
$$
$$
\sum_ {i = 1} ^ {d} m _ {i, t} \left(x _ {i} ^ {t ^ {*}}\right) \leq \frac {2 \mathbb {E} _ {\boldsymbol {x}} \sim [ \mathbb {P} _ {i} \left(x _ {i}\right) ] ^ {d} \left[ \mathcal {L} \left(\boldsymbol {f} ^ {t}\right) - \mathcal {L} \left(\boldsymbol {f} ^ {t + 1}\right) \right]}{\tilde {\eta} ^ {t}} + \sum_ {i = 1} ^ {d} \left(\left\| x _ {i} ^ {t ^ {*}} - \mu_ {i} \right\| _ {2}\right). \tag {49}
$$
$$
\frac {2 \mathbb {E} _ {\boldsymbol {x} \sim [ \mathbb {P} _ {i} \left(x _ {i}\right) ] ^ {d}} \left[ \mathcal {L} \left(\boldsymbol {f} ^ {t}\right) - \mathcal {L} \left(\boldsymbol {f} ^ {t + 1}\right) \right]}{d \tilde {\eta} ^ {t}} + \frac {1}{d} \sum_ {i = 1} ^ {d} \left(\| x _ {i} ^ {t *} - \mu_ {i} \| _ {2}\right) \geq \min  _ {i \in \mathbb {N} _ {d}} m _ {i, t} \left(x _ {i} ^ {t *}\right). \tag {50}
$$
Equivalently, replace index  $t$  by  $l$ ,  $\frac{2\mathbb{E}_{\boldsymbol{\omega} \sim [\mathbb{P}_i(x_i)]^d}[\mathcal{L}(\boldsymbol{f}^l) - \mathcal{L}(\boldsymbol{f}^{l + 1})]}{d\tilde{\eta}^l} + \frac{1}{d}\sum_{i = 1}^{d}\left(\| x_i^{l*} - \mu_i\| _2\right) \geq \min_{i\in \mathbb{N}_d}m_{i,l}(x_i^{l*})$ . Consequently, plugging  $l = 0,1,\ldots ,t - 1$  in it and summing them up, we hence have
$$
\begin{array}{l} \sum_ {l = 0} ^ {t - 1} \min  _ {i \in \mathbb {N} _ {d}} m _ {i, l} \left(x _ {i} ^ {l ^ {*}}\right) \\ \leq \frac {2}{d} \sum_ {l = 0} ^ {t - 1} \frac {\mathbb {E} _ {\boldsymbol {x} \sim [ \mathbb {P} _ {i} (x _ {i}) ] ^ {d}} \left[ \mathcal {L} (\boldsymbol {f} ^ {l}) - \mathcal {L} (\boldsymbol {f} ^ {l + 1}) \right]}{\tilde {\eta} ^ {l}} + \frac {1}{d} \sum_ {l = 0} ^ {t - 1} \sum_ {i = 1} ^ {d} \left(\| x _ {i} ^ {l ^ {*}} - \mu_ {i} \| _ {2}\right) \\ \leq \frac {2}{d \dot {\eta}} \sum_ {l = 0} ^ {t - 1} \mathbb {E} _ {\boldsymbol {x} \sim \left[ \mathbb {P} _ {i} \left(x _ {i}\right) \right] ^ {d}} \left[ \mathcal {L} \left(\boldsymbol {f} ^ {l}\right) - \mathcal {L} \left(\boldsymbol {f} ^ {l + 1}\right) \right] + \frac {1}{d} \sum_ {l = 0} ^ {t - 1} \sum_ {i = 1} ^ {d} \left(\left\| x _ {i} ^ {l ^ {*}} - \mu_ {i} \right\| _ {2}\right), \tag {51} \\ \end{array}
$$
where  $\dot{\eta} = \min_{l\in \{0\} \bigcup \mathbb{N}_{t - 1}}\tilde{\eta}^l >0$  Expanding the r.h.s. term in Eq. 51 yields
$$
\begin{array}{l} \frac {2}{d \dot {\eta}} \sum_ {l = 0} ^ {t - 1} \mathbb {E} _ {\boldsymbol {x} \sim [ \mathbb {P} _ {i} (x _ {i}) ] ^ {d}} \left[ \mathcal {L} (\boldsymbol {f} ^ {l}) - \mathcal {L} (\boldsymbol {f} ^ {l + 1}) \right] + \frac {1}{d} \sum_ {l = 0} ^ {t - 1} \sum_ {i = 1} ^ {d} \left(\| x _ {i} ^ {l ^ {*}} - \mu_ {i} \| _ {2}\right) \\ = \frac {2}{d \dot {\eta}} \mathbb {E} _ {\boldsymbol {x} \sim [ \mathbb {P} _ {i} (x _ {i}) ] ^ {d}} \left[ \mathcal {L} (\boldsymbol {f} ^ {0}) - \mathcal {L} (\boldsymbol {f} ^ {t}) \right] + \frac {1}{d} \sum_ {l = 0} ^ {t - 1} \sum_ {i = 1} ^ {d} \left(\| x _ {i} ^ {l ^ {*}} - \mu_ {i} \| _ {2}\right) \\ \leq \frac {2}{d \dot {\eta}} \mathbb {E} _ {\boldsymbol {x} \sim [ \mathbb {P} _ {i} (x _ {i}) ] ^ {d}} \left[ \mathcal {L} (\boldsymbol {f} ^ {0}) \right] + \frac {1}{d} \sum_ {l = 0} ^ {t - 1} \sum_ {i = 1} ^ {d} \left(\| x _ {i} ^ {l ^ {*}} - \mu_ {i} \| _ {2}\right). \tag {52} \\ \end{array}
$$
In terms of the l.h.s. term in Eq. 51, we must have
$$
\sum_ {l = 0} ^ {t - 1} \min  _ {i \in \mathbb {N} _ {d}} m _ {i, l} \left(x _ {i} ^ {l ^ {*}}\right) \geq t \cdot \min  _ {l \in \{0 \} \bigcup \mathbb {N} _ {t - 1}} \min  _ {i \in \mathbb {N} _ {d}} m _ {i, l} \left(x _ {i} ^ {l ^ {*}}\right). \tag {53}
$$
Combining expression 51, 52 and 53, we thus have
$$
\begin{array}{l} t \cdot \min  _ {l \in \{0 \}} \min  _ {\bigcup \mathbb {N} _ {t - 1}} \min  _ {i \in \mathbb {N} _ {d}} m _ {i, l} \left(x _ {i} ^ {l ^ {*}}\right) \leq \sum_ {l = 0} ^ {t - 1} \min  _ {i \in \mathbb {N} _ {d}} m _ {i, l} \left(x _ {i} ^ {l ^ {*}}\right) \\ \leq \frac {2}{d \dot {\eta}} \mathbb {E} _ {\boldsymbol {x} \sim [ \mathbb {P} _ {i} (x _ {i}) ] ^ {d}} \left[ \mathcal {L} (\boldsymbol {f} ^ {0}) \right] + \frac {1}{d} \sum_ {l = 0} ^ {t - 1} \sum_ {i = 1} ^ {d} \left(\| x _ {i} ^ {l *} - \mu_ {i} \| _ {2}\right). \tag {54} \\ \end{array}
$$
Since  $m_{i,t}(x_i^{t*})$  is monotonically non-increasing w.r.t. iteration  $t$ , we can derive
$$
\min  _ {i \in \mathbb {N} _ {d}} m _ {i, t - 1} \left(x _ {i} ^ {t - 1 ^ {*}}\right) \leq \frac {2}{d \dot {\eta} t} \mathbb {E} _ {\boldsymbol {x} \sim \left[ \mathbb {F} _ {i} \left(x _ {i}\right) \right] ^ {d}} \left[ \mathcal {L} \left(\boldsymbol {f} ^ {0}\right) \right] + \frac {1}{d t} \sum_ {l = 0} ^ {t - 1} \sum_ {i = 1} ^ {d} \left(\| x _ {i} ^ {l ^ {*}} - \mu_ {i} \| _ {2}\right), \tag {55}
$$
when the returned vector-valued model for multiple learners is  $f^t$ .
To compare with RFT, we can plug the r.h.s. of Eq. 38 and see that the loss reduction is more than  $\mathbb{E}_{\boldsymbol{x} \sim [\mathbb{P}_i(x_i)]^d} \left[ \mathcal{L}(\boldsymbol{f}^0) \right]$ , which indicates GFT needs less iterations to converge and the efficiency of GFT. Compared with Eq. 35 and 55, we see that the inferiority of RFT compared to GFT comes from the cumulative distance between the example mean and the example selected by GFT.
Proof of Proportion 11 When the proximity between  $\pmb{f}^t$  and  $\pmb{f}^{*}$  is sufficiently close, we have  $\epsilon < \epsilon_0$  where  $\epsilon_0$  is the pre-defined approximation error of employed two-layer perceptron. Take  $A = I_d$ , then we have  $\| A \cdot \pmb{f}^t - \pmb{f}^* \|_{\mathcal{H}^d} = \| I_d \cdot \pmb{f}^t - \pmb{f}^* \|_{\mathcal{H}^d} = \| \pmb{f}^t - \pmb{f}^* \|_{\mathcal{H}^d} \leq \epsilon \leq \epsilon_0$ , which means that this perceptron has searched the matrix  $A = I_d$  that satisfies  $\| A \cdot \pmb{f}^t - \pmb{f}^* \|_{\mathcal{H}^d} \leq \epsilon_0$ .
Proof of Lemma 12 Since the multi-learner convex loss  $\mathcal{L}(\pmb{f})$  will decrease as input  $\pmb{f}$  close to  $\pmb{f}^*$  and matrix  $A^t$  comes from  $A^t = \underset {A\in \mathbb{R}^{d\times d}}{\arg \min}\| A\pmb{f}^t -\pmb{f}^*\|_{\mathcal{H}^d}$ , searching a matrix  $A$  to minimize the disagreement between  $A\pmb{f}^t$  and  $\pmb{f}^*$ , it is trivial to see that
$$
\mathcal {L} \left(\boldsymbol {f} ^ {t}\right) - \mathcal {L} \left(A ^ {t} \boldsymbol {f} ^ {t}\right) \geq 0. \tag {56}
$$
Based on Assumption 3, we can derive that
$$
\left| \mathcal {L} \left(A ^ {t} \boldsymbol {f} ^ {t}\right) - \mathcal {L} \left(\boldsymbol {f} ^ {*}\right) \right| \leq L _ {\mathcal {L}} \| A ^ {t} \boldsymbol {f} ^ {t} - \boldsymbol {f} ^ {*} \| _ {\mathcal {H} ^ {d}} \tag {57}
$$
$$
\left| \mathcal {L} \left(\boldsymbol {f} ^ {t}\right) - \mathcal {L} \left(\boldsymbol {f} ^ {*}\right) \right| \leq L _ {\mathcal {L}} \| \boldsymbol {f} ^ {t} - \boldsymbol {f} ^ {*} \| _ {\mathcal {H} ^ {d}}. \tag {58}
$$
$$
\begin{array}{l} \mathcal {L} (\boldsymbol {f} ^ {t}) - \mathcal {L} (A ^ {t} \boldsymbol {f} ^ {t}) = \mathcal {L} (\boldsymbol {f} ^ {t}) - \mathcal {L} (\boldsymbol {f} ^ {*}) + \mathcal {L} (\boldsymbol {f} ^ {*}) - \mathcal {L} (A ^ {t} \boldsymbol {f} ^ {t}) \\ \leq | \mathcal {L} (\boldsymbol {f} ^ {t}) - \mathcal {L} (\boldsymbol {f} ^ {*}) | + | \mathcal {L} (\boldsymbol {f} ^ {*}) - \mathcal {L} (A ^ {t} \boldsymbol {f} ^ {t}) | \\ \leq L _ {\mathcal {L}} \cdot \left(\| A ^ {t} \boldsymbol {f} ^ {t} - \boldsymbol {f} ^ {*} \| _ {\mathcal {H} ^ {d}} + \| \boldsymbol {f} ^ {t} - \boldsymbol {f} ^ {*} \| _ {\mathcal {H} ^ {d}}\right) \\ \leq 2 L _ {\mathcal {L}} \cdot \| \boldsymbol {f} ^ {t} - \boldsymbol {f} ^ {*} \| _ {\mathcal {H} ^ {d}}, \tag {59} \\ \end{array}
$$
Proof of Theorem 13 Following the style of the previous proof, e.g., the proof of Lemma 7, we begin by investigating the reduction in the loss for a single learner.
$$
\begin{array}{l} \mathbb {E} _ {x _ {i} \sim \mathbb {P} _ {i} (x _ {i})} \left[ \mathcal {L} _ {i} (f _ {A ^ {t}, i} ^ {t + 1}) - \mathcal {L} _ {i} (f _ {i} ^ {t}) \right] \\ = \mathbb {E} _ {x _ {i} \sim \mathbb {P} _ {i} (x _ {i})} \left[ \mathcal {L} _ {i} \left(f _ {A ^ {t}, i} ^ {t + 1}\right) - \mathcal {L} _ {i} \left(A _ {(i, \cdot)} ^ {t} \boldsymbol {f} ^ {t}\right) + \mathcal {L} _ {i} \left(A _ {(i, \cdot)} ^ {t} \boldsymbol {f} ^ {t}\right) - \mathcal {L} _ {i} \left(f _ {i} ^ {t}\right) \right] \\ = \mathbb {E} _ {x _ {i} \sim \mathbb {P} _ {i} (x _ {i})} \left[ \mathcal {L} _ {i} \left(f _ {A ^ {t}, i} ^ {t + 1}\right) - \mathcal {L} _ {i} \left(A _ {(i, \cdot)} ^ {t} \boldsymbol {f} ^ {t}\right) \right] + \mathbb {E} _ {x _ {i} \sim \mathbb {P} _ {i} (x _ {i})} \left[ \mathcal {L} _ {i} \left(A _ {(i, \cdot)} ^ {t} \boldsymbol {f} ^ {t}\right) - \mathcal {L} _ {i} \left(f _ {i} ^ {t}\right) \right] \\ \leq^ {*} \mathbb {E} _ {x _ {i} \sim \mathbb {P} _ {i} (x _ {i})} \left[ \mathcal {L} _ {i} \left(f _ {A ^ {t}, i} ^ {t + 1}\right) - \mathcal {L} _ {i} \left(A _ {(i, \cdot)} ^ {t} \boldsymbol {f} ^ {t}\right) \right], \tag {60} \\ \end{array}
$$
where it follows from Lemma 12 that  $\stackrel{*}{\leq}$  holds. Therefore, we have
$$
\begin{array}{l} \mathbb {E} _ {\pmb {x} \sim [ \mathbb {P} _ {i} (x _ {i}) ] ^ {d}} \left[ \mathcal {L} (\pmb {f} _ {A ^ {t}} ^ {t + 1}) - \mathcal {L} (\pmb {f} ^ {t}) \right] \\ = \mathbb {E} _ {\boldsymbol {x} \sim [ \mathbb {P} _ {i} (x _ {i}) ] ^ {d}} \left[ \sum_ {i = 1} ^ {d} \mathcal {L} _ {i} \left(f _ {A ^ {t}, i} ^ {t + 1}\right) - \sum_ {i = 1} ^ {d} \mathcal {L} _ {i} \left(f _ {i} ^ {t}\right) \right] \\ = \sum_ {i = 1} ^ {d} \mathbb {E} _ {x _ {i} \sim \mathbb {P} _ {i} (x _ {i})} \left[ \mathcal {L} _ {i} \left(f _ {A ^ {t}, i} ^ {t + 1}\right) - \mathcal {L} _ {i} \left(f _ {i} ^ {t}\right) \right] \\ \leq \sum_ {i = 1} ^ {d} \mathbb {E} _ {x _ {i} \sim \mathbb {P} _ {i} (x _ {i})} \left[ \mathcal {L} _ {i} \left(f _ {A ^ {t}, i} ^ {t + 1}\right) - \mathcal {L} _ {i} \left(A _ {(i, \cdot)} ^ {t} \boldsymbol {f} ^ {t}\right) \right] \\ = \mathbb {E} _ {\boldsymbol {x} \sim [ \mathbb {P} _ {i} (x _ {i}) ] ^ {d}} \left[ \mathcal {L} \left(\boldsymbol {f} _ {A ^ {t}} ^ {t + 1}\right) - \mathcal {L} \left(A ^ {t} \boldsymbol {f} ^ {t}\right) \right] \leq 0, \tag {61} \\ \end{array}
$$
Figure 6: Extensive visualization of  $\pmb{f}^{t}$  for single-learner teaching.
Figure 7: The channel-wise visualization of specific  $\pmb{f}^{t}$  corresponding to Figure 4 (b)-(c).
(a) Vanilla MINT.
(b) Communicated MINT.
C Detailed Experiments and Extensions
Since computers operate in a discrete manner, we employ dense pairwise points  $\{(x_i,f(x_i))\}_{i\in \mathbb{N}_n}$  to represent a scalar-valued function  $f$  and points  $\{([x_i]_j^d,[f_i(x_i)]_j^d)\}_{j\in \mathbb{N}_m}$  to represent a vector-valued function  $\pmb {f} = [f_i]^d\in \mathcal{H}^d$ . To facilitate visualization, our experiments have utilized 2D (1D Gaussian data), 3D (a grayscale image) and 4D (a colored image) examples to demonstrate the insights obtained from our theoretical analysis. Generally, the domain of the functions being learned in 3D cases is determined by the  $x$  and  $y$  values, which represent the pixel locations, while the range is represented by the  $z$  values, indicating the color levels. This is comparable for 2D and 4D cases. Besides, for high-dimensional vision datasets that can be formulated as vector-valued functions, the methodology developed in this work can be applied as well. For all experiments, we align with [73] to set RBF  $K(x,x^{\prime}) = \exp \left(-\left\| \frac{x - x^{\prime}}{2}\right\|_{2}^{2}\right)$  as the kernel and to take empirical (average)  $L_{2}$  norm defined in vector-valued Hilbert space to measure the difference between  $\pmb {f}\in \mathcal{H}^d$  and  $\pmb {f}^{*}\in \mathcal{H}^{d}$
$$
\mathcal {M} (\boldsymbol {f}, \boldsymbol {f} ^ {*}) = \| \boldsymbol {f} - \boldsymbol {f} ^ {*} \| _ {\mathcal {H} ^ {d}} = \frac {1}{d n} \sqrt {\sum_ {i = 1} ^ {d} \sum_ {j = 1} ^ {n} \left(f _ {i} \left(x _ {i , j}\right) - f _ {i} ^ {*} \left(x _ {i , j}\right)\right) ^ {2}}.
$$
Our implementation relies on the Intel(R) Core(TM) i7-8750H processor and utilizes NVIDIA graphics cards, specifically the GTX 1050 Ti with Max-Q Design and RTX6000.
MINT in gray scale. For impartation of a tiger<sup>5</sup> and a cheetah [15], we assume the loss functions for both learners are square loss  $\mathcal{L}_i = (y - f_i(x))^2$ ,  $i \in \mathbb{N}_2$ . For the monochrome lion divided into  $4 \times 4$
<sup>5</sup>https://www.etsy.com/ie/listing/837781388/icy-gaze-white-tiger-original
Table 1: Peak signal-to-noise ratio (PSNR) corresponding to learnt  $\mathbf{f}^{t}$  show in Figure 4.
0	10,000	50,000	100,000	150,000	210,000
Single-learner teaching	8.73	9.16	10.23	11.66	12.99	14.49
Vanilla MINT	8.73	10.01	13.19	16.27	18.98	22.00
Communicated MINT	8.73	12.38	15.62	18.74	21.50	24.58
Table 2: PSNR corresponding to learnt  $\mathbf{f}^{t}$  show in Figure 8.
0	10,000	50,000	100,000	150,000	210,000
Single-learner teaching	8.73	8.87	9.45	10.17	10.89	11.75
Vanilla MINT	8.73	9.16	10.88	13.03	15.19	17.76
Communicated MINT	8.73	11.46	13.20	15.39	17.58	20.19
pieces, each piece is learnt by a single learner whose loss function is set to be square loss, and all loss functions are the same,  $\mathcal{L}_i = (y - f_i(x))^2$ ,  $i \in \mathbb{N}_{16}$ . This enables the teaching of a single-learner target model having large input spaces.
MINT in three (RGB) channels. Three learners, each with their own randomly-initialized  $f^0$ , are tasked with learning the three (RGB) channel lion image. Each learner is equipped with a square loss function to learn their respective channels. In the case of single-learner teaching, the teacher must repeatedly teach all three channels. In Vanilla MINT, the teacher can teach all three channels concurrently. In communicated MINT, the teacher uses a two-layer perceptron to solve for the communication matrix  $A^t$ , with the linear layer initialized by an identity matrix. This allows the teacher to teach all three channels simultaneously.
We present a comprehensive visualization of  $\pmb{f}^{t}$  for Figure 4 (a) in Figure 6, while the channel-specific visualization of certain  $\pmb{f}^{t}$  corresponding to Figure 4 (b)-(c) is exhibited in Figure 7. In order to evaluate the quality of the learned  $\pmb{f}^{t}$  (as images), we utilize the Peak Signal-to-Noise Ratio (PSNR), and the results corresponding to Figure 4 are presented in Table 1. Meanwhile, Figure 8 displays the specific learned  $\pmb{f}^{t}$  for RFT, and its corresponding PSNR is listed in Table 2. Additionally, Figures 9 (a) and (b) respectively exhibit the channel-specific visualizations of vanilla (Figure 8 (b)) and communicated (Figure 8 (c)) MINT for RFT.
In addition, we also investigate the performance of single-learner teaching, as well as vanilla and communicated MINT under a specific initialization of  $f^0$ , which is obtained through a linear combination of  $f^*$  using the inverse of matrix
$$
\left( \begin{array}{c c c} 1. 2 6 & 2. 2 2 & 3. 6 0 \\ 2. 4 7 & - 0. 5 3 & 2. 3 6 \\ 2. 4 0 & 1. 6 8 & 0. 4 0 \end{array} \right).
$$
Figure 10 displays the plot of the loss. Figure 11 showcases the specific learned  $f^t$  for three versions of GFT throughout each iteration. From Figure 11, we can observe that MINT consistently outperforms the single-learner teaching, and additionally, communicated MINT exhibits better performance compared to vanilla MINT. Figure 12 presents the channel-specific visualization of certain  $f^t$  that correspond to Figure 11 (b)-(c), while Table 3 lists the PSNR for all comparable  $f^t$  in Figure 11. Additionally, Figure 13 displays the specific learned  $f^t$  for RFT, while the channel-specific visualization of certain  $f^t$  for MINT is exhibited in Figure 14. The corresponding PSNR values are
Table 3: PSNR corresponding to learnt  $\mathbf{f}^{t}$  show in Figure 11.
0	2,000	10,000	20,000	50,000	90,000
Single-learner teaching	7.58	7.65	7.88	8.15	8.88	10.07
Vanilla MINT	7.58	7.82	8.56	9.35	11.37	13.68
Communicated MINT	7.58	16.29	17.12	17.98	20.19	22.75
$^{6}$ https://bmild.github.io/fourfeat/img/lion_orig.png
Table 4: PSNR corresponding to learnt  $\mathbf{f}^{t}$  show in Figure 13.
0	2,000	10,000	20,000	50,000	90,000
Single-learner teaching	7.58	7.60	7.72	7.87	8.33	8.92
Vanilla MINT	7.58	7.66	8.03	8.48	9.82	11.62
Communicated MINT	7.58	16.08	16.44	16.89	18.23	20.03
Figure 8: Visualization of  $\pmb{f}^{t}$  taught by RFT under the random initialization.
listed in Table 4. An interesting finding is that with this specific initialization, the teacher can assist the learners in directly learning  $f^{*}$  by providing the communication matrix
$$
\left( \begin{array}{c c c} 1. 2 7 & 2. 2 2 & 3. 6 0 \\ 2. 4 7 & - 0. 5 3 & 2. 3 6 \\ 2. 4 0 & 1. 6 8 & 0. 4 0 \end{array} \right),
$$
which is solved by a two-layer perceptron. In order to elaborate this observation, we offer an illustration in Figure 15.
Synthetic 1D Gaussian data. We use a synthetic 1D Gaussian data set for teaching a single learner to demonstrate that the greater the distance between the example mean and the example selected by GFT, the more significant the difference between RFT and GFT. We set  $f^0 = 0$  and  $f^{*} = \mathcal{N}(x;0,5^{2})$ , where  $\mathcal{N}(x;\mu ,\sigma^2)$  represents the probability density function of a Gaussian distribution with mean  $\mu$  and standard deviation  $\sigma$ . It is trivial to see that GFT will select examples primarily around  $x = 0$ . Looking at the loss depicted in Figure 16 (a), we observe that GFT reduces the loss more rapidly than
Figure 9: The channel-wise visualization of specific  $\pmb{f}^{t}$  corresponding to Figure 8 (b)-(c).
Figure 10: Convergence performance for teaching RGB lion under a particular initialization of  $f^0$ .
RFT in the settings of  $\mu = 0, \mu = -4, \mu = -7$  (in an order of decreasing speed). This is consistent with the observation in Theorem 10. Additionally, we present the specific learned function  $f^t$  in Figure 16 (b).
Synthetic bivariate mixture Gaussian data. To further showcase the effectiveness of MINT, we utilize synthetic bivariate mixture Gaussian data. It is well-established that a bivariate Gaussian distribution  $f_{X_1,X_2}(x_1,x_2)$  can be factored as the product of two independent univariate Gaussian distributions  $f_{X_1,X_2}(x_1,x_2) = f_{X_1}(x_1)f_{X_2}(x_2)^7$  [8], regardless of whether  $X_{1}$  and  $X_{2}$  are correlated. For simplicity, our primary focus is on the scenario where there is no correlation  $(\rho = 0)$ , and on GFT. We set
$$
f _ {X _ {1}, X _ {2}} ^ {0} (x _ {1}, x _ {2}) = f _ {X _ {1}} ^ {0} (x _ {1}) f _ {X _ {2}} ^ {0} (x _ {2}) = \mathcal {N} (x; - 2, 1. 5 ^ {2}) \mathcal {N} (x; 2, 1 ^ {2})
$$
$$
\begin{array}{l} f _ {X _ {1}, X _ {2}} ^ {*} (x _ {1}, x _ {2}) = f _ {X _ {1}} ^ {*} (x _ {1}) f _ {X _ {2}} ^ {*} (x _ {2}) \\ = \left(\mathcal {N} (x; - 2, 1. 5 ^ {2}) / 3 + 2 \mathcal {N} (x; 2, 1 ^ {2}) / 3\right) \left(3 \mathcal {N} (x; - 2, 1. 5 ^ {2}) / 4 + \mathcal {N} (x; 2, 1 ^ {2}) / 4\right). \\ \end{array}
$$
Figure 17 depicts the visualization of  $f_{X_1,X_2}^* (x_1,x_2)$ . One can also formulate such a single-learner target function into a multi-learner (vector-valued) one  $\pmb{f}$ , with the first component being  $f_{X_1}(x_1)$  and the second component being  $f_{X_2}(x_2)$ . Thus, we can represent  $\pmb{f}$  as  $\pmb{f} = (f_{X_1},f_{X_2})^T$ . To simplify the notation, we can represent it as  $\pmb{f} = (f_1,f_2)^T$ . In Figure 18, we can observe the specific learned  $\pmb{f}^t$  of single-learner teaching, vanilla and communicated MINT. Based on the observations from Figure 18, it can be concluded that MINT method is much more effective in terms of efficiency when compared to single-learner teaching. Furthermore, for communicated MINT, the teacher can provide the communicated matrix to let  $\pmb{f}$  being learnt within a single iteration. Besides, Figure 19 shows the loss plot which tracks convergence performance.
Synthetic 1D data. We use synthetic one-dimensional data to evaluate when the communication matrix  $A_{t}$  is advantageous. Specifically, our main focus is on  $t = 1$ , which refers to the first iteration
<sup>7</sup>http://athenasc.com/Bivariate-Normal.pdf
Figure 11: Visualization of  $\pmb{f}^t$  taught by GFT under a particular initialization.
Figure 12: The channel-wise visualization of specific  $\pmb{f}^{t}$  corresponding to Figure 11 (b)-(c). The communicated MINT exhibits better performance compared to vanilla MINT.
Figure 13: Visualization of  $\pmb{f}^t$  taught by RFT under a particular initialization.
Figure 14: The channel-wise visualization of specific  $\pmb{f}^{t}$  corresponding to Figure 13 (b)-(c).
Figure 15: Illustration of an extreme case of teaching. The teacher can help the learners directly learn  $\pmb{f}^{*}$  through providing the communication matrix.
(a) Convergence performance
Figure 16: Examination of relation between RFT and GFT.
(b) Visualization of  $f^t$  and  $f^*$
(the results of  $t > 1$  can be derived similarly), where there are two learners attempting to learn separate target functions. We set  $\pmb{f}^{0} = (f_{1},f_{2})^{T} = (x,x^{2})^{T}$  and  $\pmb{f}_{\gamma}^{*} = (\gamma x / 2 + \gamma x^{2} / 2 + (1 - \gamma)\cos x,\gamma x / 3 + 2\gamma x^{2} / 3 + (1 - \gamma)\sin x)^{T},\gamma \in [0,1]$ , and we generate  $\gamma$  by arrange  $(-1,1,0.01)$ . In Figure 20, we present the corresponding  $\| A_{\gamma}\pmb{f}^{0} - \pmb{f}_{\gamma}^{*}\|_{\mathcal{H}^{2}}$  against  $\gamma$ . The observed trend is that as  $\gamma$  increases, the distance between  $A_{\gamma}\pmb{f}^{0}$  and  $\pmb{f}_{\gamma}^{*}$  decreases. This indicates that  $A_{\gamma}$  performs better when  $\pmb{f}^{*}$  can be linearly expressed by  $\pmb{f}^{0}$ , but poorly when  $\pmb{f}^{*}$  cannot be linearly expressed by  $\pmb{f}^{0}$ . Moreover,  $A$  is beneficial when  $\pmb{f}^{*}$  can be partially expressed in a linear manner by  $\pmb{f}^{0}$ .
Figure 17: The visualization of a multivariate mixture Gaussian distribution  $f_{X_1,X_2}^* (x_1,x_2)$ .
Figure 18: The specific learned  $\pmb{f}^{t} = (f_{1}^{t},f_{2}^{t})^{T}$
Figure 19: Convergence performance for the synthetic bivariate mixture Gaussian data.
Figure 20: The distance between  $A_{\gamma}f^{0}$  and  $\pmb{f}_{\gamma}^{*}$  VS.  $\gamma$