SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The learners update  $f^t$  based on received  $A^t = A$  and  $\mathcal{D}^t = \mathcal{D}$ :
$F_{A^t}^{t + 1}\gets A^t\cdot \pmb {f}^t -\pmb {\eta}^t\odot \mathcal{G}(\mathcal{L};A^t\cdot \pmb {f}^t;\mathcal{D}^t).$
Set  $t\gets t + 1$
Further discussion about the idea behind MINT
In practical scenarios, it is a commonly accepted fact that a single object can possess multiple characteristics. For instance, when describing a fruit, we often take into account its various attributes such as shape, color, and texture. This highlights the notion that limiting oneself to scalar-vector functions in nonparametric iterative machine teaching would not be enough to capture the complexity of real-world data. We thus extend the current nonparametric teaching from the single-learner version to a multi-learner one (MINT) by considering vector-valued functions. Compared to parameterized
Figure 5: Comparison between vanilla and communicated MINT.
teaching, MINT can be seen as a type of extension which involves replacing the vector unit from values to functions.
The idea of communicated MINT partially comes from linear algebra. If  $\pmb{f}^0 \in \mathcal{H}^d$  forms a basis of  $\mathcal{H}^d$ , and  $\pmb{f}^*$  can be linearly expressed by this basis, then it is noteworthy that learners can successfully learn  $\pmb{f}^*$  with only one-time communication.
Let us consider a scenario in which the components of  $\pmb{f}^{0}$  are from different function families. Suppose we take  $\pmb{f}^{0} = (e^{x},\sin x,x)^{T}$  and  $\pmb{f}^{*} = (2e^{x} - \sin x - x,\sin x + 2x, - e^{x} + \sin x + x)^{T}$ , it holds that
$$
\binom {2 e ^ {x} - \sin x - x} {\sin x + 2 x} - e ^ {x} + \sin x + x) = \underbrace {\left( \begin{array}{c c c} 2 & - 1 & - 1 \\ 0 & 1 & 2 \\ - 1 & 1 & 1 \end{array} \right)} _ {\text {c o m m u n i c a t i o n m a t r i x A}} \cdot \binom {e ^ {x}} {\sin x}, \tag {18}
$$
which indicates that the teacher can teach  $\pmb{f}^{*}$  to multiple learners through providing the communication matrix  $A$  within one iteration. To draw a practical analogy, we can view  $e^{kx}$  as denoting shape,  $\sin kx$  as indicating color, and  $x^{1/k}$  as representing texture.
Another scenario is when the components of  $\pmb{f}^{0}$  belong to the same function family. Consider the scenario where two learners are familiar with the shapes of a circle and a square, respectively, and they need to learn two different shapes as shown in Figure 5. In this case, we can model it by assuming that  $f_{j}^{0}, j \in \mathbb{N}_{2}$  belong to the cosine family, i.e.,  $\pmb{f}^{0} = (\sqrt{2}\cos (2\pi x),\sqrt{2}\cos (4\pi x))^{T}$ . Here,  $f_{j}^{0}$  are not linearly related, which is a property of basis. The target function is denoted as  $\pmb{f}^{*} = (\sqrt{2}\cos (2\pi x) - \sqrt{2}\cos (4\pi x), - \cos (2\pi x) + \sqrt{2}\cos (4\pi x))^{T}$ . It is trivial to derive that
$$
\binom {\sqrt {2} \cos (2 \pi x) - \sqrt {2} \cos (4 \pi x)} {- \cos (2 \pi x) + \sqrt {2} \cos (4 \pi x)} = \binom {1} {- 1 / \sqrt {2}} \cdot \binom {\sqrt {2} \cos (2 \pi x)} {\sqrt {2} \cos (4 \pi x)}. \tag {19}
$$
By referring to the illustration in Figure 5, we can compare the performance of vanilla MINT and communicated MINT. It can be observed that the communicated MINT is capable of helping to learn  $f^{*}$  within just one iteration, whereas the vanilla MINT requires numerous iterations to achieve the same.
Our suggestion for further reading on functional calculus would be to consult the literature [24, 16] and the references therein.
Proof of Lemma 6 We firstly define a function  $\pmb{q}$  by adding a small disturbance  $\epsilon g$  ( $\epsilon \in \mathbb{R}, g \in \mathcal{H}^d$ ) to  $\pmb{f} \in \mathcal{H}^d$ , that is,  $\pmb{q} = \pmb{f} + \epsilon \pmb{g}$ . We see that  $\pmb{q} \in \mathcal{H}^d$  since vector-valued RKHS is closed under addition and scalar multiplication. For a evaluation functional  $E_{\pmb{x}}[\pmb{f}] = \sum_{i=1}^{d} f_i(x_{i,j_i}) : \mathcal{H}^d \mapsto \mathbb{R}$  where  $\pmb{x} = [x_{i,j_i}]^d \in \mathcal{X}^d$ , we thus can evaluate  $\pmb{q}$  at  $\pmb{x}$  as
$$
\begin{array}{l} E _ {\boldsymbol {x}} [ \boldsymbol {q} ] = E _ {\boldsymbol {x}} [ \boldsymbol {f} + \epsilon \boldsymbol {g} ] \\ = E _ {\boldsymbol {x}} [ \boldsymbol {f} ] + \epsilon E _ {\boldsymbol {x}} [ \boldsymbol {g} ] + 0 \\ = E _ {\boldsymbol {x}} [ \boldsymbol {f} ] + \epsilon \left\langle K (\boldsymbol {x}, \cdot), \boldsymbol {g} \right\rangle_ {\mathcal {H} ^ {d}} + 0 \tag {20} \\ \end{array}
$$
Recall the definition of the Fréchet derivative in a vector-valued reproducing kernel Hilbert space (RKHS), which is defined implicitly and can be found in Definition 1,  $E_{\pmb{x}}[\pmb{f} + \epsilon \pmb{g}] = E_{\pmb{x}}[\pmb{f}] + \epsilon \langle \nabla_{\pmb{f}}E_{\pmb{x}}[\pmb{f}], \pmb{g} \rangle_{\mathcal{H}^d} + \mathcal{O}(\epsilon^2)$ , it follows from Eq. 20 that the gradient of a evaluation functional in vector-valued RKHS  $\mathcal{H}^d$  is  $\nabla_{\pmb{f}}E_{\pmb{x}}[\pmb{f}] = K_{\pmb{x}} = [K_{x_{i,j_i}}]^d$ .
Proof of Lemma 7 Note that one example  $(x,y)\sim \mathbb{Q}_i(x,y)$  can be uniquely identified by its  $x$ , we take expectation on  $x\sim \mathbb{P}(x)$  only for the sake of simplicity. It follows from the convexity of  $\mathcal{L}_i$  and the definition of Fréchet derivative in Definition 1 that we have
$$
\begin{array}{l} \mathbb {E} _ {x _ {i} \sim \mathbb {P} _ {i} (x _ {i})} \left[ \mathcal {L} _ {i} \left(f _ {i} ^ {t + 1}\right) - \mathcal {L} _ {i} \left(f _ {i} ^ {t}\right) \right] \\ \leq \mathbb {E} _ {x _ {i} \sim \mathbb {P} _ {i} (x _ {i})} \left[ \langle f _ {i} ^ {t + 1} - f _ {i} ^ {t}, \nabla_ {f} \mathcal {L} _ {i} (f) | _ {f = f _ {i} ^ {t + 1}} \rangle \mathcal {H} \right] \\ = \mathbb {E} _ {x _ {i} \sim \mathbb {P} _ {i} (x _ {i})} \left[ - \eta_ {i} ^ {t} E _ {x _ {i} ^ {t}} \left[ \nabla_ {f} \mathcal {L} _ {i} (f) | _ {f = f _ {i} ^ {t}} \right] \cdot E _ {x _ {i} ^ {t}} \left[ \nabla_ {f} \mathcal {L} _ {i} (f) | _ {f = f _ {i} ^ {t + 1}} \right] \right] \\ = - \eta_ {i} ^ {t} \cdot \mathbb {E} _ {x _ {i} \sim \mathbb {P} _ {i} (x _ {i})} \left[ E _ {x _ {i} ^ {t}} \left[ \nabla_ {f} \mathcal {L} _ {i} (f) | _ {f = f _ {i} ^ {t}} \cdot \nabla_ {f} \mathcal {L} _ {i} (f) | _ {f = f _ {i} ^ {t + 1}} \right] \right]. \tag {21} \\ \end{array}
$$
Under  $L_{\mathcal{L}}$ -Lipschitz smooth Assumption 3 and bounded kernel function Assumption 4, we can show that
$$
\left. E _ {x _ {i} ^ {t}} \left[ \left. \nabla_ {f} \mathcal {L} _ {i} (f) \right| _ {f = f _ {i} ^ {t}} \cdot \nabla_ {f} \mathcal {L} _ {i} (f) \right| _ {f = f _ {i} ^ {t + 1}} \right] \geq \left(3 / 4 - L _ {\mathcal {L}} ^ {2} (\eta_ {i} ^ {t}) ^ {2} (M _ {K}) ^ {2}\right) E _ {x _ {i} ^ {t}} \left[ \left(\left. \nabla_ {f} \mathcal {L} _ {i} (f) \right| _ {f = f _ {i} ^ {t}}\right) ^ {2} \right]. \tag {22}
$$
For succinctness, we define
$$
\left(\nabla_ {f} \mathcal {L} _ {i} (f) | _ {f = f _ {i} ^ {t}}\right) ^ {2} := \left. \nabla_ {f} ^ {2} \mathcal {L} _ {i} (f) \right| _ {f = f _ {i} ^ {t}} \tag {23}
$$
$$
m _ {i, t} (\dot {x}) := E _ {\dot {x}} \left[ \left. \nabla_ {f} ^ {2} \mathcal {L} _ {i} (f) \right| _ {f = f _ {i} ^ {t}} \right] = E _ {\dot {x}} \left[ \left(\left. \nabla_ {f} \mathcal {L} _ {i} (f) \right| _ {f = f _ {i} ^ {t}}\right) ^ {2} \right]. \tag {24}
$$
Then, we can apply Taylor expansion for  $m_{i,t}(x_i^t)$  on  $\mu_{i} = \mathbb{E}_{x_{i}\sim \mathbb{P}_{i}(x_{i})}(x_{i}^{t})$  and derives
$$
m _ {i, t} \left(x _ {i} ^ {t}\right) = m _ {i, t} \left(\mu_ {i}\right) + m _ {i, t} ^ {\prime} \left(x _ {i} ^ {t}\right) \left(x _ {i} ^ {t} - \mu_ {i}\right) + \frac {m _ {i , t} ^ {\prime \prime} \left(\mu_ {i}\right)}{2} \left(x _ {i} ^ {t} - \mu_ {i}\right) ^ {2} + R _ {2} \left(x _ {i} ^ {t}\right), \tag {25}
$$
where the remainder  $R_{2}(x_{i}^{t})$  usually is omitted, and we assume  $m_{i,t}(x_i^t)$  is 2-times differentiable. We see that evaluated at mean  $\mu_{i}$ ,  $m_{i,t}(\mu_i)$  is monotonically decreasing w.r.t. iteration  $t$ . Combining Eq. 21, 22 and 25, we have
$$
\begin{array}{l} \mathbb {E} _ {x _ {i} \sim \mathbb {P} _ {i} (x _ {i})} \left[ \mathcal {L} _ {i} \left(f _ {i} ^ {t + 1}\right) - \mathcal {L} _ {i} \left(f _ {i} ^ {t}\right) \right] \\ \leq - \eta_ {i} ^ {t} \cdot \mathbb {E} _ {x _ {i} \sim \mathbb {P} _ {i} (x _ {i})} \left[ \left(3 / 4 - L _ {\mathcal {L}} ^ {2} (\eta_ {i} ^ {t}) ^ {2} (M _ {K}) ^ {2}\right) E _ {x _ {i} ^ {t}} \left[ \nabla_ {f} ^ {2} \mathcal {L} _ {i} (f) | _ {f = f _ {i} ^ {t}} \right] \right] \\ = - \eta_ {i} ^ {t} \left(3 / 4 - L _ {\mathcal {L}} ^ {2} \left(\eta_ {i} ^ {t}\right) ^ {2} \left(M _ {K}\right) ^ {2}\right) \cdot \left(m _ {i, t} \left(u _ {i}\right) + \frac {m _ {i , t} ^ {\prime \prime} \left(\mu_ {i}\right)}{2} \sigma_ {i} ^ {2}\right), \tag {26} \\ \end{array}
$$
where  $\sigma_i^2 = \mathbb{E}_{x_i\sim \mathbb{P}_i(x_i)}(x_i^t -\mu_i)^2$  is the variance of  $x_{i}$  . Therefore,
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {x} \sim [ \mathbb {P} _ {i} (x _ {i}) ] ^ {d}} \left[ \mathcal {L} (\boldsymbol {f} ^ {t + 1}) - \mathcal {L} (\boldsymbol {f} ^ {t}) \right] \\ = \mathbb {E} _ {\boldsymbol {x} \sim [ \mathbb {P} _ {i} (x _ {i}) ] ^ {d}} \left[ \sum_ {i = 1} ^ {d} \mathcal {L} _ {i} \left(f _ {i} ^ {t + 1}\right) - \sum_ {i = 1} ^ {d} \mathcal {L} _ {i} \left(f _ {i} ^ {t}\right) \right] \\ = \sum_ {i = 1} ^ {d} \mathbb {E} _ {x _ {i} \sim \mathbb {P} _ {i} (x _ {i})} \left[ \mathcal {L} _ {i} \left(f _ {i} ^ {t + 1}\right) - \mathcal {L} _ {i} \left(f _ {i} ^ {t}\right) \right] \\ \leq \sum_ {i = 1} ^ {d} - \eta_ {i} ^ {t} \left(3 / 4 - L _ {\mathcal {L}} ^ {2} \left(\eta_ {i} ^ {t}\right) ^ {2} \left(M _ {K}\right) ^ {2}\right) \cdot \left(m _ {i, t} \left(\mu_ {i}\right) + \frac {m _ {i , t} ^ {\prime \prime} \left(\mu_ {i}\right)}{2} \sigma_ {i} ^ {2}\right). \tag {27} \\ \end{array}
$$
Hence, if  $\forall i\in \mathbb{N}_d,\eta_i^t\leq \frac{1}{2L_{\mathcal{L}}\cdot M_K}$  , we have
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {x} \sim [ \mathbb {P} _ {i} (x _ {i}) ] ^ {d}} \left[ \mathcal {L} (\boldsymbol {f} ^ {t + 1}) - \mathcal {L} (\boldsymbol {f} ^ {t}) \right] \leq - \frac {\tilde {\eta} ^ {t}}{2} \sum_ {i = 1} ^ {d} \left(m _ {i, t} (\mu_ {i}) + \frac {m _ {i , t} ^ {\prime \prime} (\mu_ {i})}{2} \sigma_ {i} ^ {2}\right) \\ \leq - \frac {\tilde {\eta} ^ {t} d}{2} \cdot \min  _ {i \in \mathbb {N} _ {d}} \left(m _ {i, t} \left(\mu_ {i}\right) + \frac {m _ {i , t} ^ {\prime \prime} \left(\mu_ {i}\right)}{2} \sigma_ {i} ^ {2}\right), \tag {28} \\ \end{array}
$$
where  $\tilde{\eta}^t = \min_{i\in \mathbb{N}_d}\eta_i^t$
Proof of Theorem 8 Recall Lemma 7,  $\forall i\in \mathbb{N}_d,\eta_i^t\leq \frac{1}{2L_{\mathcal{L}}\cdot M_K}$ , we have
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {x} \sim [ \mathbb {P} _ {i} (x _ {i}) ] ^ {d}} \left[ \mathcal {L} (\boldsymbol {f} ^ {t + 1}) - \mathcal {L} (\boldsymbol {f} ^ {t}) \right] \leq - \frac {\tilde {\eta} ^ {t}}{2} \sum_ {i = 1} ^ {d} \left(m _ {i, t} (\mu_ {i}) + \frac {m _ {i , t} ^ {\prime \prime} (\mu_ {i})}{2} \sigma_ {i} ^ {2}\right) \\ \leq - \frac {\tilde {\eta} ^ {t}}{2} d \cdot \min  _ {i \in \mathbb {N} _ {d}} \left(m _ {i, t} \left(\mu_ {i}\right) + \frac {m _ {i , t} ^ {\prime \prime} \left(\mu_ {i}\right)}{2} \sigma_ {i} ^ {2}\right), \tag {29} \\ \end{array}
$$
where  $\tilde{\eta}^t = \min_{i\in \mathbb{N}_d}\eta_i^t$  . Rearranging above, we have:
$$
\frac {2 \mathbb {E} _ {\boldsymbol {x} \sim [ \mathbb {P} _ {i} (x _ {i}) ] ^ {d}} \left[ \mathcal {L} (\boldsymbol {f} ^ {t}) - \mathcal {L} (\boldsymbol {f} ^ {t + 1}) \right]}{d \tilde {\eta} ^ {t}} \geq \min  _ {i \in \mathbb {N} _ {d}} \left(m _ {i, t} (\mu_ {i}) + \frac {m _ {i , t} ^ {\prime \prime} (\mu_ {i})}{2} \sigma_ {i} ^ {2}\right). \tag {30}
$$
Equivalently, replace index  $t$  by  $l$ ,  $\frac{2\mathbb{E}_{\boldsymbol{x}\sim[\mathbb{P}_i(x_i)]^d}\left[\mathcal{L}(\boldsymbol{f}^l) - \mathcal{L}(\boldsymbol{f}^{l + 1})\right]}{d\bar{\eta}^l}\geq \min_{i\in \mathbb{N}_d}\Big(m_{i,l}(\mu_i) + \frac{m_{i,l}^{\prime\prime}(\mu_i)}{2}\sigma_i^2\Big)$ . Consequently, plugging  $l = 0,1\dots ,t - 1$  in it and summing them up, we hence have
$$
\begin{array}{l} \sum_ {l = 0} ^ {t - 1} \min  _ {i \in \mathbb {N} _ {d}} \left(m _ {i, l} (\mu_ {i}) + \frac {m _ {i , l} ^ {\prime \prime} (\mu_ {i})}{2} \sigma_ {i} ^ {2}\right) \\ \leq \frac {2}{d} \sum_ {l = 0} ^ {t - 1} \frac {\mathbb {E} _ {\boldsymbol {x} \sim [ \mathbb {P} _ {i} (x _ {i}) ] ^ {d}} \left[ \mathcal {L} (\boldsymbol {f} ^ {l}) - \mathcal {L} (\boldsymbol {f} ^ {l + 1}) \right]}{\tilde {\eta} ^ {l}} \\ \leq \frac {2}{d \dot {\eta}} \sum_ {l = 0} ^ {t - 1} \mathbb {E} _ {\boldsymbol {x} \sim [ \mathbb {P} _ {i} (x _ {i}) ] ^ {d}} \left[ \mathcal {L} (\boldsymbol {f} ^ {l}) - \mathcal {L} (\boldsymbol {f} ^ {l + 1}) \right], \tag {31} \\ \end{array}
$$
where  $\dot{\eta} = \min_{l\in \{0\} \bigcup \mathbb{N}_{t - 1}}\tilde{\eta}^l >0$  Expanding the r.h.s. term in Eq. 31 yields
$$
\begin{array}{l} \frac {2}{d \dot {\eta}} \sum_ {l = 0} ^ {t - 1} \mathbb {E} _ {\boldsymbol {x} \sim [ \mathbb {P} _ {i} (x _ {i}) ] ^ {d}} \left[ \mathcal {L} (\boldsymbol {f} ^ {l}) - \mathcal {L} (\boldsymbol {f} ^ {l + 1}) \right] \\ = \frac {2}{d \dot {\eta}} \mathbb {E} _ {\boldsymbol {x} \sim [ \mathbb {P} _ {i} (x _ {i}) ] ^ {d}} \left[ \mathcal {L} (\boldsymbol {f} ^ {0}) - \mathcal {L} (\boldsymbol {f} ^ {t}) \right] \\ \leq \frac {2}{d \dot {\eta}} \mathbb {E} _ {\boldsymbol {x} \sim \left[ \mathbb {P} _ {i} \left(x _ {i}\right) \right] ^ {d}} \left[ \mathcal {L} \left(\boldsymbol {f} ^ {0}\right) \right]. \tag {32} \\ \end{array}
$$
In terms of the l.h.s. term in Eq. 31, we must have
$$
\sum_ {l = 0} ^ {t - 1} \min  _ {i \in \mathbb {N} _ {d}} \left(m _ {i, l} \left(\mu_ {i}\right) + \frac {m _ {i , l} ^ {\prime \prime} \left(\mu_ {i}\right)}{2} \sigma_ {i} ^ {2}\right) \geq t \cdot \min  _ {l \in \{0 \} \bigcup \mathbb {N} _ {t - 1}} \min  _ {i \in \mathbb {N} _ {d}} \left(m _ {i, l} \left(\mu_ {i}\right) + \frac {m _ {i , l} ^ {\prime \prime} \left(\mu_ {i}\right)}{2} \sigma_ {i} ^ {2}\right). \tag {33}
$$
Combining expression 32 and 33, we thus have
$$
\begin{array}{l} t \cdot \min  _ {l \in \{0 \}} \min  _ {\mathbb {N} _ {t - 1}} \min  _ {i \in \mathbb {N} _ {d}} \left(m _ {i, l} (\mu_ {i}) + \frac {m _ {i , l} ^ {\prime \prime} (\mu_ {i})}{2} \sigma_ {i} ^ {2}\right) \leq \sum_ {l = 0} ^ {t - 1} \min  _ {i \in \mathbb {N} _ {d}} \left(m _ {i, l} (\mu_ {i}) + \frac {m _ {i , l} ^ {\prime \prime} (\mu_ {i})}{2} \sigma_ {i} ^ {2}\right) \\ \leq \frac {2}{d \dot {\eta}} \mathbb {E} _ {\boldsymbol {x} \sim [ \mathbb {P} _ {i} (x _ {i}) ] ^ {d}} \left[ \mathcal {L} (\boldsymbol {f} ^ {0}) \right]. \tag {34} \\ \end{array}
$$
Since  $m_{i,t}(\mu_i)$  is monotonically decreasing w.r.t. iteration  $t$ , we have
$$
\min  _ {l \in \{0 \} \bigcup \mathbb {N} _ {t - 1}} \min  _ {i \in \mathbb {N} _ {d}} \left(m _ {i, l} (\mu_ {i}) + m _ {i, l} ^ {\prime \prime} (\mu_ {i}) \sigma_ {i} ^ {2} / 2\right) = \min  _ {i \in \mathbb {N} _ {d}} \left(m _ {i, t - 1} (\mu_ {i}) + m _ {i, t - 1} ^ {\prime \prime} (\mu_ {i}) \sigma_ {i} ^ {2} / 2\right).
$$
Therefore, we can derive
$$
\min  _ {i \in \mathbb {N} _ {d}} \left(m _ {i, t - 1} \left(\mu_ {i}\right) + \frac {m _ {i , t - 1} ^ {\prime \prime} \left(\mu_ {i}\right)}{2} \sigma_ {i} ^ {2}\right) \leq \frac {2}{d \dot {\eta} t} \mathbb {E} _ {\boldsymbol {x} \sim [ \mathbb {P} _ {i} (x _ {i}) ] ^ {d}} \left[ \mathcal {L} \left(\boldsymbol {f} ^ {0}\right) \right], \tag {35}
$$
when the returned vector-valued model for multiple learners is  $\pmb{f}^t$  and index  $t - 1$  denotes the last iteration.
On the other hand, it follows from Eq. 32 and the fact that  $m_{i,t}(\mu_i)$  is monotonically decreasing w.r.t. iteration  $t$  that we have
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {x} \sim [ \mathbb {P} _ {i} (x _ {i}) ] ^ {d}} \left[ \mathcal {L} (\boldsymbol {f} ^ {t}) - \mathcal {L} (\boldsymbol {f} ^ {0}) \right] \leq - \frac {d \dot {\eta}}{2} \sum_ {l = 0} ^ {t - 1} \min  _ {i \in \mathbb {N} _ {d}} \left(m _ {i, l} (\mu_ {i}) + \frac {m _ {i , l} ^ {\prime \prime} (\mu_ {i})}{2} \sigma_ {i} ^ {2}\right) \\ \leq - \frac {d \dot {\eta}}{2} t \cdot \min  _ {i \in \mathbb {N} _ {d}} \left(m _ {i, t - 1} \left(\mu_ {i}\right) + \frac {m _ {i , t - 1} ^ {\prime \prime} \left(\mu_ {i}\right)}{2} \sigma_ {i} ^ {2}\right). \tag {36} \\ \end{array}
$$
After rearranging, we obtain
$$
\begin{array}{l} \min  _ {i \in \mathbb {N} _ {d}} \left(m _ {i, t - 1} (\mu_ {i}) + \frac {m _ {i , t - 1} ^ {\prime \prime} (\mu_ {i})}{2} \sigma_ {i} ^ {2}\right) \leq \frac {2}{d \dot {\eta} t} \cdot \mathbb {E} _ {\boldsymbol {x} \sim [ \mathbb {P} _ {i} (x _ {i}) ] ^ {d}} \left[ \mathcal {L} (\boldsymbol {f} ^ {0}) - \mathcal {L} (\boldsymbol {f} ^ {t}) \right] \\ \leq \frac {2}{d \dot {\eta} t} \cdot \mathbb {E} _ {\boldsymbol {x} \sim [ \mathbb {P} _ {i} (x _ {i}) ] ^ {d}} \left[ \mathcal {L} \left(\boldsymbol {f} ^ {0}\right) \right] \tag {37} \\ \end{array}
$$
Let r.h.s. of Eq. 37 be controlled by a small constant  $\epsilon >0$ , we have
$$
t \geq \frac {2}{d \dot {\eta} \epsilon} \cdot \mathbb {E} _ {\boldsymbol {x} \sim [ \mathbb {P} _ {i} (x _ {i}) ] ^ {d}} \left[ \mathcal {L} (\boldsymbol {f} ^ {0}) \right], \tag {38}
$$
which means given a small constant  $\epsilon > 0$  it would take approximately
$$
\frac {2}{d \dot {\eta} \epsilon} \cdot \mathbb {E} _ {\boldsymbol {x} \sim [ \mathbb {P} _ {i} (x _ {i}) ] ^ {d}} \left[ \mathcal {L} (\boldsymbol {f} ^ {0}) \right]
$$
iterations to reach a stationary point.
Additionally, this suggests that multiple learners could achieve the stationary state as: In each iteration, check if  $m_{i,t-1}(\mu_i) + \frac{m_{i,t-1}''(\mu_i)}{2}\sigma_i^2$  is small enough. Assuming this condition is satisfied, the learners will have already reduced the multi-learner loss to an acceptably low level, allowing them to send a signal indicating termination back to the teachers. If the condition is not fulfilled, the teachers will continue with the process. The termination occurs within  $\frac{2}{d\eta\epsilon} \cdot \mathbb{E}_{\boldsymbol{x} \sim [\mathbb{P}_i(x_i)]^d}[\mathcal{L}(\boldsymbol{f}^0)]$  iterations.
Proof of Lemma 9 Recall practical Greedy Functional Teaching in Eq. 12
$$
\left(\boldsymbol {x} ^ {t ^ {*}} = \underset {\boldsymbol {x} ^ {t} \in \mathcal {X}} {\arg \max } \left| E _ {\boldsymbol {x} ^ {t}} \left[ \nabla_ {f} \mathcal {L} (f) \right| _ {f = f ^ {t}} \right] \mid , y ^ {*} = E _ {\boldsymbol {x} ^ {t ^ {*}}} [ f ^ {*} ]\right). \tag {39}
$$
Obviously, it is trivial to see that  $\forall \pmb{x}^t\in \mathcal{X}$
$$
\left| E _ {\boldsymbol {x} ^ {t *}} \left[ \nabla_ {f} \mathcal {L} (f) | _ {f = f ^ {t}} \right] \right| ^ {2} \geq \left| E _ {\boldsymbol {x} ^ {t}} \left[ \nabla_ {f} \mathcal {L} (f) | _ {f = f ^ {t}} \right] \right| ^ {2}. \tag {40}
$$
Analogous to the Proof of Lemma 7 in B, we can derive
$$
\begin{array}{l} \mathbb {E} _ {x _ {i} \sim \mathbb {P} _ {i} (x _ {i})} \left[ \mathcal {L} _ {i} \left(f _ {i} ^ {t + 1}\right) - \mathcal {L} _ {i} \left(f _ {i} ^ {t}\right) \right] \\ \leq - \eta_ {i} ^ {t} \cdot \mathbb {E} _ {x _ {i} \sim \mathbb {P} _ {i} (x _ {i})} \left[ E _ {x _ {i} ^ {t *}} \left[ \nabla_ {f} \mathcal {L} _ {i} (f) | _ {f = f _ {i} ^ {t}} \cdot \nabla_ {f} \mathcal {L} _ {i} (f) | _ {f = f _ {i} ^ {t + 1}} \right] \right] \\ \stackrel {*} {=} - \eta_ {i} ^ {t} \cdot E _ {x _ {i} ^ {t *}} \left[ \nabla_ {f} \mathcal {L} _ {i} (f) | _ {f = f _ {i} ^ {t}} \cdot \nabla_ {f} \mathcal {L} _ {i} (f) | _ {f = f _ {i} ^ {t + 1}} \right] \\ \leq - \eta_ {i} ^ {t} \cdot \left(3 / 4 - L _ {\mathcal {L}} ^ {2} (\eta_ {i} ^ {t}) ^ {2} (M _ {K}) ^ {2}\right) E _ {x _ {i} ^ {t *}} \left[ \nabla_ {f} ^ {2} \mathcal {L} _ {i} (f) \big | _ {f = f _ {i} ^ {t}} \right] \\ = - \eta_ {i} ^ {t} \cdot \left(3 / 4 - L _ {\mathcal {L}} ^ {2} (\eta_ {i} ^ {t}) ^ {2} (M _ {K}) ^ {2}\right) m _ {i, t} \left(x _ {i} ^ {t ^ {*}}\right), \tag {41} \\ \end{array}
$$
where  $\stackrel{*}{=}$  holds because selected  $x_{i}^{t^{*}}$  by GFT is determined. Therefore,
$$
\mathbb {E} _ {\boldsymbol {x} \sim [ \mathbb {P} _ {i} (x _ {i}) ] ^ {d}} \left[ \mathcal {L} (\boldsymbol {f} ^ {t + 1}) - \mathcal {L} (\boldsymbol {f} ^ {t}) \right] \leq \sum_ {i = 1} ^ {d} - \eta_ {i} ^ {t} \left(3 / 4 - L _ {\mathcal {L}} ^ {2} \left(\eta_ {i} ^ {t}\right) ^ {2} \left(M _ {K}\right) ^ {2}\right) m _ {i, t} \left(x _ {i} ^ {t ^ {*}}\right). \tag {42}
$$
$$
\mathbb {E} _ {\boldsymbol {x} \sim [ \mathbb {P} _ {i} (x _ {i}) ] ^ {d}} \left[ \mathcal {L} (\boldsymbol {f} ^ {t + 1}) - \mathcal {L} (\boldsymbol {f} ^ {t}) \right] \leq - \frac {\tilde {\eta} ^ {t}}{2} \sum_ {i = 1} ^ {d} m _ {i, t} \left(x _ {i} ^ {t *}\right) \leq - \frac {\tilde {\eta} ^ {t} d}{2} \cdot \min  _ {i \in \mathbb {N} _ {d}} m _ {i, t} \left(x _ {i} ^ {t *}\right), \tag {43}
$$
where  $\tilde{\eta}^t = \min_{i\in \mathbb{N}_d}\eta_i^t$  . Note that GFT is to select examples such that the gradient is the steepest, thus  $\min_{i\in \mathbb{N}_d}m_{i,t}(x_i^{t*})\geq \min_{i\in \mathbb{N}_d}\left(m_{i,t}(\mu_i) + \frac{m_{i,t}^{\prime\prime}(\mu_i)}{2}\sigma_i^2\right)$
Proof of Theorem 10 Recall the result of Lemma 7, when  $\forall i\in \mathbb{N}_d,\eta_i^t\leq \frac{1}{2L_\mathcal{L}\cdot M_K}$
$$
\sum_ {i = 1} ^ {d} \left(m _ {i, t} \left(\mu_ {i}\right) + \frac {m _ {i , t} ^ {\prime \prime} \left(\mu_ {i}\right)}{2} \sigma_ {i} ^ {2}\right) \leq \frac {2 \mathbb {E} _ {\boldsymbol {x} \sim \left[ \mathbb {P} _ {i} \left(x _ {i}\right) \right] ^ {d}} \left[ \mathcal {L} \left(\boldsymbol {f} ^ {t}\right) - \mathcal {L} \left(\boldsymbol {f} ^ {t + 1}\right) \right]}{\tilde {\eta} ^ {t}}, \tag {44}
$$
Before converging to the stationary state,  $\sum_{i=1}^{d} m_{i,t}(x_i^{t^*}) > 0$ . Therefore, we can express it as
$$
\sum_ {i = 1} ^ {d} m _ {i, t} \left(x _ {i} ^ {t *}\right) \cdot \frac {\sum_ {i = 1} ^ {d} \left(m _ {i , t} \left(\mu_ {i}\right) + \frac {m _ {i , t} ^ {\prime \prime} \left(\mu_ {i}\right)}{2} \sigma_ {i} ^ {2}\right)}{\sum_ {i = 1} ^ {d} m _ {i , t} \left(x _ {i} ^ {t *}\right)} \leq \frac {2 \mathbb {E} _ {\boldsymbol {x}} \sim [ \mathbb {P} _ {i} \left(x _ {i}\right) ] ^ {d} \left[ \mathcal {L} \left(\boldsymbol {f} ^ {t}\right) - \mathcal {L} \left(\boldsymbol {f} ^ {t + 1}\right) \right]}{\tilde {\eta} ^ {t}}. \tag {45}
$$
We see that  $\frac{\sum_{i=1}^{d}(m_{i,t}(\mu_i) + \frac{m_{i,t}^{\prime\prime}(\mu_i)}{2}\sigma_i^2)}{\sum_{i=1}^{d}m_{i,t}(x_i^{t*})} \leq 1$  measures the difference between two algorithms at  $t$  iteration. And this is deterministic in each iteration, that is, this can be estimated before sampling, so we can see the superiority of GFT.
$$
\begin{array}{l} \frac {\sum_ {i = 1} ^ {d} (m _ {i , t} (\mu_ {i}) + \frac {m _ {i , t} ^ {\prime \prime} (\mu_ {i})}{2} \sigma_ {i} ^ {2})}{\sum_ {i = 1} ^ {d} m _ {i , t} (x _ {i} ^ {t *})} \\ = 1 - \frac {\sum_ {i = 1} ^ {d} \left(m _ {i , t} \left(x _ {i} ^ {t ^ {*}}\right) - m _ {i , t} \left(\mu_ {i}\right) - \frac {m _ {i , t} ^ {\prime \prime} \left(\mu_ {i}\right)}{2} \sigma_ {i} ^ {2}\right)}{\sum_ {i = 1} ^ {d} m _ {i , t} \left(x _ {i} ^ {t ^ {*}}\right)} \\ \geq 1 - \frac {\sum_ {i = 1} ^ {d} \left(\| x _ {i} ^ {t ^ {*}} - \mu_ {i} \| _ {2} - \frac {m _ {i , t} ^ {\prime \prime} (\mu_ {i})}{2} \sigma_ {i} ^ {2}\right)}{\sum_ {i = 1} ^ {d} m _ {i , t} (x _ {i} ^ {t ^ {*}})} \\ \geq 1 - \frac {\sum_ {i = 1} ^ {d} \left(\left\| x _ {i} ^ {t ^ {*}} - \mu_ {i} \right\| _ {2}\right)}{\sum_ {i = 1} ^ {d} m _ {i , t} \left(x _ {i} ^ {t ^ {*}}\right)} \tag {46} \\ \end{array}
$$
where we assume  $m_{i,t}(\dot{x})$  is Lipschitz continuous w.r.t. input  $\dot{x}$  to tract the relation between this quantity and the distance between  $\mu_i$  and  $x_i^{t^*}$ . Now, we have
$$
1 - \frac {\sum_ {i = 1} ^ {d} \left(\left\| x _ {i} ^ {t ^ {*}} - \mu_ {i} \right\| _ {2}\right)}{\sum_ {i = 1} ^ {d} m _ {i , t} \left(x _ {i} ^ {t ^ {*}}\right)} \leq \frac {\sum_ {i = 1} ^ {d} \left(m _ {i , t} \left(\mu_ {i}\right) + \frac {m _ {i , t} ^ {\prime \prime} \left(\mu_ {i}\right)}{2} \sigma_ {i} ^ {2}\right)}{\sum_ {i = 1} ^ {d} m _ {i , t} \left(x _ {i} ^ {t ^ {*}}\right)} \leq 1. \tag {47}
$$