SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {P} ^ {+} (\tau) = \mathbb {P} \left(\left\{\mathcal {M} _ {\vartheta} ^ {- 1} (\ell) \mid \ell \left(\mathfrak {D} _ {\tau} ^ {T}, \mathfrak {D} _ {\tau} ^ {C}; \vartheta\right) \geq \xi_ {\alpha}, \tau \in \Omega_ {\tau} \right\}\right). \tag {17c}
$$
$$
\varphi_ {\alpha} \left(\hat {\xi} _ {\alpha}; \vartheta\right) - \varphi_ {\alpha} \left(\xi_ {\alpha}; \vartheta\right) = \hat {\xi} _ {\alpha} - \xi_ {\alpha} \tag {18a}
$$
$$
+ \frac {1}{1 - \alpha} \mathbb {E} _ {\tau \sim \mathbb {P} ^ {-} (\tau)} \left[ \ell \left(\mathfrak {D} _ {\tau} ^ {T}, \mathfrak {D} _ {\tau} ^ {C}; \vartheta\right) - \hat {\xi} _ {\alpha} \right] + \frac {1}{1 - \alpha} \mathbb {E} _ {\tau \sim \mathbb {P} ^ {+} (\tau)} \left[ \xi_ {\alpha} - \hat {\xi} _ {\alpha} \right] \tag {18b}
$$
$$
\leq - \delta + \frac {1}{1 - \alpha} \mathbb {E} _ {\tau \sim \mathbb {P} ^ {-} (\tau)} [ \delta ] + \frac {1}{1 - \alpha} \mathbb {E} _ {\tau \sim \mathbb {P} ^ {+} (\tau)} [ \delta ] \leq - \delta + \frac {1}{1 - \alpha} \delta = \frac {\alpha}{1 - \alpha} \delta . \tag {18c}
$$
Based on the inequalities (16)/(18), we can have  $\kappa_{\alpha} = \max \left\{\frac{2 - \alpha}{1 - \alpha}, \frac{\alpha}{1 - \alpha}\right\}$  such that the proposition is verified as  $\varphi_{\alpha}(\hat{\xi}_{\alpha}; \vartheta) - \kappa_{\alpha} \delta < \varphi_{\alpha}(\xi_{\alpha}; \vartheta)$ .
H Proof of Improvement Guarantee in Theorem (1)
Theorem (1): Under assumptions (1)/(2)/(3), suppose that the estimate error with the crude Monte Carlo holds:  $|\hat{\xi}_{\alpha_t} - \xi_{\alpha_t}| \leq \frac{\lambda}{\beta_\ell(1 - \alpha)^2}, \forall t \in \mathbb{N}^+$ , with the subscript  $t$  the iteration number,  $\lambda$  the learning rate in stochastic gradient descent,  $\beta_\ell$  the Lipschitz constant of the risk cumulative distribution function, and  $\alpha$  the confidence level. Then the proposed heuristic algorithm with the crude Monte Carlo can produce at least a local optimum for distributionally robust fast adaptation.
Proof: In the main paper, Fig. (2) provides the outline of the improvement guarantee proof. Note that  $\hat{\xi}_{\alpha_t}$  is an estimate of  $\xi_{\alpha_t}$  with the help of Monte Carlo samples, and this depends on the model parameters  $\vartheta_t$  in optimization. Performing the gradient updates w.r.t. the surrogate function  $\varphi_{\alpha}(\hat{\xi}_{\alpha};\vartheta)$ , we can have the following equation with a small step-size learning rate  $\lambda$ :
$$
\text {G r a d i e n t D e s c e n t}: \vartheta_ {t + 1} = \vartheta_ {t} - \lambda \nabla_ {\vartheta} \varphi_ {\alpha} (\hat {\xi} _ {\alpha}; \vartheta) \tag {19}
$$
$$
\Rightarrow \text {M o n o t o n i c S e q u e n c e}: \varphi_ {\alpha} \left(\dot {\xi} _ {\alpha_ {t}}; \vartheta_ {t + 1}\right) \leq \varphi_ {\alpha} \left(\dot {\xi} _ {\alpha_ {t}}; \vartheta_ {t}\right).
$$
To verify the improvement guarantee, we need to show that with the meta model parameters derived from the surrogate function:
$$
\varphi_ {\alpha} \left(\xi_ {\alpha_ {t + 1}}; \vartheta_ {t + 1}\right) \leq \varphi_ {\alpha} \left(\xi_ {\alpha_ {t}}; \vartheta_ {t}\right). \tag {20}
$$
With the previous deduction  $\varphi_{\alpha}(\xi_{\alpha_{t + 1}};\vartheta_{t + 1})\leq \varphi_{\alpha}(\xi_{\alpha_t};\vartheta_{t + 1})$  from the property of  $\mathrm{CVaR}_{\alpha}$ , the demonstration is equivalently reduced to show that:
$$
\varphi_ {\alpha} \left(\xi_ {\alpha_ {t}}; \vartheta_ {t + 1}\right) \leq \varphi_ {\alpha} \left(\xi_ {\alpha_ {t}}; \vartheta_ {t}\right). \tag {21}
$$
We can perform the one order Taylor expansion with Peano's form of remainders w.r.t.  $\varphi_{\alpha}(\xi_{\alpha_t};\vartheta)$  around the point  $\vartheta_t$ :
$$
\begin{array}{l} \varphi_ {\alpha} \left(\xi_ {\alpha_ {t}}; \vartheta_ {t + 1}\right) = \varphi_ {\alpha} \left(\xi_ {\alpha_ {t}}; \vartheta_ {t}\right) - \lambda \left[ \nabla_ {\vartheta} \varphi_ {\alpha} \left(\hat {\xi} _ {\alpha_ {t}}; \vartheta\right) | _ {\vartheta = \vartheta_ {t}} ^ {T} \cdot \nabla_ {\vartheta} \varphi_ {\alpha} \left(\xi_ {\alpha_ {t}}; \vartheta\right) | _ {\vartheta = \vartheta_ {t}} \right] \tag {22} \\ + \mathcal {O} \left(\left| \vartheta_ {t + 1} - \vartheta_ {t} \right|\right) \leq \varphi_ {\alpha} \left(\xi_ {\alpha_ {t}}; \vartheta_ {t}\right). \\ \end{array}
$$
In the case when  $\hat{\xi}_{\alpha} = \xi_{\alpha} + \delta$  with  $\delta \in \mathbb{R}^{+}$ , we use the partitioned task probability space in Eq. (15) and can derive the gradient estimate:
$$
\begin{array}{l} \left. \nabla_ {\vartheta} \varphi_ {\alpha} \left(\hat {\xi} _ {\alpha_ {t}}; \vartheta\right) \right| _ {\vartheta = \vartheta_ {t}} ^ {T} = \frac {1}{1 - \alpha} \left[ \mathbb {E} _ {\tau \sim \mathbb {P} ^ {+} (\tau)} \left[ \nabla_ {\vartheta} \ell \left(\mathfrak {D} _ {\tau} ^ {T}, \mathfrak {D} _ {\tau} ^ {C}; \vartheta\right) \mid_ {\vartheta = \vartheta_ {t}} \right] \right] \tag {23} \\ + \frac {1}{1 - \alpha} \left[ \mathbb {E} _ {\tau \sim \mathbb {P} ^ {+ +} (\tau)} \left[ \nabla_ {\vartheta} \ell \left(\mathfrak {D} _ {\tau} ^ {T}, \mathfrak {D} _ {\tau} ^ {C}; \vartheta\right) | _ {\vartheta = \vartheta_ {t}} \right] \right]. \\ \end{array}
$$
With  $|F_{\ell}(\hat{\xi}_{\alpha};\vartheta) - F_{\ell}(\xi_{\alpha};\vartheta)| \leq \beta_{\ell}\delta$  in the Assumption (2),
$$
- \mathbb {E} _ {\tau \sim \mathbb {P} ^ {+} (\tau)} \left[ \nabla_ {\vartheta} \ell \left(\mathfrak {D} _ {\tau} ^ {T}, \mathfrak {D} _ {\tau} ^ {C}; \vartheta\right) | _ {\vartheta = \vartheta_ {t}} \right] ^ {T} \cdot \mathbb {E} _ {\tau \sim \mathbb {P} ^ {+ +} (\tau)} \left[ \nabla_ {\vartheta} \ell \left(\mathfrak {D} _ {\tau} ^ {T}, \mathfrak {D} _ {\tau} ^ {C}; \vartheta\right) | _ {\vartheta = \vartheta_ {t}} \right] \tag {24a}
$$
$$
\leq \left|\left| \mathbb {E} _ {\tau \sim \mathbb {P} ^ {+} (\tau)} \right| \nabla_ {\vartheta} \ell \left(\mathfrak {D} _ {\tau} ^ {T}, \mathfrak {D} _ {\tau} ^ {C}; \vartheta\right)\right| _ {\vartheta = \vartheta_ {t}} \left. \right| | | \cdot \left|\left| \mathbb {E} _ {\tau \sim \mathbb {P} ^ {+ +} (\tau)} \right| \nabla_ {\vartheta} \ell \left(\mathfrak {D} _ {\tau} ^ {T}, \mathfrak {D} _ {\tau} ^ {C}; \vartheta\right)\right| _ {\vartheta = \vartheta_ {t}} \left. \right| | | _ {(2 4 b)}
$$
$$
\begin{array}{l} \leq \mathbb {E} _ {\tau \sim \mathbb {P} ^ {+} (\tau)} \left[ \sup  _ {\tau \in \Omega_ {\tau}} | | \nabla_ {\vartheta} \ell \left(\mathfrak {D} _ {\tau} ^ {T}, \mathfrak {D} _ {\tau} ^ {C}; \vartheta\right) | _ {\vartheta = \vartheta_ {t}} | | \right] \cdot \mathbb {E} _ {\tau \sim \mathbb {P} ^ {+ +} (\tau)} \left[ \sup  _ {\tau \in \Omega_ {\tau}} | | \nabla_ {\vartheta} \ell \left(\mathfrak {D} _ {\tau} ^ {T}, \mathfrak {D} _ {\tau} ^ {C}; \vartheta\right) | _ {\vartheta = \vartheta_ {t}} | | \right] (24c) \\ \leq \beta_ {\ell} \delta (1 - \alpha) \left(\sup  _ {\tau \in \Omega_ {\tau}} | | \nabla_ {\vartheta} \ell \left(\mathfrak {D} _ {\tau} ^ {T}, \mathfrak {D} _ {\tau} ^ {C}; \vartheta\right) | _ {\vartheta = \vartheta_ {t}} | |\right) ^ {2} = \beta_ {\ell} \delta (1 - \alpha) \mu^ {2}, (24d) \\ \end{array}
$$
where  $\mu$  defines the sup  $\tau \in \Omega_{\tau}||\nabla_{\vartheta}\ell (\mathfrak{D}_{\tau}^{T},\mathfrak{D}_{\tau}^{C};\vartheta)|_{\vartheta = \vartheta_{t}}||$
Then we can derive the following inequality with the deduction from Eq. (24):
$$
\begin{array}{l} \varphi_ {\alpha} \left(\xi_ {\alpha_ {t}}; \vartheta_ {t + 1}\right) = \varphi_ {\alpha} \left(\xi_ {\alpha_ {t}}; \vartheta_ {t}\right) (25a) \\ - \lambda \left[ \nabla_ {\vartheta} \varphi_ {\alpha} \left(\hat {\xi} _ {\alpha_ {t}}; \vartheta\right) \right| _ {\vartheta = \vartheta_ {t}} ^ {T} \cdot \nabla_ {\vartheta} \varphi_ {\alpha} \left(\xi_ {\alpha_ {t}}; \vartheta\right) | _ {\vartheta = \vartheta_ {t}} \bigg ] + \mathcal {O} \left(\left| \vartheta_ {t + 1} - \vartheta_ {t} \right|\right) (25b) \\ = \varphi_ {\alpha} \left(\xi_ {\alpha_ {t}}; \vartheta_ {t}\right) (25c) \\ \end{array}
$$
$$
\begin{array}{l} - \frac {\lambda}{(1 - \alpha) ^ {2}} \left[ \mathbb {E} _ {\tau \sim \mathbb {P} ^ {+ +} (\tau)} \left[ \nabla_ {\vartheta} \ell \left(\mathfrak {D} _ {\tau} ^ {T}, \mathfrak {D} _ {\tau} ^ {C}; \vartheta\right) | _ {\vartheta = \vartheta_ {t}} \right] ^ {T} \cdot \mathbb {E} _ {\tau \sim \mathbb {P} ^ {+ +} (\tau)} \left[ \nabla_ {\vartheta} \ell \left(\mathfrak {D} _ {\tau} ^ {T}, \mathfrak {D} _ {\tau} ^ {C}; \vartheta\right) | _ {\vartheta = \vartheta_ {t}} \right] \right] (25d) \\ - \frac {\lambda}{(1 - \alpha) ^ {2}} \left[ \mathbb {E} _ {\tau \sim \mathbb {P} ^ {+} (\tau)} \left[ \nabla_ {\vartheta} \ell \left(\mathfrak {D} _ {\tau} ^ {T}, \mathfrak {D} _ {\tau} ^ {C}; \vartheta\right) | _ {\vartheta = \vartheta_ {t}} \right] ^ {T} \cdot \mathbb {E} _ {\tau \sim \mathbb {P} ^ {+ +} (\tau)} \left[ \nabla_ {\vartheta} \ell \left(\mathfrak {D} _ {\tau} ^ {T}, \mathfrak {D} _ {\tau} ^ {C}; \vartheta\right) | _ {\vartheta = \vartheta_ {t}} \right] \right] (25e) \\ + \mathcal {O} \left(\left| \vartheta_ {t + 1} - \vartheta_ {t} \right|\right) \quad (2 5 f) \\ \leq \varphi_ {\alpha} \left(\xi_ {\alpha_ {t}}; \vartheta_ {t}\right) - \frac {\lambda \left| \left| \nu_ {1} \right| \right| _ {2} ^ {2}}{(1 - \alpha) ^ {2}} + \beta_ {\ell} \delta (1 - \alpha) \mu^ {2} + \mathcal {O} \left(\left| \vartheta_ {t + 1} - \vartheta_ {t} \right|\right), (25g) \\ \end{array}
$$
where  $\nu_{1} = \mathbb{E}_{\tau \sim \mathbb{P}^{+ + }(\tau)}\Big[\nabla_{\vartheta}\ell (\mathfrak{D}_{\tau}^{T},\mathfrak{D}_{\tau}^{C};\vartheta)|_{\vartheta = \vartheta_{t}}\Big].$
To ensure the existence of improvement guarantee, we need that the following inequality holds:
$$
\varphi_ {\alpha} \left(\xi_ {\alpha_ {t}}; \vartheta_ {t}\right) - \frac {\lambda \left| \left| \nu_ {1} \right| \right| _ {2} ^ {2}}{(1 - \alpha) ^ {2}} + \beta_ {\ell} \delta (1 - \alpha) \mu^ {2} + \mathcal {O} \left(\left| \vartheta_ {t + 1} - \vartheta_ {t} \right|\right) \leq \varphi_ {\alpha} \left(\xi_ {\alpha_ {t}}; \vartheta_ {t}\right). \tag {26}
$$
Similarly, in the case when  $\hat{\xi}_{\alpha} = \xi_{\alpha} - \delta$  with  $\delta \in \mathbb{R}^{+}$ , we use the probability space of partitioned tasks in Eq. (17) and can derive the gradient estimate:
$$
\begin{array}{l} \left. \nabla_ {\vartheta} \varphi_ {\alpha} \left(\hat {\xi} _ {\alpha_ {t}}; \vartheta\right) \right| _ {\vartheta = \vartheta_ {t}} ^ {T} = \frac {1}{1 - \alpha} \left[ \mathbb {E} _ {\tau \sim \mathbb {P} ^ {-} (\tau)} \left[ \nabla_ {\vartheta} \ell \left(\mathfrak {D} _ {\tau} ^ {T}, \mathfrak {D} _ {\tau} ^ {C}; \vartheta\right) \right| _ {\vartheta = \vartheta_ {t}} \right] \tag {27} \\ + \mathbb {E} _ {\tau \sim \mathbb {P} ^ {+} (\tau)} \left[ \left. \nabla_ {\vartheta} \ell \left(\mathfrak {D} _ {\tau} ^ {T}, \mathfrak {D} _ {\tau} ^ {C}; \vartheta\right) \right| _ {\vartheta = \vartheta_ {t}} \right]. \\ \end{array}
$$
With  $|F_{\ell}(\hat{\xi}_{\alpha};\vartheta) - F_{\ell}(\xi_{\alpha};\vartheta)| \leq \beta_{\ell}\delta$  in the Assumption (2), the following formula can be easily verified:
$$
\left. - \mathbb {E} _ {\tau \sim \mathbb {P} ^ {-} (\tau)} \left[ \nabla_ {\vartheta} \ell \left(\mathfrak {D} _ {\tau} ^ {T}, \mathfrak {D} _ {\tau} ^ {C}; \vartheta\right) \mid_ {\vartheta = \vartheta_ {t}} \right] ^ {T} \cdot \mathbb {E} _ {\tau \sim \mathbb {P} ^ {+} (\tau)} \left[ \nabla_ {\vartheta} \ell \left(\mathfrak {D} _ {\tau} ^ {T}, \mathfrak {D} _ {\tau} ^ {C}; \vartheta\right) \mid_ {\vartheta = \vartheta_ {t}} \right] \right. \tag {28a}
$$
$$
\leq \left| \left| \mathbb {E} _ {\tau \sim \mathbb {P} ^ {-} (\tau)} \right| \nabla_ {\vartheta} \ell \left(\mathfrak {D} _ {\tau} ^ {T}, \mathfrak {D} _ {\tau} ^ {C}; \vartheta) \mid_ {\vartheta = \vartheta_ {t}} \right| \right| | | \cdot \left| \left| \mathbb {E} _ {\tau \sim \mathbb {P} ^ {+} (\tau)} \right| \nabla_ {\vartheta} \ell \left(\mathfrak {D} _ {\tau} ^ {T}, \mathfrak {D} _ {\tau} ^ {C}; \vartheta) \mid_ {\vartheta = \vartheta_ {t}} \right| \right| | | \tag {28b}
$$
$$
\begin{array}{l} \leq \mathbb {E} _ {\tau \sim \mathbb {P} ^ {-} (\tau)} \left[ \sup  _ {\tau \in \Omega_ {\tau}} | | \nabla_ {\vartheta} \ell \left(\mathfrak {D} _ {\tau} ^ {T}, \mathfrak {D} _ {\tau} ^ {C}; \vartheta\right) | _ {\vartheta = \vartheta_ {t}} | | \right] \cdot \mathbb {E} _ {\tau \sim \mathbb {P} ^ {+} (\tau)} \left[ \sup  _ {\tau \in \Omega_ {\tau}} | | \nabla_ {\vartheta} \ell \left(\mathfrak {D} _ {\tau} ^ {T}, \mathfrak {D} _ {\tau} ^ {C}; \vartheta\right) | _ {\vartheta = \vartheta_ {t}} | | \right] (28c) \\ \leq \beta_ {\ell} \delta (1 - \alpha) \left(\sup  _ {\tau \in \Omega_ {\tau}} | | \nabla_ {\vartheta} \ell \left(\mathfrak {D} _ {\tau} ^ {T}, \mathfrak {D} _ {\tau} ^ {C}; \vartheta\right) | _ {\vartheta = \vartheta_ {t}} | |\right) ^ {2} = \beta_ {\ell} \delta (1 - \alpha) \mu^ {2}. (28d) \\ \end{array}
$$
Once again, we can derive the following inequality with the deduction from Eq. (28):
$$
\begin{array}{l} \varphi_ {\alpha} \left(\xi_ {\alpha_ {t}}; \vartheta_ {t + 1}\right) = \varphi_ {\alpha} \left(\xi_ {\alpha_ {t}}; \vartheta_ {t}\right) - \lambda \left[ \nabla_ {\vartheta} \varphi_ {\alpha} \left(\hat {\xi} _ {\alpha_ {t}}; \vartheta\right) \right| _ {\vartheta = \vartheta_ {t}} ^ {T} \cdot \nabla_ {\vartheta} \varphi_ {\alpha} \left(\xi_ {\alpha_ {t}}; \vartheta\right) | _ {\vartheta = \vartheta_ {t}} \bigg ] + \mathcal {O} \left(| \vartheta_ {t + 1} - \vartheta_ {t} |\right) (29a) \\ = \varphi_ {\alpha} \left(\xi_ {\alpha_ {t}}; \vartheta_ {t}\right) (29b) \\ - \frac {\lambda}{(1 - \alpha) ^ {2}} \left[ \mathbb {E} _ {\tau \sim \mathbb {P} ^ {+} (\tau)} \left[ \nabla_ {\vartheta} \ell \left(\mathfrak {D} _ {\tau} ^ {T}, \mathfrak {D} _ {\tau} ^ {C}; \vartheta\right) | _ {\vartheta = \vartheta_ {t}} \right] ^ {T} \cdot \mathbb {E} _ {\tau \sim \mathbb {P} ^ {+} (\tau)} \left[ \nabla_ {\vartheta} \ell \left(\mathfrak {D} _ {\tau} ^ {T}, \mathfrak {D} _ {\tau} ^ {C}; \vartheta\right) | _ {\vartheta = \vartheta_ {t}} \right] \right] (29c) \\ - \frac {\lambda}{(1 - \alpha) ^ {2}} \left[ \mathbb {E} _ {\tau \sim \mathbb {P} ^ {-} (\tau)} \left[ \nabla_ {\vartheta} \ell \left(\mathfrak {D} _ {\tau} ^ {T}, \mathfrak {D} _ {\tau} ^ {C}; \vartheta\right) | _ {\vartheta = \vartheta_ {t}} \right] ^ {T} \cdot \mathbb {E} _ {\tau \sim \mathbb {P} ^ {+} (\tau)} \left[ \nabla_ {\vartheta} \ell \left(\mathfrak {D} _ {\tau} ^ {T}, \mathfrak {D} _ {\tau} ^ {C}; \vartheta\right) | _ {\vartheta = \vartheta_ {t}} \right] \right] (29d) \\ + \mathcal {O} (| \vartheta_ {t + 1} - \vartheta_ {t} |) (29e) \\ \leq \varphi_ {\alpha} \left(\xi_ {\alpha_ {t}}; \vartheta_ {t}\right) - \frac {\lambda \left| \left| \nu_ {2} \right| \right| _ {2} ^ {2}}{(1 - \alpha) ^ {2}} + \beta_ {\ell} \delta (1 - \alpha) \mu^ {2} + \mathcal {O} \left(\left| \vartheta_ {t + 1} - \vartheta_ {t} \right|\right), (29f) \\ \end{array}
$$
where  $\nu_{2} = \mathbb{E}_{\tau \sim \mathbb{P}^{+}(\tau)}\Big[\nabla_{\vartheta}\ell (\mathfrak{D}_{\tau}^{T},\mathfrak{D}_{\tau}^{C};\vartheta)|_{\vartheta = \vartheta_{t}}\Big].$
To ensure the existence of improvement guarantee, we need that the following holds:
$$
\varphi_ {\alpha} \left(\xi_ {\alpha_ {t}}; \vartheta_ {t}\right) - \frac {\lambda \left\| \nu_ {2} \right\| _ {2} ^ {2}}{(1 - \alpha) ^ {2}} + \beta_ {\ell} \delta (1 - \alpha) \mu^ {2} + \mathcal {O} \left(\left| \vartheta_ {t + 1} - \vartheta_ {t} \right|\right) \leq \varphi_ {\alpha} \left(\xi_ {\alpha_ {t}}; \vartheta_ {t}\right). \tag {30}
$$
Considering the above two cases and estimated bounds Eq. (26)/(30), we can roughly estimate the upper bound of the required  $\delta$  to guarantee performance improvement using our developed strategy in optimization:
$$
\delta \leq \frac {\lambda \left| \left| \nu_ {\mathrm {m}} \right| \right| _ {2} ^ {2}}{\beta_ {\ell} (1 - \alpha) ^ {3} \mu^ {2}}, \tag {31}
$$
where  $\lambda$  is the formerly mentioned learning rate,  $||\nu_{\mathfrak{m}}||_2^2$  is  $\max \{||\nu_1||_2^2,||\nu_2||_2^2\}$ , and  $\mu$  is supremum of the meta risk function derivatives in the task domain  $\sup_{\tau \in \Omega_{\tau}}||\nabla_{\vartheta}\ell (\mathfrak{D}_{\tau}^{T},\mathfrak{D}_{\tau}^{C};\vartheta)|_{\vartheta = \vartheta_{t}}||$
With the help of Jensen inequality,  $||\nu_i||_2^2$  can be roughly bounded as:
$$
\begin{array}{l} \left. \right.\left|\left| \nu_ {i} \right|\right| _ {2} ^ {2} \leq \mathbb {E} _ {\tau \sim \mathbb {P} ^ {+} (\tau)} \left[\left[ \nabla_ {\vartheta} \ell \left(\mathfrak {D} _ {\tau} ^ {T}, \mathfrak {D} _ {\tau} ^ {C}; \vartheta\right) \mid_ {\vartheta = \vartheta_ {t}} \right] ^ {T} \cdot \left[ \nabla_ {\vartheta} \ell \left(\mathfrak {D} _ {\tau} ^ {T}, \mathfrak {D} _ {\tau} ^ {C}; \vartheta\right) \mid_ {\vartheta = \vartheta_ {t}} \right]\right] \tag {32} \\ \leq (1 - \alpha) \left(\sup  _ {\tau \in \Omega_ {\tau}} | | \nabla_ {\vartheta} \ell (\mathfrak {D} _ {\tau} ^ {T}, \mathfrak {D} _ {\tau} ^ {C}; \vartheta) | _ {\vartheta = \vartheta_ {t}} | |\right) ^ {2} = (1 - \alpha) \mu^ {2}. \\ \end{array}
$$
With Eq.s (31)/(32), we can finally obtain the necessary condition for improvement guarantee:
$$
\delta \leq \frac {\lambda}{\beta_ {\ell} (1 - \alpha) ^ {2}}. \tag {33}
$$
I Proof of Approximation Error in Theorem (2)
This section is to build up connections between the number of Monte Carlo samples in estimating  $\mathrm{VaR}_{\alpha}$  and the gap of solutions between the approximately derived one and the theoretical one.
Theorem 2 (Gaps of Optimized Solutions) Suppose  $F_{\ell}(l;\vartheta)\in \mathcal{C}^{2}$  in  $l$ -domain. With meta trained  $\vartheta_*$ , the crude Monte Carlo estimate of  $\hat{\xi}_{\alpha}$ , the constant  $\kappa_{\alpha}$  in Proposition (2), and the sufficiently large number of the task batch  $\mathcal{B}$ , and  $\mathcal{R}_{\mathcal{B}} = \mathcal{O}(\mathcal{B}^{-3 / 4}\ln \mathcal{B})$ , we have the expected error between the exact optimum and the approximate optimum:
$$
\mathcal {E} _ {\alpha} (\vartheta_ {*}) \leq \varphi_ {\alpha} (\hat {\xi} _ {\alpha}; \vartheta_ {*}) + \kappa_ {\alpha} \left[ \frac {\alpha - \hat {F} _ {\ell} (\hat {\xi} _ {\alpha} ; \mathcal {B} , \vartheta_ {*})}{\frac {d F _ {\ell} (\xi ; \vartheta)}{d \xi} | _ {\xi = \xi_ {\alpha}}} + \mathcal {R} _ {\mathcal {B}} \right].
$$
Proof: As noted in (Bahadur, 1966), with assumptions that the cumulative distribution function  $F_{\ell}(l;\vartheta)$ 's second order derivative is continuous in  $l$ -domain, namely  $F_{\ell}(l;\vartheta)\in \mathcal{C}^{2}$ , and  $\frac{dF_{\ell}(\xi;\vartheta)}{d\xi}\big|_{\xi = \xi_{\alpha}}$ , the quantile estimate with crude Monte Carlo can be asymptotically written in the form with the help of central limit theory (Rosenblatt, 1956):
$$
\hat {\xi} _ {\alpha} - \xi_ {\alpha} = \frac {\alpha - \hat {F} _ {\ell} \left(\hat {\xi} _ {\alpha} ; \mathcal {B} , \vartheta_ {*}\right)}{\frac {d F _ {\ell} (\xi ; \vartheta_ {*})}{d \xi} | _ {\xi = \xi_ {\alpha}}} + \mathcal {R} _ {\mathcal {B}}, \quad \text {w i t h} \mathcal {R} _ {\mathcal {B}} = \mathcal {O} \left(\mathcal {B} ^ {- 3 / 4} \ln \mathcal {B}\right) \text {w h e n} \mathcal {B} \rightarrow \infty , \tag {34}
$$
where the empirical cumulative distribution function is computed as follows.
With the sampled meta risk function values  $\mathcal{L}_{\mathcal{B}} = \{\ell(\mathfrak{D}_{\tau_i}^T, \mathfrak{D}_{\tau_i}^C; \vartheta)\}_{i=1}^{\mathcal{B}}$ , we rank them by values as  $\hat{\mathcal{L}}_{\mathcal{B}} = \{\ell(\mathfrak{D}_{\hat{\tau}_i}^T, \mathfrak{D}_{\hat{\tau}_i}^C; \vartheta)\}_{i=1}^{\mathcal{B}}$ , which means  $\ell(\mathfrak{D}_{\hat{\tau}_i-1}^T, \mathfrak{D}_{\hat{\tau}_i-1}^C; \vartheta) \leq \ell(\mathfrak{D}_{\hat{\tau}_i}^T, \mathfrak{D}_{\hat{\tau}_i}^C; \vartheta)$ . Then the empirical cumulative distribution function with these order statistics (Barabás, 1987) can be written as:
$$
\hat {F} _ {\ell} (\xi ; \mathcal {B}, \vartheta) = \left\{ \begin{array}{l l} 0, & \text {i f} \xi \leq \ell \left(\mathfrak {D} _ {\hat {\tau} _ {1}} ^ {T}, \mathfrak {D} _ {\hat {\tau} _ {1}} ^ {C}; \vartheta\right) \\ \frac {k}{\mathcal {B}}, & \text {i f} \ell \left(\mathfrak {D} _ {\hat {\tau} _ {k}} ^ {T}, \mathfrak {D} _ {\hat {\tau} _ {k}} ^ {C}; \vartheta\right) <   \xi \leq \ell \left(\mathfrak {D} _ {\hat {\tau} _ {k + 1}} ^ {T}, \mathfrak {D} _ {\hat {\tau} _ {k + 1}} ^ {C}; \vartheta\right) (k = 1, 2, \dots , \mathcal {B}) \\ 1, & \text {i f} \ell \left(\mathfrak {D} _ {\hat {\tau} _ {\mathcal {B}}} ^ {T}, \mathfrak {D} _ {\hat {\tau} _ {\mathcal {B}}} ^ {C}; \vartheta\right) <   \xi . \end{array} \right. \tag {35}
$$
Based on Proposition (2) and  $\kappa_{\alpha} = \max \left\{\frac{2 - \alpha}{1 - \alpha},\frac{\alpha}{1 - \alpha}\right\}$ , we know the inequality holds:
$$
\varphi_ {\alpha} \left(\hat {\xi} _ {\alpha} (\vartheta); \vartheta\right) - \mathcal {E} _ {\alpha} (\vartheta) <   \kappa_ {\alpha} \delta . \tag {36}
$$
With Eq. (34)/(36), the following inequality naturally holds when  $\mathcal{B}$  is large enough:
$$
\varphi_ {\alpha} (\hat {\xi} _ {\alpha}; \vartheta_ {*}) \leq \mathcal {E} _ {\alpha} (\vartheta_ {*}) + \kappa_ {\alpha} \left[ \frac {\alpha - \hat {F} _ {\ell} (\xi_ {\alpha} ; \mathcal {B} , \vartheta_ {*})}{\frac {d F _ {\ell} (\xi ; \vartheta)}{d \xi} | _ {\xi = \xi_ {\alpha}}} + \mathcal {R} _ {\mathcal {B}} \right].
$$
J Experimental Set-up & Implementation Details
This section is to provide experimental details in this paper. For the implementation of few-shot sinusoid regression and few-shot image classification, we respectively refer the reader to TR-MAML's codes (https://github.com/lgcollins/tr-maml) in (Collins et al., 2020) and vanilla MAML's codes (https://github.com/AnreasAntoniou/HowToTrainYourMAMLPytorch) in (Antoniou et al., 2019). And ours is built on top of the above codes except for simple modification of loss functions. The learning rates for the inner loop and the outer loop of all methods are the same as the above ones.
To facilitate the use of our heuristic optimization strategy, we leave the pytorch version of loss functions within the expected tail risk minimization. The example is provided in the case of mean square errors after MAML's inner loop, which is simple to implement yet effective in robustifying fast adaptation, as follows:
Listing 1: Loss Functions in Two-Stage Heuristic Algorithm with Crude Monte Carlo
import torch   
from torch(nn import MSELoss   
def cvar_mse(y_pred,y_true,conf_level  $= 0.5$  .   
batch_MSE  $\equiv$  MSELoss Reduction  $\equiv$  'none') batch_loss  $\equiv$  batch_MSE(y_pred,y_true) # average risk values over non-task dimensions batch_avg_loss  $\equiv$  torch.mean(batch_loss,dim=-1) # crude Monte Carlo to estimate VaR and sub-tasks topk_mse,topk_idxs  $\equiv$  torch.topk(batch_avg_loss,int((1-conf_level)* y_true.size() [O])）   
return torch.mean(topk_mse)
J.1 Meta Learning Datasets & Tasks
Sinusoid Regression. MAML (Finn et al., 2017), TR-MAML (Collins et al., 2020), and DR-MAML are considered in this experiment. We retain the setup in task generation and partition in (Collins et al., 2020). More specifically, there exists a distribution drift between the meta-training and the meta-testing function families  $\{f_m(x) = a_m\sin (x - b_m)\}_{m = 1}^M$