SlowGuess/NeurIPS_2023_OCR huggingface dataset:
B.1 Solving Non-smooth FCCO with Coordinate Moving Average
We first assume the followings assumptions hold.
Assumption B.1. For all  $i\in S$  , we assume that
-  $f_{i}(\cdot)$  is  $\rho_f$ -weakly-convex,  $C_f$ -Lipschitz continuous and non-decreasing;
-  $g_{i}(\cdot)$  is  $\rho_{g}$ -weakly-convex and  $C_g$ -Lipschitz continuous;
- Stochastic gradient estimators  $g_{i}(\mathbf{w};\xi)$  and  $\partial g_{i}(\mathbf{w};\xi)$  have bounded variance  $\sigma^2$
With coordinate moving average update, we present the following lemma of error bound.
Lemma B.2. Consider the coordinate moving average update for  $\{u_{i,t}:i\in S_1\}$  in Algorithm 3, assume  $g_{i}(\mathbf{w})$  is  $C_g$ -Lipschitz continuous for all  $i\in S_1$  and  $\tau \leq 1$ , then we have
$$
\mathbb {E} [ \| u _ {i, t + 1} - g _ {i} (\mathbf {w} _ {t + 1}) \| ] \leq (1 - \frac {B _ {1} \tau}{4 n _ {1}}) ^ {t + 1} \| u _ {i, 0} - g _ {i} (\mathbf {w} _ {0}) \| + \frac {2 \sqrt {2} \tau^ {1 / 2} \sigma}{B _ {2} ^ {1 / 2}} + \frac {4 \sqrt {2} n _ {1} C _ {g} M \eta}{B _ {1} \tau},
$$
$$
\mathbb {E} [ \| u _ {i, t + 1} - g _ {i} (\mathbf {w} _ {t + 1}) \| ^ {2} ] \leq (1 - \frac {B _ {1} \tau}{4 n _ {1}}) ^ {2 (t + 1)} \| u _ {i, 0} - g _ {i} (\mathbf {w} _ {0}) \| ^ {2} + \frac {8 \tau \sigma^ {2}}{B _ {2}} + \frac {3 2 n _ {1} ^ {2} C _ {g} ^ {2} M ^ {2} \eta^ {2}}{B _ {1} ^ {2} \tau^ {2}}.
$$
Then we have a convergence analysis similar to Theorem 4.6.
Theorem B.3. Consider non-smooth weakly-convex FCCO problem, under Assumption B.1, setting  $\tau = \mathcal{O}(B_2\epsilon^4) \leq 1$ ,  $\eta = \mathcal{O}(\frac{B_1B_2}{n_1}\epsilon^6)$ , Algorithm 3 converges to an  $\epsilon$ -stationary point of the Moreau envelope  $F_{1 / \bar{\rho}}$  in  $T = \mathcal{O}(\frac{n_1}{B_1B_2}\epsilon^{-8})$  iterations.
Proof of Theorem B.3. Since the only difference between SONX and Algorithm 3 is the update for  $\{u_{i,t}:i\in S_1\}$ , the proof of Theorem 4.6 still holds with the error bound replaced by Lemma B.2, i.e.,
$$
\begin{array}{l} \mathbb {E} \left[ \frac {1}{n} \sum_ {i \in \mathcal {S}} \| u _ {i, t + 1} - g _ {i} (\mathbf {w} _ {t + 1}) \| \right] \leq (1 - \mu) ^ {t + 1} \frac {1}{n} \sum_ {i \in \mathcal {S}} \| u _ {i, 0} - g _ {i} (\mathbf {w} _ {0}) \| + R _ {1}, \\ \mathbb {E} \left[ \frac {1}{n} \sum_ {i \in \mathcal {S}} \| u _ {i, t + 1} - g _ {i} (\mathbf {w} _ {t + 1}) \| ^ {2} \right] \leq (1 - \mu) ^ {t + 1} \frac {1}{n} \sum_ {i \in \mathcal {S}} \| u _ {i, 0} - g _ {i} (\mathbf {w} _ {0}) \| ^ {2} + R _ {2}, \\ \end{array}
$$
$$
\mu = \frac {B _ {1} \tau}{4 n _ {1}}, R _ {1} = \frac {2 \sqrt {2} \tau^ {1 / 2} \sigma}{B _ {2} ^ {1 / 2}} + \frac {4 \sqrt {2} n _ {1} C _ {g} M \eta}{B _ {1} \tau}, R _ {2} = \frac {8 \tau \sigma^ {2}}{B _ {2}} + \frac {3 2 n _ {1} ^ {2} C _ {g} ^ {2} M ^ {2} \eta^ {2}}{B _ {1} ^ {2} \tau^ {2}}.
$$
Then proof proceeds to
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} [ \| \nabla F _ {1 / \bar {\rho}} (\mathbf {w} _ {t}) \| ^ {2} ] \leq \mathcal {O} \left(\frac {1}{T} \left(\frac {1}{\eta} + \frac {1}{\mu}\right) + \eta + R _ {1} + R _ {2}\right) \\ = \mathcal {O} \left(\frac {1}{T} (\frac {1}{\eta} + \frac {n _ {1}}{B _ {1} \tau}) + \eta + \frac {\tau^ {1 / 2} \sigma}{B _ {2} ^ {1 / 2}} + \frac {n _ {1} \eta}{B _ {1} \tau} + \frac {\tau \sigma^ {2}}{B _ {2}} + \frac {n _ {1} ^ {2} \eta^ {2}}{B _ {1} ^ {2} \tau^ {2}}\right). \\ \end{array}
$$
$$
\tau = \mathcal {O} (B _ {2} \epsilon^ {4}), \quad \eta = \mathcal {O} (\frac {B _ {1} B _ {2}}{n _ {1}} \epsilon^ {6}),
$$
then to reach a nearly  $\epsilon$ -stationary point, Algorithm 3 needs
$$
T = \mathcal {O} (\frac {n _ {1}}{B _ {1} B _ {2}} \epsilon^ {- 8})
$$
B.2 Solving Non-smooth TCCO with Coordinate Moving Average
We first assume the following assumptions hold.
Assumption B.4. For all  $(i,j)\in S_1\times S_2$ , we assume that
-  $f_{i}(\cdot)$  is  $\rho_{f}$ -weakly-convex,  $C_f$ -Lipschitz continuous and non-decreasing;
-  $g_{i}(\cdot)$  is  $\rho_{g}$ -weakly-convex and  $C_g$ -Lipschitz continuous.  $h_{i,j}(\cdot)$  is differentiable and  $C_h$ -Lipschitz continuous.
- Either  $g_{i}$  is monotone and  $h_{i,j}(\cdot)$  is  $L_{h}$ -smooth, or  $g_{i}$  is non-decreasing and  $h_{i,j}(\cdot)$  is  $L_{h}$ -weakly-convex.
- Stochastic estimators  $h_{i,j}(\mathbf{w},\xi)$ ,  $\partial h_{i,j}(\mathbf{w},\xi)$  and  $g_{i}(v_{i,j})$  have bounded variance  $\sigma^2$ , and  $\| h_{i,j}(\mathbf{w})\| \leq \hat{C}_h$ .
With coordinate moving average update, we present the following lemmas of error bounds.
Algorithm 4 Stochastic Optimization algorithm for Non-smooth TCCO with coordinate moving average
1: Initialization:  $\mathbf{w}_0$ ,  $\{u_{i,0}: i \in S_1\}$ ,  $v_{i,j,0} = h_{i,j}(\mathbf{w}_0; \mathcal{B}_{3,i,j}^0)$  for all  $(i,j) \in S_1 \times S_2$ .
2: for  $t = 0, \dots, T - 1$  do
3: Sample batches  $\mathcal{B}_1^t\subset \mathcal{S}_1,\mathcal{B}_2^t\subset \mathcal{S}_2$  , and  $\mathcal{B}_{3,i,j}^{t}\subset \mathcal{D}_{i,j}$  for  $i\in \mathcal{B}_1^t$  and  $j\in \mathcal{B}_2^t$
4:  $v_{i,j,t+1} = \begin{cases} (1 - \tau_1)v_{i,j,t} + \tau_1h_{i,j}(\mathbf{w}_t;\mathcal{B}_{3,i,j}^t), & (i,j) \in \mathcal{B}_1^t \times \mathcal{B}_2^t \\ v_{i,j,t}, & (i,j) \notin \mathcal{B}_1^t \times \mathcal{B}_2^t \end{cases}$
5:  $u_{i,t + 1} = \left\{ \begin{array}{ll}(1 - \tau_2)u_{i,t} + \frac{1}{B_2}\sum_{j\in \mathcal{B}_2^t}\tau_2g_i(v_{i,j,t}), & i\in \mathcal{B}_1^t\\ u_{i,t},\quad i\notin \mathcal{B}_1^t \end{array} \right.$
6:  $G_{t} = \frac{1}{B_{1}}\sum_{i\in \mathcal{B}_{1}^{t}}\left[\left(\frac{1}{B_{2}}\sum_{i\in \mathcal{B}_{2}^{t}}\nabla h_{i,j}(\mathbf{w}_{t};\mathcal{B}_{3,i,j}^{t})\partial g_{i}(v_{i,j,t})\right)\partial f_{i}(u_{i,t})\right]$
7: Update  $\mathbf{w}_{t + 1} = \mathbf{w}_t - \eta G_t$
8: end for
9: return  $\mathbf{w}_{\bar{t}}$  with uniformly sampled  $\bar{t} \in \{0, T - 1\}$ .
Lemma B.5. Consider the coordinate moving average update for  $\{v_{i,j,t}:(i,j)\in S_1\times S_2\}$  in Algorithm 4, assume  $h_{i,j}(\mathbf{w})$  is  $C_h$ -Lipschitz continuous for all  $(i,j)\in S_1\times S_2$  and  $\tau_{1}\leq 1$ , then we have
$$
\begin{array}{l} \mathbb {E} \left[ \frac {1}{n _ {1} n _ {2}} \sum_ {i \in \mathcal {S} _ {1}} \sum_ {j \in \mathcal {S} _ {2}} \| v _ {i, j, t + 1} - h _ {i, j} (\mathbf {w} _ {t + 1}) \| \right] \\ \leq (1 - \frac {B _ {1} B _ {2} \tau_ {1}}{4 n _ {1} n _ {2}}) ^ {t + 1} \frac {1}{n _ {1} n _ {2}} \sum_ {i \in \mathcal {S} _ {1}} \sum_ {j \in \mathcal {S} _ {2}} \| v _ {i, j, 0} - h _ {i, j} (\mathbf {w} _ {0}) \| + \frac {2 \sqrt {2} \tau_ {1} ^ {1 / 2} \sigma}{B _ {3} ^ {1 / 2}} + \frac {4 \sqrt {2} n _ {1} n _ {2} C _ {h} M \eta}{B _ {1} B _ {2} \tau_ {1}}, \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} \left[ \frac {1}{n _ {1} n _ {2}} \sum_ {i \in \mathcal {S} _ {1}} \sum_ {j \in \mathcal {S} _ {2}} \| v _ {i, j, t + 1} - h _ {i, j} (\mathbf {w} _ {t + 1}) \| ^ {2} \right] \\ \leq (1 - \frac {B _ {1} B _ {2} \tau_ {1}}{4 n _ {1} n _ {2}}) ^ {2 (t + 1)} \frac {1}{n _ {1} n _ {2}} \sum_ {i \in \mathcal {S} _ {1}} \sum_ {j \in \mathcal {S} _ {2}} \| v _ {i, j, 0} - h _ {i, j} (\mathbf {w} _ {0}) \| ^ {2} + \frac {8 \tau_ {1} \sigma^ {2}}{B _ {3}} + \frac {3 2 n _ {1} ^ {2} n _ {2} ^ {2} C _ {h} ^ {2} M ^ {2} \eta^ {2}}{B _ {1} ^ {2} B _ {2} ^ {2} \tau_ {1} ^ {2}}. \\ \end{array}
$$
Lemma B.6. Consider the coordinate moving average update for  $\{u_{i,t}:i\in S_1\}$  in Algorithm 4, assume  $g_{i}(\cdot)$  is  $C_g$ -Lipschitz continuous for all  $i\in S_1$  and  $\tau_{2}\leq 1$ , then we have
$$
\begin{array}{l} \mathbb {E} \left[ \frac {1}{n _ {1}} \sum_ {i \in \mathcal {S} _ {1}} \| u _ {i, t + 1} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} (v _ {i, j, t + 1}) \| \right] \\ \leq (1 - \frac {B _ {1} \tau_ {2}}{4 n _ {1}}) ^ {t + 1} \frac {1}{n _ {1}} \sum_ {i \in \mathcal {S} _ {1}} \| u _ {i, 0} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} (v _ {i, j, 0}) \| + \frac {2 \sqrt {2} \tau_ {2} ^ {1 / 2} \sigma}{B _ {2} ^ {1 / 2}} + \frac {4 \sqrt {2} C _ {g} M n _ {1} ^ {1 / 2} B _ {2} ^ {1 / 2} \tau_ {1}}{B _ {1} ^ {1 / 2} n _ {2} ^ {1 / 2} \tau_ {2}}, \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} \left[ \frac {1}{n _ {1}} \sum_ {i \in \mathcal {S} _ {1}} \| u _ {i, t + 1} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} (v _ {i, j, t + 1}) \| ^ {2} \right] \\ \leq (1 - \frac {B _ {1} \tau_ {2}}{4 n _ {1}}) ^ {2 (t + 1)} \frac {1}{n _ {1}} \sum_ {i \in \mathcal {S} _ {1}} \| u _ {i, 0} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} (v _ {i, j, 0}) \| ^ {2} + \frac {8 \tau_ {2} \sigma^ {2}}{B _ {2}} + \frac {3 2 C _ {g} ^ {2} M ^ {2} n _ {1} B _ {2} \tau_ {1} ^ {2}}{B _ {1} n _ {2} \tau_ {2} ^ {2}}. \\ \end{array}
$$
Then we have a convergence analysis similar to Theorem A.2.
Theorem B.7. Consider non-smooth weakly-convex TCCO problem, under Assumption B.4, setting  $\tau_{1} = \mathcal{O}\left(\min \left\{B_{3}\epsilon^{4},\frac{B_{1}^{1 / 2}n_{2}^{1 / 2}}{n_{1}^{1 / 2}B_{2}^{1 / 2}} B_{2}\epsilon^{6}\right\}\right)$ $\leq 1,\tau_2 = \mathcal{O}(B_2\epsilon^4)\leq 1,\eta =$ $\mathcal{O}\left(\min \left\{B_3\epsilon^4,\frac{B_1^{1 / 2}n_2^{1 / 2}}{n_1^{1 / 2}B_2^{1 / 2}} B_2\epsilon^6\right\} \frac{B_1B_2}{n_1n_2}\epsilon^2\right),$  Algorithm 4 converges to an  $\epsilon$  -stationary point of the Moreau envelope  $F_{1 / \bar{\rho}}$  in  $T = \mathcal{O}\left(\max \left\{\frac{1}{B_3},\frac{n_1^{1 / 2}}{B_1^{1 / 2}B_2^{1 / 2}n_2^{1 / 2}}\epsilon^{-2}\right\} \frac{n_1n_2}{B_1B_2}\epsilon^{-8}\right)$  iterations.
Proof of Theorem B.7. Since the only difference between SONT and Algorithm 4 is the update for  $\{u_{i,t}:i\in \mathcal{S}_1\}$  and  $\{v_{i,j,t}:(i,j)\in \mathcal{S}_1\times \mathcal{S}_2\}$ , the proof of Theorem A.2 still holds with the error bound replaced by Lemma B.5 and Lemma B.6, i.e.,
$$
\frac {1}{n _ {1} n _ {2}} \sum_ {i \in S _ {1}} \sum_ {j \in S _ {2}} \mathbb {E} [ \| h _ {i, j} (\mathbf {w} _ {t}) - v _ {i, j, t} \| ] \leq (1 - \mu_ {1}) ^ {t} \frac {1}{n _ {1} n _ {2}} \sum_ {i \in S _ {1}} \sum_ {j \in S _ {2}} \| h _ {i, j} (\mathbf {w} _ {0}) - v _ {i, j, 0} \| + R _ {1},
$$
$$
\frac {1}{n _ {1} n _ {2}} \sum_ {i \in S _ {1}} \sum_ {j \in S _ {2}} \mathbb {E} [ \| h _ {i, j} (\mathbf {w} _ {t}) - v _ {i, j, t} \| ^ {2} ] \leq (1 - \mu_ {1}) ^ {t} \frac {1}{n _ {1} n _ {2}} \sum_ {i \in S _ {1}} \sum_ {j \in S _ {2}} \| h _ {i, j} (\mathbf {w} _ {0}) - v _ {i, j, 0} \| ^ {2} + R _ {2},
$$
$$
\frac {1}{n _ {1}} \sum_ {i \in S _ {1}} \mathbb {E} \left[ \left\| \frac {1}{n _ {2}} \sum_ {j \in S _ {2}} g _ {i} (v _ {i, j, t}) - u _ {i, t} \right\| \right] \leq (1 - \mu_ {2}) ^ {t} \frac {1}{n _ {+}} \sum_ {i \in S _ {+}} \left\| \frac {1}{n _ {2}} \sum_ {j \in S _ {2}} g _ {i} (v _ {i, j, 0}) - u _ {i, 0} \right\| + R _ {3},
$$
$$
\mu_ {1} = \frac {B _ {1} B _ {2} \tau_ {1}}{4 n _ {1} n _ {2}}, \quad \mu_ {2} = \frac {B _ {1} \tau_ {2}}{4 n _ {1}}, \quad R _ {1} = \frac {2 \sqrt {2} \tau_ {1} ^ {1 / 2} \sigma}{B _ {3} ^ {1 / 2}} + \frac {4 \sqrt {2} n _ {1} n _ {2} C _ {h} M \eta}{B _ {1} B _ {2} \tau_ {1}}
$$
$$
R _ {2} = \frac {8 \tau_ {1} \sigma^ {2}}{B _ {3}} + \frac {3 2 n _ {1} ^ {2} n _ {2} ^ {2} C _ {h} ^ {2} M ^ {2} \eta^ {2}}{B _ {1} ^ {2} B _ {2} ^ {2} \tau_ {1} ^ {2}}, R _ {3} = \frac {2 \sqrt {2} \tau_ {2} ^ {1 / 2} \sigma}{B _ {2} ^ {1 / 2}} + \frac {4 \sqrt {2} C _ {g} M n _ {1} ^ {1 / 2} B _ {2} ^ {1 / 2} \tau_ {1}}{B _ {1} ^ {1 / 2} n _ {2} ^ {1 / 2} \tau_ {2}},
$$
$$
R _ {4} = \frac {8 \tau_ {2} \sigma^ {2}}{B _ {2}} + \frac {3 2 C _ {g} ^ {2} M ^ {2} n _ {1} B _ {2} \tau_ {1} ^ {2}}{B _ {1} n _ {2} \tau_ {2} ^ {2}}
$$
Then the proof proceeds to
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} [ \| \nabla F _ {1 / \bar {\rho}} (\mathbf {w} _ {t}) \| ^ {2} ] \\ \leq \mathcal {O} \left(\frac {1}{T} \left(\frac {1}{\eta} + \frac {1}{\mu_ {\text {m i n}}}\right) + \eta + R _ {1} + R _ {2} + R _ {3} + R _ {4}\right) \\ \leq \mathcal {O} \bigg (\frac {1}{T} (\frac {1}{\eta} + \frac {1}{\mu_ {m i n}}) + \eta + \frac {\tau_ {1} ^ {1 / 2} \sigma}{B _ {3} ^ {1 / 2}} + \frac {\tau_ {1} \sigma^ {2}}{B _ {3}} + \frac {\tau_ {2} ^ {1 / 2} \sigma}{B _ {2} ^ {1 / 2}} + \frac {\tau_ {2} \sigma^ {2}}{B _ {2}} + \frac {n _ {1} n _ {2} \eta}{B _ {1} B _ {2} \tau_ {1}} + \frac {n _ {1} ^ {2} n _ {2} ^ {2} \eta^ {2}}{B _ {1} ^ {2} B _ {2} ^ {2} \tau_ {1} ^ {2}} \\ \left. + \frac {n _ {1} ^ {1 / 2} B _ {2} ^ {1 / 2} \tau_ {1}}{B _ {1} ^ {1 / 2} n _ {2} ^ {1 / 2} \tau_ {2}} + \frac {n _ {1} B _ {2} \tau_ {1} ^ {2}}{B _ {1} n _ {2} \tau_ {2} ^ {2}}\right) \\ \leq \mathcal {O} \left(\frac {1}{T} \left(\frac {1}{\eta} + \frac {1}{\mu_ {\text {m i n}}}\right) + \frac {\tau_ {1} ^ {1 / 2} \sigma}{B _ {3} ^ {1 / 2}} + \frac {\tau_ {2} ^ {1 / 2} \sigma}{B _ {2} ^ {1 / 2}} + \frac {n _ {1} n _ {2} \eta}{B _ {1} B _ {2} \tau_ {1}} + \frac {n _ {1} ^ {1 / 2} B _ {2} ^ {1 / 2} \tau_ {1}}{B _ {1} ^ {1 / 2} n _ {2} ^ {1 / 2} \tau_ {2}}\right). \\ \end{array}
$$
$$
\tau_ {1} = \mathcal {O} \left(\min  \left\{B _ {3} \epsilon^ {4}, \frac {B _ {1} ^ {1 / 2} n _ {2} ^ {1 / 2}}{n _ {1} ^ {1 / 2} B _ {2} ^ {1 / 2}} B _ {2} \epsilon^ {6} \right\}\right), \quad \tau_ {2} = \mathcal {O} (B _ {2} \epsilon^ {4}),
$$
$$
\eta = \mathcal {O} \left(\min  \left\{B _ {3} \epsilon^ {4}, \frac {B _ {1} ^ {1 / 2} n _ {2} ^ {1 / 2}}{n _ {1} ^ {1 / 2} B _ {2} ^ {1 / 2}} B _ {2} \epsilon^ {6} \right\} \frac {B _ {1} B _ {2}}{n _ {1} n _ {2}} \epsilon^ {2}\right),
$$
then to reach a nearly  $\epsilon$ -stationary point, Algorithm 4 need
$$
T = \mathcal {O} \left(\max  \left\{\frac {1}{B _ {3}}, \frac {n _ {1} ^ {1 / 2}}{B _ {1} ^ {1 / 2} B _ {2} ^ {1 / 2} n _ {2} ^ {1 / 2}} \epsilon^ {- 2} \right\} \frac {n _ {1} n _ {2}}{B _ {1} B _ {2}} \epsilon^ {- 1 0}\right)
$$
C Details for TPAUC Maximization
C.1 Assumption Verification
We first present two lemmas about the weak convexity of the objective in the regular learning setting and in the multi-instance learning setting with mean pooling.
Lemma C.1. Consider the formulation in problem (9) in the regular learning setting and assume that function  $\ell(\cdot)$  is non-decreasing,  $C_{\ell}$  Lipschitz continuous and  $\rho_{\ell}$ -weakly-convex, and function  $h_{\mathbf{w}}(X_i)$  is  $C_h$  Lipschitz continuous and  $\rho_h$ -weakly-convex. Then the following statements are true:
-  $f_{i}(g, s')$  is convex and  $C_{f}$ -Lipschitz continuous w.r.t.  $(g, s')$ , and non-decreasing w.r.t.  $g$ .
-  $\psi_i(\mathbf{w}, s_i)$  is  $\rho_{\psi}$ -weakly-convex w.r.t.  $(\mathbf{w}, s_i)$ , and the stochastic estimator of the finite sum function value  $\psi_i(\mathbf{w}, s_i)$  is  $C_{\psi}$ -Lipschitz continuous w.r.t.  $(\mathbf{w}, s_i)$ .
-  $\frac{1}{n_{+}}\sum_{i\in S_{+}}f_{i}\bigl{(}\psi_{i}(\mathbf{w},s_{i}),s^{\prime}\bigr)$  is  $\rho_F$  -weakly-convex w.r.t.  $(\mathbf{w},\mathbf{s},s^{\prime})$
Lemma C.2. Consider the formulation in problem (9) in the multi-instance learning setting with mean pooling, and assume that function  $h_i(\mathbf{w}) = \frac{1}{|X_i|} \sum_{\mathbf{x} \in X_i} e(\mathbf{w}_e; \mathbf{x})^\top \mathbf{w}_c$  is  $\tilde{L}_h$ -smooth and is bounded by  $\tilde{C}_h$ , and  $h_i(\mathbf{w}; \xi) = e(\mathbf{w}_e; \xi)^\top \mathbf{w}_c$  is  $C_h$ -Lipschitz continuous and has bounded variance  $\sigma^2$ ,  $\ell$  is non-decreasing and  $L_\ell$ -weakly-convex, then the followings are true:
-  $f_{i}(g, s')$  is convex and  $C_{f}$ -Lipschitz-continuous w.r.t.  $(g, s')$ , and non-decreasing w.r.t.  $g$ ;
-  $g_{i}(v, s_{i}) = s_{i} + \frac{(\ell(v) - s_{i})_{+}}{\beta}$  is  $\rho_{g}$ -weakly convex and non-decreasing w.r.t.  $v$ , convex w.r.t.  $s_{i}$ , and  $C_{g}$ -Lipschitz continuous w.r.t.  $(v, s_{i})$ ;
-  $h_{i,j}(\mathbf{w}) = h_j(\mathbf{w}) - h_i(\mathbf{w})$  is  $L_h$ -weakly-convex, and  $h_{i,j}(\mathbf{w};\xi ,\zeta)$  is  $C_h$ -Lipschitz continuous;
-  $\frac{1}{n_{+}}\sum_{X_{i}\in \mathcal{S}_{+}}f_{i}(g_{i}(h_{i,j}(\mathbf{w}),s_{i}),s^{\prime})$  is  $\rho_F$ -weakly-convex w.r.t.  $(\mathbf{w},\mathbf{s},s^{\prime})$ .
C.1.1 Proof of Lemma C.1
Proof of Lemma C.1. The convexity of  $f_{i}(g,s^{\prime})$  with respect to  $(g,s^{\prime})$  follows from the convexity definition. With subgradients  $\partial_{s^{\prime}}f_{i}(g,s^{\prime})\in [1 - \frac{1}{\alpha},1]$ ,  $\partial_{g}f_{i}(g,s^{\prime})\in [0,\frac{1}{\alpha}]$ , we can see that  $f_{i}(g,s^{\prime})$  is  $\frac{1}{\alpha}$ -Lipschitz continuous w.r.t.  $(g,s^{\prime})$ , and non-decreasing w.r.t.  $u$ .
We first show that  $\ell(h_{\mathbf{w}}(X_j) - h_{\mathbf{w}}(X_i))$  is weakly-convex w.r.t.  $\mathbf{w}$ .
$$
\begin{array}{l} \ell \left(h _ {\tilde {\mathbf {w}}} \left(X _ {j}\right) - h _ {\tilde {\mathbf {w}}} \left(X _ {i}\right)\right) \\ \geq \ell \left(h _ {\mathbf {w}} \left(X _ {j}\right) - h _ {\mathbf {w}} \left(X _ {i}\right)\right) + \langle \partial \ell \left(h _ {\mathbf {w}} \left(X _ {j}\right) - h _ {\mathbf {w}} \left(X _ {i}\right)\right), \left(h _ {\tilde {\mathbf {w}}} \left(X _ {j}\right) - h _ {\tilde {\mathbf {w}}} \left(X _ {i}\right)\right) - \left(h _ {\mathbf {w}} \left(X _ {j}\right) - h _ {\mathbf {w}} \left(X _ {i}\right)\right) \rangle \\ + \frac {\rho_ {\ell}}{2} \left\| \left(h _ {\tilde {\mathbf {w}}} (X _ {j}) - h _ {\tilde {\mathbf {w}}} (X _ {i})\right) - \left(h _ {\mathbf {w}} (X _ {j}) - h _ {\mathbf {w}} (X _ {i})\right) \right\| ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(a)} {\geq} \ell \left(h _ {\mathbf {w}} \left(X _ {j}\right) - h _ {\mathbf {w}} \left(X _ {i}\right)\right) + \langle \partial \ell \left(h _ {\mathbf {w}} \left(X _ {j}\right) - h _ {\mathbf {w}} \left(X _ {i}\right)\right), \langle \nabla h _ {\mathbf {w}} \left(X _ {j}\right) - \nabla h _ {\mathbf {w}} \left(X _ {i}\right), \tilde {\mathbf {w}} - \mathbf {w} \rangle \rangle \\ + 2 \rho_ {\ell} C _ {h} ^ {2} \| \tilde {\mathbf {w}} - \mathbf {w} \| ^ {2} \\ \end{array}
$$
where (a) uses the weak-convexity of  $h_{\mathbf{w}}(X_i)$  and  $h_{\mathbf{w}}(X_j)$ ,
$$
\begin{array}{l} h _ {\tilde {\mathbf {w}}} (X _ {j}) - h _ {\mathbf {w}} (X _ {j}) \geq \langle \nabla h _ {\mathbf {w}} (X _ {j}), \tilde {\mathbf {w}} - \mathbf {w} \rangle - \frac {\rho_ {h}}{2} \| \tilde {\mathbf {w}} - \mathbf {w} \| ^ {2}, \\ - h _ {\tilde {\mathbf {w}}} (X _ {i}) + h _ {\mathbf {w}} (X _ {i}) \geq - \left\langle \nabla h _ {\mathbf {w}} (X _ {i}), \tilde {\mathbf {w}} - \mathbf {w} \right\rangle + \frac {\rho_ {h}}{2} \| \tilde {\mathbf {w}} - \mathbf {w} \| ^ {2}. \\ \end{array}
$$
Thus  $\ell(h_{\mathbf{w}}(X_j) - h_{\mathbf{w}}(X_i))$  is  $4\rho_{\ell}C_h^2$ -weakly-convex w.r.t.  $\mathbf{w}$ .
By convexity of  $(\ell, s_i) \mapsto s_i + \frac{(\ell - s_i)_+}{\beta}$ , we have
$$
\psi_ {i} (\tilde {\mathbf {w}}, \tilde {s} _ {i})
$$
$$
\geq \psi_ {i} (\mathbf {w}, s _ {i}) + \left\langle \partial_ {\ell} \psi_ {i} (\mathbf {w}, s _ {i}), \ell \left(h _ {\tilde {\mathbf {w}}} \left(X _ {j}\right) - h _ {\tilde {\mathbf {w}}} \left(X _ {i}\right)\right) - \ell \left(h _ {\mathbf {w}} \left(X _ {j}\right) - h _ {\mathbf {w}} \left(X _ {i}\right)\right) \right\rangle + \left\langle \partial_ {s _ {i}} \psi_ {i} (\mathbf {w}, s _ {i}), \tilde {s} _ {i} - s _ {i} \right\rangle
$$
$$
\begin{array}{l} \stackrel {(a)} {\geq} \psi_ {i} (\mathbf {w}, s _ {i}) + \partial_ {\ell} \psi_ {i} (\mathbf {w}, s _ {i}) \bigg [ \partial \ell (h _ {\mathbf {w}} (X _ {j}) - h _ {\mathbf {w}} (X _ {i})) \langle \nabla h _ {\mathbf {w}} (X _ {j}) - \nabla h _ {\mathbf {w}} (X _ {i}), \tilde {\mathbf {w}} - \mathbf {w} \rangle \rangle - 2 \rho_ {\ell} C _ {h} ^ {2} \| \tilde {\mathbf {w}} - \mathbf {w} \| ^ {2} \bigg ] \\ + \left\langle \partial_ {s _ {i}} \psi_ {i} (\mathbf {w}, s _ {i}), \tilde {s} _ {i} - s _ {i} \right\rangle \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(b)} {\geq} \psi_ {i} (\mathbf {w}, s _ {i}) + \partial_ {\ell} \psi_ {i} (\mathbf {w}, s _ {i}) \partial \ell \left(h _ {\mathbf {w}} \left(X _ {j}\right) - h _ {\mathbf {w}} \left(X _ {i}\right)\right) \langle \nabla h _ {\mathbf {w}} \left(X _ {j}\right) - \nabla h _ {\mathbf {w}} \left(X _ {i}\right), \tilde {\mathbf {w}} - \mathbf {w} \rangle \rangle \\ + \left\langle \partial_ {s _ {i}} \psi_ {i} (\mathbf {w}, s _ {i}), \tilde {s} _ {i} - s _ {i} \right\rangle - \frac {2 \rho_ {\ell} C _ {h} ^ {2}}{\beta} \| \tilde {\mathbf {w}} - \mathbf {w} \| ^ {2} \\ \end{array}
$$
where (a) follows from the monotonicity of  $\psi_{i}$  w.r.t.  $\ell$  and weak-convexity of  $\ell(h_{\mathbf{w}}(X_j) - h_{\mathbf{w}}(X_i))$ , and (b) is due to the Lipschitz continuity of  $(\ell, s_i) \mapsto s_i + \frac{(\ell - s_i)_+}{\beta}$  w.r.t.  $\ell$ . Thus  $\psi_{i}$  is  $\frac{4\rho_{\ell}C_h^2}{\beta}$ -weakly-convex w.r.t.  $(\mathbf{w}, s_i)$ .
With a similar argument using the convexity and Lipschitz continuity of  $f_{i}(g,s^{\prime})$  w.r.t.  $(g,s^{\prime})$  and the weak-convexity of  $\psi_i(\mathbf{w},s_i)$ , we can show that  $f_{i}(\psi_{i}(\mathbf{w},s_{i}),s^{\prime})$  is  $\frac{4\rho_{\ell}C_h^2}{\beta}$ -weakly-convex w.r.t.  $(\mathbf{w},s_i,s')$ . Thus,  $F(\mathbf{w},s_i,s')$  is  $\rho_F = \frac{4\rho_\ell C_h^2}{\beta}$ -weakly-convex w.r.t.  $(\mathbf{w},\mathbf{s},s')$ .
Now we show the Lipschitz continuity of  $\psi_i(\mathbf{w}, s_i; X_j)$ , i.e. an unbiased stochastic estimator of  $\psi_i(\mathbf{w}, s_i)$ . We have
$$
\begin{array}{l} \left\| \psi_ {i} (\mathbf {w}, s _ {i}; X _ {j}) - \psi_ {i} (\tilde {\mathbf {w}}, \tilde {s} _ {i}; X _ {j}) \right\| ^ {2} \\ = \left\| \left(s _ {i} + \frac {\left(\ell \left(h _ {\mathbf {w}} \left(X _ {j}\right) - h _ {\mathbf {w}} \left(X _ {i}\right)\right) - s _ {i}\right) _ {+}}{\beta}\right) - \left(\tilde {s} _ {i} + \frac {\left(\ell \left(h _ {\tilde {\mathbf {w}}} \left(X _ {j}\right) - h _ {\tilde {\mathbf {w}}} \left(X _ {i}\right)\right) - \tilde {s} _ {i}\right) _ {+}}{\beta}\right) \right\| ^ {2} \\ \leq 2 \| s _ {i} - \tilde {s} _ {i} \| ^ {2} + 2 \left\| \frac {\left(\ell \left(h _ {\mathbf {w}} \left(X _ {j}\right) - h _ {\mathbf {w}} \left(X _ {i}\right)\right) - s _ {i}\right) _ {+}}{\beta} - \frac {\left(\ell \left(h _ {\tilde {\mathbf {w}}} \left(X _ {j}\right) - h _ {\tilde {\mathbf {w}}} \left(X _ {i}\right)\right) - \tilde {s} _ {i}\right) _ {+}}{\beta} \right\| ^ {2} \\ \leq 2 \| s _ {i} - \tilde {s} _ {i} \| ^ {2} + \frac {2}{\beta^ {2}} (8 C _ {\ell} ^ {2} C _ {h} ^ {2} \| \tilde {\mathbf {w}} - \mathbf {w} \| ^ {2} + 2 \| \tilde {s} _ {i} - s _ {i} \| ^ {2}) \\ \leq (2 + \frac {4 + 1 6 C _ {\ell} ^ {2} C _ {h} ^ {2}}{\beta^ {2}}) (\| \tilde {\mathbf {w}} - \mathbf {w} \| ^ {2} + \| \tilde {s} _ {i} - s _ {i} \| ^ {2}). \\ \end{array}
$$
Thus  $\psi_{i}(\mathbf{w},s_{i};X_{j})$  is  $(2 + \frac{4 + 16C_{\ell}^{2}C_{h}^{2}}{\beta^{2}})^{1 / 2}$ -Lipschitz continuous w.r.t.  $(\mathbf{w},s_{i})$ .
C.1.2 Proof of Lemma C.2
Proof of Lemma C.2. First of all, the convexity of  $f_{i}(u,s^{\prime})$  w.r.t.  $(u,s^{\prime})$  and the convexity of  $g_{i}(v_{ij},s_{i})$  w.r.t.  $(\ell ,s_i)$  directly follows from the convexity definition. Moreover, one can see from the formulation that  $\partial_{s'}f_i(g,s')\in [1 - \frac{1}{\alpha},1]$ ,  $\partial_{u}f_{i}(g,s^{\prime})\in [0,\frac{1}{\alpha}]$ ,  $\partial_{\ell}g_{i}(v_{ij},s_{i})\in [1 - \frac{1}{\beta},1]$ ,  $\partial_{s_i}g_i(v_{ij},s_i)\in [0,\frac{1}{\beta}]$ . Thus  $f_{i}$  is  $C_f = \frac{1}{\alpha}$ -Lipschitz continuous w.r.t.  $(u,s^{\prime})$  and non-decreasing w.r.t.  $u$ ,  $g_{i}$  is  $\frac{1}{\beta}$ -Lipschitz continuous w.r.t.  $(\ell ,s_i)$  and non-decreasing w.r.t.  $\ell$ . Since  $\ell (\cdot)$  is nondecreasing,  $g_{i}(v_{ij},s_{i})$  is non-decreasing w.r.t.  $v_{ij}$ . As a result of Proposition 4.2,  $g_{i}(v_{ij},s_{i})$  is  $\rho_g = \frac{1}{\beta} L_\ell$ -weakly-convex w.r.t.  $v_{ij}$ . Due to the composition structure and the Lipschitz continuity of  $g_{i}$  and  $\ell$ , one can see that  $g_{i}(v_{ij},s_{i})$  is  $C_g = \frac{1}{\beta} C_\ell$ -Lipschitz continuous w.r.t.  $(v_{ij},s_i)$ .