SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {C L - D R O}} ^ {K L} = - \mathbb {E} _ {P _ {0}} [ f _ {\theta} ] + \min  _ {\alpha \geq 0, \beta} \left\{\alpha \eta - \beta + \alpha \mathbb {E} _ {Q _ {0}} \left[ \phi^ {*} \left(\frac {f _ {\theta} + \beta}{\alpha}\right) \right] \right\} \\ = - \mathbb {E} _ {P _ {0}} [ f _ {\theta} ] + \min  _ {\alpha > 0, \beta} \left\{\alpha \eta - \beta + \alpha \mathbb {E} _ {Q _ {0}} \left[ e ^ {\frac {f _ {\theta} + \beta}{\alpha}} - 1 \right] \right\} \\ = - \mathbb {E} _ {P _ {0}} [ f _ {\theta} ] + \min  _ {\alpha \geq 0} \left\{\alpha \eta + \alpha \log \mathbb {E} _ {Q _ {0}} \left[ e ^ {\frac {f _ {\theta}}{\alpha}} \right] \right\} \tag {18} \\ = - \mathbb {E} _ {P _ {0}} [ f _ {\theta} ] + \min  _ {\alpha > 0} \left\{\alpha \eta + \alpha \log \mathbb {E} _ {Q _ {0}} \left[ e ^ {\frac {f _ {\theta}}{\alpha}} \right] \right\} \\ = - \mathbb {E} _ {P _ {0}} \left[ \alpha^ {*} \log \frac {e ^ {f _ {\theta} / \alpha^ {*}}}{\mathbb {E} _ {Q _ {0}} \left[ e ^ {f _ {\theta} / \alpha^ {*}} \right]} \right] + \alpha \eta \\ = \alpha^ {*} \mathcal {L} _ {\text {I n f o N C E}} + \text {C o n s t a n t} \\ \end{array}
$$
Here  $\alpha^{*} = \arg \min_{\alpha \geq 0}\left\{\alpha \eta +\alpha \log \mathbb{E}_{Q_0}[e^{\frac{f_\theta}{\alpha}}]\right\} .$
Theorem 3.3. [Generalization Bound] Let  $\widehat{\mathcal{L}}_{InfoNCE}$  be an estimation of InfoNCE with  $N$  negative samples. Given any finite hypothesis space  $\mathbb{F}$  of models, suppose  $f_{\theta} \in [M_1, M_2]$  and the ideal negative sampling distribution  $Q^{ideal}$  satisfies  $D_{KL}(Q^{ideal})||Q_0) \leq \eta$ , we have that with probability at least  $1 - \rho$ :
$$
\mathcal {L} _ {\text {u n b i a s e d}} \leq \tau \hat {\mathcal {L}} _ {\text {I n f o N C E}} + \mathcal {B} (\rho , N, \tau), \tag {5}
$$
Here we simply disregard the constant term present in Equation (4) as it does not impact optimization, and omit the error from the positive instances.
Proof. Before detailing the proof process, we first introduce a pertinent theorem:
Theorem A.4 (Union Bound). For any two sets  $A, B$  and a distribution  $\mathcal{D}$  we have
$$
\mathcal {D} (A \cup B) \leq \mathcal {D} (A) + \mathcal {D} (B) \tag {19}
$$
Theorem A.5 (McDiarmid's inequality [58]). Let  $X_{1}, \dots, X_{n}$  be independent random variables, where  $X_{i}$  has range  $\mathcal{X}$ . Let  $f: \mathcal{X}_1 \times \dots \times \mathcal{X}_n \to \mathbb{R}$  be any function with the  $(c_{1}, \ldots, c_{n})$ -bounded difference property: for every  $i = 1, \ldots, n$  and every  $(x_{1}, \ldots, x_{n}), (x_{1}', \ldots, x_{n}') \in \mathcal{X}_{1} \times \dots \times \mathcal{X}_{n}$  that differ only in the  $i$ -th coordinate  $(x_{j} = x_{j}'$  for all  $j \neq i$ ), we have  $|f(x_{1}, \ldots, x_{n}) - f(x_{1}', \ldots, x_{n}')| \leq c_{i}$ . For any  $\epsilon > 0$ ,
$$
\mathbb {P} \left(f \left(X _ {1}, \dots , X _ {n}\right) - \mathbb {E} \left[ f \left(X _ {1}, \dots , X _ {n}\right) \right] \geq \epsilon\right) \leq \exp \left(\frac {- 2 \epsilon^ {2}}{\sum_ {i = 1} ^ {N} c _ {i} ^ {2}}\right) \tag {20}
$$
Now we delve into the proof. As  $Q^{ideal}$  satisfies  $D_{KL}(Q||Q_0)\leq \eta$ , we can bound  $\mathcal{L}_{unbiased}$  with:
$$
\begin{array}{l} \mathcal {L} _ {\text {u n b i a s e d}} = - \mathbb {E} _ {P _ {0}} [ f _ {\theta} ] + \mathbb {E} _ {Q ^ {i d e a l}} [ f _ {\theta} ] \\ \leq - \mathbb {E} _ {P _ {0}} [ f _ {\theta} ] + \max  _ {D _ {K L} (Q | | Q _ {0}) \leq \eta} \mathbb {E} _ {Q} [ f _ {\theta} ] \tag {21} \\ = \mathcal {L} _ {\mathrm {C L - D R O}} ^ {K L} \\ \end{array}
$$
where  $Q^{\mathrm{ideal}}$ ,  $Q^{*}$  denotes the ideal negative distribution and the worst-case distribution in CL-DRO. From the Theorem 3.2, we have the equivalence between InfoNCE and CL-DRO. Thus here we
choose CL-DRO for analyses. Suppose we have  $N$  negative samples, and for any pair of samples  $(x_{i},y_{i}),(x_{j},y_{j})$ , we have the following bound:
$$
\left| Q ^ {*} \left(x _ {i}, y _ {i}\right) f _ {\theta} \left(x _ {i}, y _ {i}\right) - Q ^ {*} \left(x _ {j}, y _ {j}\right) f _ {\theta} \left(x _ {j}, y _ {j}\right) \right| \leq \sup  _ {(x, y) \sim Q _ {0}} \left| Q ^ {*} (x, y) f _ {\theta} (x, y) \right| \leq \frac {\exp \left(\frac {1}{\tau}\right)}{N - 1 + \exp \left(\frac {1}{\tau}\right)} \tag {22}
$$
where the first inequality holds as  $Q^{*}(x,y)f_{\theta}(x,y) > 0$ . The second inequality holds based on the expression of  $Q^{*} = Q_{0}\frac{\exp[f_{\theta} / \tau]}{E_{Q_{0}}\exp[f_{\theta} / \tau]}$  (refer to Appendix A.6). Suppose  $f_{\theta}\in [M_1,M_2]$ , the upper bound of  $\sup_{(x,y)\sim Q_0}|Q^* (x,y)f_\theta (x,y)|$  arrives if  $f_{\theta}(x,y) = M_2$  for the sample  $(x,y)$  and  $f_{\theta}(x,y) = M_1$  for others. We have  $\sup_{(x,y)\sim Q_0}|Q^* (x,y)f_\theta (x,y)|\leq \frac{M_2\exp((M_2 - M_1) / \tau)}{N - 1 + \exp((M_2 - M_1) / \tau)}$ .
By using McDiarmid's inequality in Theorem A.5, for any  $\epsilon$ , we have:
$$
\mathbb {P} \left[ \left(\mathcal {L} _ {\mathrm {C L - D R O}} ^ {K L} - \tau \widehat {\mathcal {L}} _ {\text {I n f o N C E}}\right) \geq \epsilon \right] \leq \exp \left(\frac {- 2 \epsilon^ {2}}{N} \left(\frac {N - 1 + \exp \left(\left(M _ {2} - M _ {1}\right) / \tau\right)}{M _ {2} \exp \left(\left(M _ {2} - M _ {1}\right) / \tau\right)}\right) ^ {2}\right) \tag {23}
$$
Equation (23) holds for the unique model  $f$ . For any finite hypothesis space of models  $\mathbb{F} = \{f_1, f_2, \dots, f_{|\mathbb{F}|}\}$ , we combine this with Theorem A.4 and have following generalization error bound of the learned model  $\hat{f}$ :
$$
\begin{array}{l} \mathbb {P} \left[ \left(\mathcal {L} _ {\mathrm {C L - D R O}} ^ {K L} - \tau \widehat {\mathcal {L}} _ {\text {I n f o N C E}}\right) \geq \epsilon \right] \leq \sum_ {f \in \mathbb {F}} \exp \left(\frac {- 2 \epsilon^ {2}}{N} \left(\frac {N - 1 + \exp \left((M _ {2} - M _ {1}) / \tau\right)}{M _ {2} \exp \left((M _ {2} - M _ {1}) / \tau\right)}\right) ^ {2}\right) \tag {24} \\ = | \mathbb {F} | \exp \left(\frac {- 2 \epsilon^ {2}}{N} \left(\frac {N - 1 + \exp ((M _ {2} - M _ {1}) / \tau)}{M _ {2} \exp ((M _ {2} - M _ {1}) / \tau)}\right) ^ {2}\right) \\ \end{array}
$$
$$
\rho = | \mathbb {F} | \exp \left(\frac {- 2 \epsilon^ {2}}{N} \left(\frac {N - 1 + \exp \left((M _ {2} - M _ {1}) / \tau\right)}{M _ {2} \exp \left((M _ {2} - M _ {1}) / \tau\right)}\right) ^ {2}\right) \tag {25}
$$
$$
\epsilon = \frac {M _ {2} \exp \left(\left(M _ {2} - M _ {1}\right) / \tau\right)}{N - 1 + \exp \left(\left(M _ {2} - M _ {1}\right) / \tau\right)} \sqrt {\frac {N}{2} \ln \left(\frac {2 | \mathbb {F} |}{\rho}\right)} \tag {26}
$$
Thus, for  $\forall \rho \in (0,1)$ , we conclude that with probability at least  $1 - \rho$ .
$$
\mathcal {L} _ {\text {u n b i a s e d}} \leq \mathcal {L} _ {\mathrm {C L - D R O}} ^ {K L} \leq \tau \widehat {\mathcal {L}} _ {\text {I n f o N C E}} + \frac {M _ {2} \exp \left(\left(M _ {2} - M _ {1}\right) / \tau\right)}{N - 1 + \exp \left(\left(M _ {2} - M _ {1}\right) / \tau\right)} \sqrt {\frac {N}{2} \ln \left(\frac {2 | \mathbb {F} |}{\rho}\right)} \tag {27}
$$
Proof. While Corollary 3.4 has already been proven in [46], we present a brief outline of the proof here for the sake of completeness and to ensure that our article is self-contained. To verify the relationship between  $\tau$  and  $\eta$ , we could utilize the approximate expression of InfoNCE (cf. Equation (31)) and focus on the first order conditions for  $\tau$ . In detail, we have:
$$
- \mathbb {E} _ {P _ {0}} [ f _ {\theta} ] + \inf  _ {\alpha \geq 0} \left\{\mathbb {E} _ {Q _ {0}} [ f _ {\theta} ] + \frac {1}{2 \alpha} \frac {1}{\phi^ {(2)} (1)} \mathbb {V} _ {Q _ {0}} [ f _ {\theta} ] + \alpha \eta \right\}
$$
To find the optimal value of  $\alpha$  (or equivalently,  $\tau$ ), we differentiate the above equation and set it to 0. This yields a fixed-point equation
$$
\tau = \sqrt {\frac {\mathbb {V} _ {Q _ {0}} [ f _ {\theta} ]}{2 \eta}}
$$
The corollary gets proved.
A.4 Proof of Theorem 3.5
Proof. We start with introducing a useful lemma.
Lemma A.6 (Lemma A.2 of [47]). Suppose that  $\phi : \mathbb{R} \to \mathbb{R} \cup \{+\infty\}$  is a closed, convex function such that  $\phi(z) \geq \phi(1) = 0$  for all  $z$ , is two times continuously differentiable around  $z = 1$ , and  $\phi(1) > 0$ . Then
$$
\begin{array}{l} \phi^ {*} (\zeta) = \max  _ {z} \{z \zeta - \phi (z) \} \\ = \zeta + \frac {1}{2 !} \left[ \frac {1}{\phi^ {\prime \prime} (1)} \right] \zeta^ {2} + o \left(\zeta^ {2}\right) \tag {28} \\ \end{array}
$$
Note that most of the  $\phi$ -divergences [54] (e.g., KL divergence, Cressie-Read divergence, Burg entropy, J-divergence,  $\chi^2$ -distance, modified  $\chi^2$ -distance, and Hellinger distance) satisfy the smoothness conditions. When  $n = 2$ ,  $\phi^{*}[\zeta] \approx \zeta + \frac{1}{2}\left[\frac{1}{\phi^{(2)}(1)}\right]\zeta^2$ . Substituting this back to Equation (17) we have:
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {C L - D R O}} ^ {\phi} = - \mathbb {E} _ {P _ {0}} [ f _ {\theta} ] + \min  _ {\alpha \geq 0, \beta} \left\{\alpha \eta - \beta + \alpha \mathbb {E} _ {Q _ {0}} \left[ \phi^ {*} \left(\frac {f _ {\theta} + \beta}{\alpha}\right) \right] \right\} \\ = - \mathbb {E} _ {P _ {0}} [ f _ {\theta} ] + \min  _ {\alpha \geq 0, \beta} \left\{\alpha \eta - \beta + \alpha \mathbb {E} _ {Q _ {0}} \left[ \frac {f _ {\theta} + \beta}{\alpha} + \frac {1}{2} \frac {1}{\phi^ {(2)} (1)} \left(\frac {f _ {\theta} + \beta}{\alpha}\right) ^ {2} \right] \right\} \tag {29} \\ = - \mathbb {E} _ {P _ {0}} [ f _ {\theta} ] + \min  _ {\alpha \geq 0, \beta} \left\{\alpha \eta - \beta + \mathbb {E} _ {Q _ {0}} [ f _ {\theta} + \beta + \frac {1}{2} \frac {1}{\phi^ {(2)} (1) \alpha} (f _ {\theta} + \beta) ^ {2} ] \right\} \\ = - \mathbb {E} _ {P _ {0}} [ f _ {\theta} ] + \min _ {\alpha \geq 0, \beta} \left\{\alpha \eta + \mathbb {E} _ {Q _ {0}} [ f _ {\theta} + \frac {1}{2} \frac {1}{\phi^ {(2)} (1) \alpha} (f _ {\theta} + \beta) ^ {2} ] \right\} \\ \end{array}
$$
If we differentiate it w.r.t.  $\beta$ :
$$
\frac {\partial \left\{\alpha \eta + \mathbb {E} _ {Q _ {0}} \left[ f _ {\theta} + \frac {1}{2} \frac {1}{\phi^ {(2)} (1) \alpha} \left(f _ {\theta} + \beta\right) ^ {2} \right] \right\}}{\partial \beta} = 0 \tag {30}
$$
we have  $\beta^{*} = -\mathbb{E}_{Q_{0}}[f_{\theta}]$ , and the objective transforms into:
$$
\begin{array}{l} - \mathbb {E} _ {P _ {0}} [ f _ {\theta} ] + \inf  _ {\alpha \geq 0, \beta} \left\{\alpha \eta + \mathbb {E} _ {Q _ {0}} [ f _ {\theta} + \frac {1}{2} \frac {1}{\phi^ {(2)} (1) \alpha} (f _ {\theta} + \beta) ^ {2} ] \right\} \\ = - \mathbb {E} _ {P _ {0}} [ f _ {\theta} ] + \inf  _ {\alpha \geq 0} \left\{\mathbb {E} _ {Q _ {0}} [ f _ {\theta} + \frac {1}{2 \alpha} \frac {1}{\phi^ {(2)} (1)} \left(f _ {\theta} - \mathbb {E} _ {Q _ {0}} [ f _ {\theta} ]\right) ^ {2} ] + \alpha \eta \right\} \tag {31} \\ = - \mathbb {E} _ {P _ {0}} [ f _ {\theta} ] + \inf  _ {\alpha \geq 0} \left\{\mathbb {E} _ {Q _ {0}} [ f _ {\theta} ] + \frac {1}{2 \alpha} \frac {1}{\phi^ {(2)} (1)} \mathbb {V} _ {Q _ {0}} [ f _ {\theta} ] + \alpha \eta \right\} \\ \end{array}
$$
Choosing KL-divergence, we have  $\phi^{(2)}(1) = 1$ . Substituting  $\alpha^{*}(\tau)$  into Equation (31) and ignoring the constant  $\alpha \eta$ :
$$
- \mathbb {E} _ {P _ {0}} [ f _ {\theta} ] + \mathbb {E} _ {Q _ {0}} [ f _ {\theta} ] + \frac {1}{2 \tau} \mathbb {V} _ {Q _ {0}} [ f _ {\theta} ]
$$
Then Theorem gets proved.
A.5 Proof of Theorem 4.2
Proof. Regarding this theorem, our proof primarily relies on the variational representation of  $\phi$ -divergence and optimized certainty equivalent (OCE) risk. Towards this end, we start to introduce the basic concepts:
Definition A.7 (OCE [56]). Let  $X$  be a random variable and let  $u$  be a convex, lower-semicontinuous function satisfies  $u(0) = 0$ ,  $u^{*}(1) = 0$ , then optimized certainty equivalent (OCE) risk  $\rho(X)$  is defined as:
$$
\rho (X) = \inf  _ {\lambda \in \mathbb {R}} \left\{\lambda + \mathbb {E} [ u (f - \lambda) ] \right\} \tag {32}
$$
OCE is a type of risk measure that is widely used by both practitioners and academics [59; 56]. With duality theory, its various properties have been inspiring in our study of DRO.
Definition A.8 (Variational formulation).
$$
D _ {\phi} (P | | Q) := \sup  _ {f \in \mathcal {F}} \left\{\mathbb {E} _ {P} [ f ] - \mathbb {E} _ {Q} \left[ \phi^ {*} (f) \right] \right\} \tag {33}
$$
where the supremum is taken over all bounded real-valued measurable functions  $\mathcal{F}$  defined on  $\mathcal{X}$ .
Note that in order to keep consistent with the definition of CL-DRO, we transform Equation (10) to :
$$
D _ {\phi} (P | | Q) = \max  _ {f \in \mathcal {F}} - \mathcal {L} _ {\mathrm {C L - D R O}} ^ {\phi} (P, Q) = \max  _ {f \in \mathcal {F}} \mathbb {E} _ {P} [ f ] - \min  _ {\beta \in \mathbb {R}} \{- \beta + \mathbb {E} _ {Q} [ \phi^ {*} (f + \beta) ] \} \tag {34}
$$
Our proof for this theorem primarily relies on utilizing OCE risk as a bridge and can be divided into two distinct steps:
Step 1:  $\max_{f\in \mathcal{F}} - \mathcal{L}_{\mathrm{CL - DRO}}^{\phi}(P,Q) = \max_{f\in \mathcal{F}}\mathbb{E}_P[f] - \min_{\beta \in \mathbb{R}}\{-\beta +\mathbb{E}_Q[\phi^* (f + \beta)]\} .$
Step 2:  $D_{\phi}(P||Q) = \max_{f\in \mathcal{F}}\mathbb{E}_P[f] - \min_{\beta \in \mathbb{R}}\{-\beta +\mathbb{E}_Q[\phi^* (f + \beta)]\}$
1. We show that  $\max_{f\in \mathcal{F}} - \mathcal{L}_{\mathbf{CL - DRO}}^{\phi}(P,Q) = \max_{f\in \mathcal{F}}\mathbb{E}_P[f] - \min_{\beta \in \mathbb{R}}\{-\beta +\mathbb{E}_Q[\phi^* (f + \beta)]\} .$
$$
\begin{array}{l} - \mathcal {L} _ {\mathrm {C L - D R O}} ^ {\phi} = \mathbb {E} _ {P} [ f ] - \min  _ {\alpha \geq 0, \beta} \max  _ {L} \left\{\mathbb {E} _ {Q} [ f L ] - \alpha \left[ \mathbb {E} _ {Q} [ \phi (L) ] - \eta \right] + \beta \left(\mathbb {E} _ {Q} [ L ] - 1\right) \right\} \\ = \mathbb {E} _ {P} [ f ] - \min  _ {\alpha \geq 0, \beta} \left\{\alpha \eta - \beta + \alpha \mathbb {E} _ {Q} \left[ \max  _ {L} \left\{\frac {f + \beta}{\alpha} L - \phi (L) \right\} \right] \right\} \\ = \mathbb {E} _ {P} [ f ] - \min  _ {\alpha \geq 0, \beta} \left\{\alpha \eta - \beta + \alpha \mathbb {E} _ {Q} \left[ \phi^ {*} \left(\frac {f + \beta}{\alpha}\right) \right] \right\} \tag {35} \\ = \mathbb {E} _ {P} [ f ] - \min  _ {\beta} \left\{\alpha^ {*} \eta - \beta + \alpha^ {*} \mathbb {E} _ {Q} \left[ \phi^ {*} \left(\frac {f + \beta}{\alpha^ {*}}\right) \right] \right\} \\ = \mathbb {E} _ {P} [ f ] - \min  _ {\beta} \left\{- \beta + \alpha^ {*} \mathbb {E} _ {Q} \left[ \phi^ {*} \left(\frac {f + \beta}{\alpha^ {*}}\right) \right] + C o n s t a n t \right\} \\ \end{array}
$$
When  $\alpha^{*} = 1$ , step 1 gets proved.
2. We show that  $D_{\phi}(P||Q) = \max_{f\in \mathcal{F}}\mathbb{E}_P[f] - \min_{\beta \in \mathbb{R}}\{-\beta +\mathbb{E}_Q[\phi^* (f + \beta)]\}$ .
Firstly, we transform  $\mathbb{E}_P[f]$  to  $\mathbb{E}_Q\left[f\frac{dP}{dQ}\right]$  as:
$$
\begin{array}{l} \max  _ {f \in \mathcal {F}} \mathbb {E} _ {P} [ f ] - \min  _ {\beta} \left\{- \beta + \mathbb {E} _ {Q} \left[ \phi^ {*} (f + \beta) \right] \right\} \\ = \max  _ {f \in \mathcal {F}} \mathbb {E} _ {Q} [ f \frac {d P}{d Q} ] - \min  _ {\beta} \left\{- \beta + \mathbb {E} _ {Q} \left[ \phi^ {*} (f + \beta) \right] \right\} \tag {36} \\ \end{array}
$$
Let  $f + \beta = Y$ , we have:
$$
\begin{array}{l} \max  _ {f \in \mathcal {F}} \mathbb {E} _ {Q} \left[ f \frac {d P}{d Q} \right] - \min  _ {\beta} \left\{- \beta + \mathbb {E} _ {Q} \left[ \phi^ {*} (f + \beta) \right] \right\} \\ = \max  _ {Y \in \mathcal {F}} \min  _ {\beta} \mathbb {E} _ {Q} [ (Y - \beta) \frac {d P}{d Q} ] - \left\{- \beta + \mathbb {E} _ {Q} [ \phi^ {*} (Y) ] \right\} (37) \\ = \max  _ {Y \in \mathcal {F}} \mathbb {E} _ {Q} \left[ Y \frac {d P}{d Q} - \phi^ {*} (Y) \right] + \min  _ {\beta} \beta \mathbb {E} _ {Q} \left[ 1 - \frac {d P}{d Q} \right] (37) \\ = \max  _ {Y \in \mathcal {F}} \mathbb {E} _ {Q} \left[ Y \frac {d P}{d Q} - \phi^ {*} (Y) \right] + 0 \\ \end{array}
$$
The first equality follows from replacing  $f + \theta_{1}$  with  $Y$ . The second equality is a re-arrangement for optimizing  $\beta$ . The third equation holds as  $\mathbb{E}_Q[1 - \frac{dP}{dQ}] = 0$ .
Applying Theorem A.3, the last supremum reduces to:
$$
\begin{array}{l} \max  _ {Y \in \mathcal {F}} \mathbb {E} _ {Q} \left[ Y \frac {d P}{d Q} - \phi^ {*} (Y) \right] \\ = \mathbb {E} _ {Q} \left[ \sup  _ {Y \in \mathcal {F}} \left\{Y \frac {d P}{d Q} - \phi^ {*} (Y) \right\} \right] \\ = \mathbb {E} _ {Q} \left[ \phi^ {* *} \left(\frac {d P}{d Q}\right) \right] \tag {38} \\ = \mathbb {E} _ {Q} [ \phi (\frac {d P}{d Q}) ] \\ = D _ {\phi} (P | | Q) \\ \end{array}
$$
where the last equality follows from the fact that  $\phi^{**} = \phi$ . This concludes the proof.
A.6 Proof of  $Q^{*}$
Proof. From theorem3.2, CL-DRO can be rewritten as:
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {C L - D R O}} ^ {\phi} = - \mathbb {E} _ {P _ {0}} [ f _ {\theta} ] + \min  _ {\beta} \left\{\alpha^ {*} \eta - \beta + \alpha^ {*} \mathbb {E} _ {Q _ {0}} \left[ \max  _ {L} \left\{\frac {f _ {\theta} + \beta}{\alpha^ {*}} L - \phi (L) \right\} \right] \right\} \tag {39} \\ = - \mathbb {E} _ {P _ {0}} [ f _ {\theta} ] + \min  _ {\beta} \left\{\alpha^ {*} \eta - \beta + \alpha^ {*} \mathbb {E} _ {Q _ {0}} \left[ \phi^ {*} \left(\frac {f _ {\theta} + \beta}{\alpha^ {*}}\right) \right] \right\} \\ \end{array}
$$
For the inner optimization, we can draw the optimal  $L$  for  $\max_L\{\frac{f_\theta + \beta}{\alpha^*} L - \phi (L)\}$  as:
$$
L = e ^ {\frac {f _ {\theta} + \beta}{\alpha^ {*}}} \tag {40}
$$
For the outer optimization, we can draw the optimal  $\beta$  for  $\min_{\beta}\left\{\alpha^{*}\eta -\beta +\alpha^{*}\mathbb{E}_{Q_0}[e^{\frac{f_\theta + \beta}{\alpha^*}} - 1]\right\}$  as
$$
\beta = \alpha^ {*} \log \mathbb {E} _ {Q _ {0}} e ^ {- \frac {f _ {\theta}}{\alpha^ {*}}} \tag {41}
$$
Then we plug Equation (41) into Equation (40).
$$
L = \frac {e ^ {\frac {f _ {\theta}}{\alpha^ {*}}}}{\mathbb {E} _ {Q _ {0}} \left[ e ^ {\frac {f _ {\theta}}{\alpha^ {*}}} \right]} \tag {42}
$$
Based on the definition of  $L$ , we can derive the expression for  $Q^{*}$ :
$$
Q ^ {*} = \frac {e ^ {\frac {f _ {\theta}}{\alpha^ {*}}}}{\mathbb {E} _ {Q _ {0}} \left[ e ^ {\frac {f _ {\theta}}{\alpha^ {*}}} \right]} Q _ {0} \tag {43}
$$
Figure 5 shows PyTorch-style pseudocode for the standard objective, the adjusted InfoNCE objective. The proposed adjusted reweighting loss is very simple to implement, requiring only two extra lines of code compared to the standard objective.
# pos : exp of inner products for positive examples
# neg : exp of inner products for negative examples
# N : number of negative examples
# t : temperature scaling
# mu : center position
# sigma : height scale
# InfoNCE
standard_loss = -log(pos.sum() / (pos.sum() + neg.sum))
# ADNCE
weight = 1 / (sigma * sqrt(2*pi)) * exp(-0.5 * ((neg-mu)/sigma)**2)
weight = weight / weight.mean()
Adjusted_loss = -log(pos.sum() / (pos.sum() + (neg * weightdetach().sum()))
Figure 5: Pseudocode for our proposed adjusted InfoNCE objective, as well as the original NCE contrastive objective. The implementation of our adjusted reweighting method only requires two additional lines of code compared to the standard objective.
B.1 Visual Representation
Model. For contrastive learning on images, we adopt SimCLR [4] as our baseline and follow the same experimental setup as [13]. Specifically, we use the ResNet-50 network as the backbone. To ensure a fair comparison, we set the embedded dimension to 2048 (the representation used in linear readout) and project it into a 128-dimensional space (the actual embedding used for contrastive learning). Regarding the temperature parameter  $\tau$ , we use the default value  $\tau_0$  of 0.5 in most researches, and we also perform grid search on  $\tau$  varying from 0.1 to 1.0 at an interval of 0.1, denoted by  $\tau^*$ . The best parameters for each dataset is reported in Table 6. Note that  $\{\cdot\}$  indicates the range of hyperparameters that we tune and the numbers in bold are the final settings. For  $\alpha$ -CL, we follow the setting of [16], where  $p = 4$  and  $\tau = 0.5$ . We use the Adam optimizer with a learning rate of 0.001 and weight decay of  $1e - 6$ . All models are trained for 400 epochs.
Table 6: hyperparameters setting on each datasets.
DATASETS	CIFAR10	STL10	CIFAR100
BEST τ	{0.1, 0.2, 0.3, 0.4, 0.5, 0.6}	{0.1, 0.2, 0.3, 0.4, 0.5, 0.6}	{0.1, 0.2, 0.3, 0.4, 0.5, 0.6}
μ	{0.5, 0.6, 0.7, 0.8, 0.9}	{0.5, 0.6, 0.7, 0.8, 0.9}	0.5, 0.6, 0.7, 0.8, 0.9}
σ	{0.5, 1.0}	{0.5, 1.0}	{0.5, 1.0}
Noisy experiments in Section 3.4. To investigate the relationship between the temperature parameter  $\tau$  (or  $\eta$ ) and the noise ratio, we follow the approach outlined in [13] and utilize the class information of each image to select negative samples as a combination of true negative samples and false negative samples. Specifically,  $r_{ratio} = 0$  indicates all negative samples are true negative samples,  $r_{ratio} = 0.5$  suggests  $50\%$  of true positive samples existing in negative samples,  $r_{ratio} = 1$  means uniform sampling.
Variance analysis in Section 3.4. To verify the mean-variance objective of InfoNCE, we adopt the approach outlined in [60] and record the negative prediction scores for 256 samples (assuming a batch size of 256) in each minibatch. Specifically, we randomly select samples from a batch to calculate the statistics and visualize them. (1) For positive samples, we calculate cosine similarity by taking the inner product after normalization, and retain the mean value of the 256 positive scores as 'pos mean'. (2) For negative samples, we average the means and variances of 256 negative samples to show the statistical characteristics of these N negative samples ('mean neg; var neg'). We record this data at each training step to track score distribution throughout the training process.
B.2 Sentence Representation
For the sentence contrastive learning, we adopt the approach outlined in [9] and evaluate our method on 7 popular STS datasets: STS tasks from 2012-2016, STS-B and SICK-R. We utilize the SentEval toolkit to obtain all 7 datasets. Each dataset includes sentence pairs which are rated on a scale of 0 to 5, indicating the degree of semantic similarity. To validate the effective of our proposed method, we utilize several methods as baselines: average GloVe embeddings, BERT-flow, BERT-whitening, CT-BERT and SimCSE. The best parameters for each dataset is reported in Table 7. To ensure fairness, we employed the official code, which can be accessed at https://github.com/princeton-nlp/SimCSE.
Table 7: hyperparameters setting on sentence CL. Note that  $\{\cdot\}$  indicates the range of hyperparameters that we tune and the numbers in bold are the final settings.
DATASETS	SIMCSE-BERTBASE	SIMCSE-ROBERTABASE
BEST τ	{0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09, 0.10, 0.15, 0.20}	{0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09, 0.10, 0.15, 0.20}
μ	{0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9}	{0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 1.5, 2.0, 2.5, 3.0}
σ	{0.5, 1.0}	{0.5, 1.0}
B.3 Graph Representation
For the graph contrastive learning experiments on TU-Dataset [61], we adopted the same experimental setup as outlined in [12]. The dataset statistics can be found in Table 8. To ensure fairness, we employed the official code, which can be accessed at https://github.com/Shen-Lab/GraphCL/tree/master/unsupervised_TU. We made only modifications to the script by incorporating our ADNCE method and conducting experiments on the hyper-parameter  $\mu \in \{0.5, 0.6, 0.7, 0.8, 0.9, 1.0\}$  and  $\sigma = 1$  on most datasets. Each parameter was repeated from scratch five times, and the best parameter was selected by evaluating on the validation dataset. The best parameters for each dataset is reported in Table 9.
We summarize the statistics of TU-datasets [61] for unsupervised learning in Table 8. Table 10 demonstrates the consistent superiority of our proposed ADNCE approach.