SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Junkang  $\mathbf{W}\mathbf{u}^{1}$  Jiawei Chen\* Jiancan  $\mathbf{W}\mathbf{u}^{1}$  Wentao Shi Xiang Wang\* Xiangnan He\*
2Zhejiang University
{jkwu0909, wujcan, xiangwang1223, xiangnanhe}@gmail.com,
sleepyhunt@zju.edu.cn, shiwentao123@mail.ustc.edu.cn
This study reveals the inherent tolerance of contrastive learning (CL) towards sampling bias, wherein negative samples may encompass similar semantics (e.g., labels). However, existing theories fall short in providing explanations for this phenomenon. We bridge this research gap by analyzing CL through the lens of distributionally robust optimization (DRO), yielding several key insights: (1) CL essentially conducts DRO over the negative sampling distribution, thus enabling robust performance across a variety of potential distributions and demonstrating robustness to sampling bias; (2) The design of the temperature  $\tau$  is not merely heuristic but acts as a Lagrange Coefficient, regulating the size of the potential distribution set; (3) A theoretical connection is established between DRO and mutual information, thus presenting fresh evidence for "InfoNCE as an estimate of MI" and a new estimation approach for  $\phi$ -divergence-based generalized mutual information. We also identify CL's potential shortcomings, including over-conservatism and sensitivity to outliers, and introduce a novel Adjusted InfoNCE loss (ADNCE) to mitigate these issues. It refines potential distribution, improving performance and accelerating convergence. Extensive experiments on various domains (image, sentence, and graphs) validate the effectiveness of the proposal. The code is available at https://github.com/junkangwu/ADNCE.
Recently, contrastive learning (CL) [1; 2; 3] has emerged as one of the most prominent self-supervised methods, due to its empirical success in computer vision [4; 5; 6; 7; 8], natural language processing [9; 10] and graph [11; 12]. The core idea is to learn representations that draw positive samples (e.g., augmented data from the same image) nearby and push away negative samples (e.g., augmented data from different images). By leveraging this intuitive concept, unsupervised learning has even begun to challenge supervised learning.
Despite its effectiveness, recent study [13] has expressed concerns about sampling bias in CL. Negative samples are often drawn uniformly from the training data, potentially sharing similar semantics (e.g., labels). Blindly separating these similar instances could result in a performance decline. To address sampling bias, existing research has proposed innovative solutions that either approximate the optimal distribution of negative instances [13; 14] or integrate an auxiliary detection module to identify false negative samples [15]. Nevertheless, our empirical analyses challenge the prevailing understanding of CL. Our findings indicate that CL inherently exhibits robustness towards
* Jiawei Chen and Xiangnan He are the corresponding authors.
†Xiang Wang is also affiliated with Institute of Artificial Intelligence, Institute of Dataspace, Hefei Comprehensive National Science Center.
sampling bias and, by tuning the temperature  $\tau$ , can even achieve comparable performance with representative methods for addressing sampling bias (e.g., DCL [13] and HCL [14]). This revelation prompts two intriguing questions: 1) Why does CL display tolerance to sampling bias? and 2) How can we thoroughly comprehend the role of the temperature  $\tau$ ? Despite recent analytical work on CL from various perspectives [13; 14; 16], including mutual information and hard-mining, these studies fail to provide a satisfactory explanation for our observations.
To address this research gap, we investigate CL from the standpoint of Distributionally Robust Optimization (DRO) [17; 18; 19; 20; 21; 22]. DRO refers to a learning algorithm that seeks to minimize the worst-case loss over a set of possible distributions. Through rigorous theoretical analysis, we prove that CL essentially performs DRO optimization over a collection of negative sampling distributions that surround the uniform distribution, constrained by a KL-divergence-based measure (we term the DRO-type equivalent expression of CL as CL-DRO). By enabling CL to perform well across various potential distributions, DRO equips it with the capacity to alleviate sampling bias. Furthermore, our findings emphasize that the design of  $\tau$  is not merely heuristic but acts as a Lagrange Coefficient, regulating the size of the potential distribution set. By integrating the principles of DRO, we can also gain insights into the key properties of CL such as hard-mining, and investigate novel attributes such as variance control.
Furthermore, we delve into the properties of CL-DRO, transcending the boundary of KL divergence, but exploring a broader family of distribution divergence metrics —  $\phi$ -divergence. It exhibits greater flexibility and potentially superior characteristics compared to KL-divergence. Interestingly, we establish the equivalence between CL-DRO under any  $\phi$ -divergence constraints and the corresponding tight variational representation of  $\phi$ -divergence [23]. This discovery provides direct theoretical support for the assertion that "InfoNCE is an estimate of mutual information (MI)", without requiring redundant approximation as before work [24; 3]. Additionally, it provides a novel perspective for estimating arbitrary  $\phi$ -divergence, consequently enabling the estimation of  $\phi$ -divergence-based mutual information.
In addition, we discover some limitations in CL through the lens of DRO. One is that the DRO paradigm is too conservative, focusing on worst-case distribution with giving unadvisedly over-large weights on hard negative samples. Secondly, the presence of outliers can significantly impact DRO [25], resulting in both a drop in performance and training instability. In fact, both challenges stem from the unreasonably worst-case distribution weights assigned to negative samples. To address this weakness, we propose ADjusted InfoNCE (ADNCE) that reshapes the worst-case distribution without incurring additional computational overhead, allowing us to explore the potentially better negative distribution. Empirical studies show that our method can be applied to various domains (such as images, sentences, and graphs), and yield superior performance with faster convergence.
Contrastive Learning. With numerous CL methodologies (e.g., SimCLR [4], Moco [6], BYOL [26], SwAV [27], Barlow Twins [8]) having been proposed and demonstrated the ability to produce high-quality representations for downstream tasks, theoretical research in this field has begun to emerge. Wang et al. [28] revealed the hard-mining property and Liu et al. [29] emphasized its robustness against dataset imbalance. Meanwhile, Xue et al. [30] showed that representations learned by CL provably boosts robustness against noisy labels, and Ko et al. [31] investigated the relationship between CL and neighborhood component analysis. More closely related to our work is the result of Tian et al. [16] that analyzed CL from minmax formulation.
Distributionally Robust Optimization. In contrast to conventional robust optimization approaches [32; 33; 34], DRO aims to solve the distribution shift problem by optimizing the worst-case error in a pre-defined uncertainty set, which is often constrained by  $\phi$ -divergence [20; 17], Wasserstein distance [21; 22; 19; 18] and shape constraints [35; 36; 37]. Meanwhile, Michel et al. [38; 39] parametrized the uncertainty set, allowing more flexibility in the choice of its architecture. Zhai et al. [25] focused on issues related to the existing DRO, such as sensitivity to outliers, which is different from the aforementioned research.
Mutual Information. The MI between two random variables,  $X$  and  $Y$ , measures the amount of information shared between them. To estimate MI, the Donsker-Varadhan target [40] expresses the lower bound of the Kullback-Leibler (KL) divergence, with MINE [24] being its parameterized
version. However, according to studies [41; 42], the MINE estimator may suffer from high variance. To address this issue, InfoNCE [3] extends the unnormalized bounds to depend on multiple samples, thereby eliminating the constraints of both MINE's reliance on conditional and marginal densities. Another line of MI focuses on boundedness and low variance. Towards this end, RPC [43] introduced relative parameters to regularize the objective and FLO [44] leveraged Fenchel-Legendre optimization. Nevertheless, they are still hard to explain how MI optimization elucidates the properties of CL and facilitates effective CL in practice.
3 Contrastive learning as Distributionally Robust Optimization
In this section, we firstly uncover the tolerance of CL towards sampling bias. Then we analyze CL and  $\tau$  from the perspective of DRO. Finally, we empirically verify the above findings.
Notation. Contrastive Learning is conducted on two random variables  $X, Y$  over a feature space  $\mathcal{A}$ . Suppose we have original samples  $x$ , which obey the distribution  $P_{X}$ , positive samples drawn from the distribution  $(x, y^{+}) \sim P_{Y|X}$ , and negative samples selected from the distribution  $(x, y) \sim P_{Y}$ . The goal of CL is to learn an embedding  $g_{\theta}: \mathcal{A} \to \mathbb{R}^d$  that maps an observation (i.e.,  $x, y, y^{+}$ ) into a hypersphere. Let  $f(g_{\theta}(x), g_{\theta}(y))$  be the similarity between two observations  $x$  and  $y$  on the embedding space. CL draws positive samples closer (i.e.,  $f(g_{\theta}(x), g_{\theta}(y^{+})) \uparrow$  for  $(x, y^{+}) \sim P_{XY}$ ) and pushes the negative samples away (i.e.,  $f(g_{\theta}(x), g_{\theta}(y)) \downarrow$  for  $(x, y) \sim P_{X}P_{Y}$ ). For brevity, in this work, we simply shorten the notations  $P_{Y|X}$  as  $P_{0}$ , and  $P_{Y}$  as  $Q_{0}$ . Also, we primarily focus on InfoNCE loss  $\mathcal{L}_{\mathrm{InfoNCE}}$  as a representative for analyses, while other CL's loss functions exhibit similar properties. InfoNCE can be written as follow:
$$
\mathcal {L} _ {\text {I n f o N C E}} = - \mathbb {E} _ {P _ {X}} \left[ \mathbb {E} _ {P _ {0}} \left[ f _ {\theta} \left(x, y ^ {+}\right) / \tau \right] - \log \mathbb {E} _ {Q _ {0}} \left[ e ^ {f _ {\theta} \left(x, y\right) / \tau} \right] \right] = - \mathbb {E} _ {P _ {X}} \mathbb {E} _ {P _ {0}} \left[ \log \frac {e ^ {f _ {\theta} \left(x , y ^ {+}\right) / \tau}}{\mathbb {E} _ {Q _ {0}} \left[ e ^ {f _ {\theta} \left(x , y\right) / \tau} \right]} \right]. \tag {1}
$$
In practical applications of contrastive learning (CL), negative samples  $(y)$  are typically drawn uniformly from the training data, which might have similar semantics (for instance, labels). This introduces a potential issue of sampling bias, as posited by Chuang et al. [13], since similar samples are being forcefully separated. Recent research [15; 13] observed that compared to the employment of ideal negative sampling that selects instances with distinctly dissimilar semantics, the existence of sampling bias could result in a significant reduction in performance.
In this study, we observe an intriguing phenomenon where CL inherently demonstrates re
Table 1: InfoNCE has the ability to mitigate sampling bias. We compare the performance of various CL methods with/without fine-tuning  $\tau$  (marked with  $\tau^{*} / \tau_{0}$ ). We also report relative improvements compared with SimCLR  $(\tau^{*})$ .
Model	CIFAR10	STL10
Top-1	τ	Top-1	τ
SimCLR(τ0)	91.10	0.5	81.05	0.5
SimCLR(τ*)	92.19	0.3	87.91	0.2
DCL(τ0)	92.00 (-0.2%)	0.5	84.26 (-4.2%)	0.5
DCL(τ*)	92.09 (-0.1%)	0.3	88.20 (+1.0%)	0.2
HCL(τ0)	92.12 (-0.0%)	0.5	87.44 (-0.5%)	0.5
HCL(τ*)	92.10 (-0.0%)	0.3	87.46 (-0.5%)	0.2
silience to sampling bias. We empirically evaluate CL on two benchmark datasets, CIFAR10 and STL10, with the findings detailed in Table 1. Notably, we make the following observations: 1) By fine-tuning the temperature  $\tau$ , basic SimCLR demonstrates significant improvement, achieving performance levels comparable to methods specifically designed to address sampling bias (i.e.,  $\mathrm{SimCLR}(\tau^{*})$  versus  $\mathrm{SimCLR}(\tau_0)$ , DCL and HCL); 2) With an appropriately selected  $\tau$ , the relative improvements realized by DCL and HCL are marginal. These insights lead us to pose two compelling questions: 1) Why does CL exhibit tolerance to sampling bias? and 2) What role does  $\tau$  play, and why is it so important?
3.2 Understanding CL from DRO
In this subsection, we first introduce DRO and then analyze CL from the perspective of DRO.
Note that our formula adheres to the commonly used InfoNCE expression (e.g., the SimCLR setting [4]), in which the denominator does not include  $\mathbb{E}_{P_0}[e^{f\theta (x,y^+) / \tau}]$ . This is also used in Yeh et al. [45].
A. Distributionally Robust Optimization (DRO).
The success of machine learning heavily relies on the premise that the test and training distributions are identical (a.k.a. iid assumption). How to handle different data distributions (a.k.a. distribution shift) poses a great challenge to machine learning systems. Fortunately, Distributionally Robust Optimization (DRO) provides a potential solution to mitigate this problem. DRO aims to minimize the worst-case expected loss over a set of potential distributions  $Q$ , which surround the training distribution  $Q_{0}$  and are constrained by a distance metric  $\mathcal{D}\phi$  within a specified robust radius  $\eta$ . Formally, DRO can be written as:
$$
\mathcal {L} _ {\mathrm {D R O}} = \max  _ {Q} \mathbb {E} _ {Q} [ \mathcal {L} (x; \theta) ] \quad s. t. D _ {\phi} (Q | | Q _ {0}) \leq \eta , \tag {2}
$$
where  $D_{\phi}(Q||Q_0)$  measures the  $\phi$ -divergence between two distributions, and  $\mathcal{L}(x;\theta)$  denotes the training loss on input  $x$ . Intuitively, models incorporating DRO enjoy stronger robustness due to the presence of  $Q$  that acts as an "adversary", optimizing the model under a distribution set with adversarial perturbations instead of a single training distribution.
B. Analyzing CL from DRO. We first introduce the CL-DRO objective:
Definition 3.1 (CL-DRO). Let  $P_0$  and  $Q_0$  be the distributions of positive and negative samples respectively. Let  $\eta$  be the robust radius. CL-DRO objective is defined as:
$$
\mathcal {L} _ {\mathrm {C L - D R O}} ^ {\phi} = - \mathbb {E} _ {P _ {X}} \left[ \mathbb {E} _ {P _ {0}} \left[ f _ {\theta} \left(x, y ^ {+}\right) \right] - \max  _ {Q} \mathbb {E} _ {Q} \left[ f _ {\theta} (x, y) \right] \right] \quad s. t. D _ {\phi} (Q | | Q _ {0}) \leq \eta . \tag {3}
$$
The objective CL-DRO can be understood as the DRO enhancement of the basic objective  $L_{\mathrm{basic}} = -\mathbb{E}_{P_X}\left[\mathbb{E}_{P_0}[f_\theta (x,y^+)] - \mathbb{E}_{Q_0}[f_\theta (x,y)]\right]$ , which aims to increase the embedding similarity between the positive instances and decreases that of the negative ones. CL-DRO imporves  $L_{\mathrm{basic}}$  by incorporating DRO on the negative side, where  $f_{\theta}(x,y)$  is optimized a range of potential distributions. Consequently, CL-DRO equips the model with resilience to distributional shifts of negative samples.
Theorem 3.2. By choosing KL divergence  $D_{KL}(Q||Q_0) = \int Q\log \frac{Q}{Q_0} dx$ , optimizing CL-DRO (cf. Equation (3)) is equivalent to optimizing CL (InfoNCE,cf. Equation (1)):
$$
\begin{array}{l} \mathcal {L} _ {C L - D R O} ^ {K L} = - \mathbb {E} _ {P _ {X}} \left[ \mathbb {E} _ {P _ {0}} \left[ f _ {\theta} \left(x, y ^ {+}\right) \right] - \min  _ {\alpha \geq 0, \beta} \max  _ {Q \in \mathbb {Q}} \left\{\mathbb {E} _ {Q} \left[ f _ {\theta} (x, y) \right] - \alpha \left[ D _ {K L} \left(Q | | Q _ {0}\right) - \eta \right] + \beta \left(\mathbb {E} _ {Q _ {0}} \left[ \frac {Q}{Q _ {0}} \right] - 1\right) \right\} \right] \\ = - \mathbb {E} _ {P _ {X}} \mathbb {E} _ {P _ {0}} \left[ \alpha^ {*} (\eta) \log \frac {e ^ {f _ {\theta} (x , y ^ {+}) / \alpha^ {*} (\eta)}}{\mathbb {E} Q _ {0} \left[ e ^ {f _ {\theta} (x , y) / \alpha^ {*} (\eta)} \right]} \right] + C o n s t a n t \\ = \alpha^ {*} (\eta) \mathcal {L} _ {\text {I n f o N C E}} + \text {C o n s t a n t}, \tag {4} \\ \end{array}
$$
where  $\alpha, \beta$  represent the Lagrange multipliers, and  $\alpha^{*}(\eta)$  signifies the optimal value of  $\alpha$  that minimizes the Equation (4), serving as the temperature  $\tau$  in  $CL$ .
The proof is presented in Appendix A.1. Theorem 3.2 admits the merit of CL as it is equivalent to performing DRO on the negative samples. The DRO enables CL to perform well across various potential distributions and thus equips it with the capacity to alleviate sampling bias. We further establish the connection between InfoNCE loss with the unbiased loss that employs ideal negative distribution  $L_{\mathrm{unbiased}} = -\mathbb{E}_{P_X}\left[\mathbb{E}_{P_0}[f_\theta (x,y^+)] - \mathbb{E}_{Q^{\mathrm{iddeal}}}[f_\theta (x,y)]\right]$ .
Theorem 3.3. [Generalization Bound] Let  $\widehat{\mathcal{L}}_{InfoNCE}$  be an estimation of InfoNCE with  $N$  negative samples. Given any finite hypothesis space  $\mathbb{F}$  of models, suppose  $f_{\theta} \in [M_1, M_2]$  and the ideal negative sampling distribution  $Q^{ideal}$  satisfies  $D_{KL}(Q^{ideal}||Q_0) \leq \eta$ , we have that with probability at least  $1 - \rho$ :
$$
\mathcal {L} _ {\text {u n b i a s e d}} \leq \tau \widehat {\mathcal {L}} _ {\text {I n f o N C E}} + \mathcal {B} (\rho , N, \tau), \tag {5}
$$
where  $\mathcal{B}(\rho, N, \tau) = \frac{M_2 \exp((M_2 - M_1) / \tau)}{N - 1 + \exp((M_2 - M_1) / \tau)} \sqrt{\frac{N}{2} \ln\left(\frac{2|\mathbb{F}|}{\rho}\right)}$ .
As we mainly focus on model robustness on negative distribution, here we disregard the constant term present in Equation (4), and omit the error from the positive instances. The proof is presented in
Appendix A.2. Theorem 3.3 exhibits the unbiased loss is upper bound by InfoNCE if a sufficiently large data size is employed (considering  $\mathcal{B}(\rho, N, \tau) \to 0$  as  $N \to 0$ ). This bound illustrates how  $\tau$  impacts the model performance, which will be detailed in the next subsection.
3.3 Understanding  $\tau$  from DRO
Building on the idea of DRO, we uncover the role of  $\tau$  from the following perspectives:
A. Adjusting robust radius. The temperature  $\tau$  and the robust radius  $\eta$  conform to:
Corollary 3.4. [The optimal  $\alpha$ -Lemma 5 of Faury et al. [46]] The value of the optimal  $\alpha$  (i.e.,  $\tau$ ) can be approximated as follow:
$$
\tau \approx \sqrt {\mathbb {V} _ {Q _ {0}} [ f _ {\theta} (x , y) ] / 2 \eta}, \tag {6}
$$
where  $\mathbb{V}_{Q_0}[f_\theta (x,y)]$  denotes the variance of  $f_{\theta}(x,y)$  under the distribution  $Q_{0}$ .
This corollary suggests that the temperature parameter  $\tau$  is a function of the robust radius  $\eta$  and the variance of  $f_{\theta}$ . In practical applications, tuning  $\tau$  in CL is essentially equivalent to adjusting the robustness radius  $\eta$ .
The aforementioned insight enhances our understanding of the impact of temperature on CL performance. An excessively large value of  $\tau$  implies a small robustness radius  $\eta$ , potentially violating the condition  $D_{KL}(Q^{\mathrm{ideal}}||Q_0)\leq \eta$  as stipulated in Theorem 3.2. Intuitively, a large  $\tau$  delineates a restricted distribution set in DRO that may fail to encompass the ideal negative distribution, leading to diminished robustness. Conversely, as  $\tau$  decreases, the condition  $D_{KL}(Q^{\mathrm{ideal}}||Q_0)\leq \eta$  may be satisfied, but it expands the term  $\mathcal{B}(\rho ,N,\tau)$  and loosens the generalization bound. These two factors establish a trade-off in the selection of  $\tau$ , which we will empirically validate in Subsection 3.4.
B. Controlling variance of negative samples.
Theorem 3.5. Given any  $\phi$ -divergence, the corresponding CL-DRO objective could be approximated as a mean-variance objective:
$$
\mathcal {L} _ {C L - D R O} ^ {\phi} \approx - \mathbb {E} _ {P _ {X}} \left[ \mathbb {E} _ {P _ {0}} \left[ f _ {\theta} \left(x, y ^ {+}\right) \right] - \left(\mathbb {E} _ {Q _ {0}} \left[ f _ {\theta} (x, y) \right] + \frac {1}{2 \tau} \frac {1}{\phi^ {(2)} (1)} \cdot \mathbb {V} _ {Q _ {0}} \left[ f _ {\theta} (x, y) \right]\right) \right], \tag {7}
$$
where  $\phi^{(2)}(1)$  denotes the second derivative value of  $\phi (\cdot)$  at point 1, and  $\mathbb{V}_{Q_0}[f_\theta ]$  denotes the variance of  $f$  under the distribution  $Q_{0}$ .
Specially, if we consider KL divergence, the approximation transforms:
$$
\mathcal {L} _ {C L - D R O} ^ {K L} \approx - \mathbb {E} _ {P _ {X}} \left[ \mathbb {E} _ {P _ {0}} \left[ f _ {\theta} \left(x, y ^ {+}\right) \right] - \left(\mathbb {E} _ {Q _ {0}} \left[ f _ {\theta} (x, y) \right] + \frac {1}{2 \tau} \mathbb {V} _ {Q _ {0}} \left[ f _ {\theta} (x, y) \right]\right) \right]. \tag {8}
$$
The proof is presented in Appendix A.4. Theorem 3.5 provides a mean-variance formulation for CL. Compared to the basic objective  $L_{\mathrm{basic}}$ , CL introduces an additional variance regularization on negative samples, governed by the parameter  $\tau$ , which modulates the fluctuations in similarity among negative samples. The efficacy of variance regularizers has been extensively validated in recent research [20; 47; 48]. We consider that variance control might be a reason of the success of CL.
C. Hard-mining. Note that the worst-case distribution in CL-DRO can be written as  $Q^{*} = Q_{0}\frac{\exp[f_{\theta}(x,y^{+}) / \tau]}{\mathbb{E}_{Q_{0}}\exp[f_{\theta}(x,y) / \tau]}$  (refer to Appendix A.6). This implies that the model in CL is optimized under the negative distribution  $Q^{*}$ , where each negative instance is re-weighted by  $\frac{\exp[f_{\theta}(x,y^{+}) / \tau]}{\mathbb{E}_{Q_{0}}\exp[f_{\theta}(x,y) / \tau]}$ . This outcome vividly exhibits the hard-mining attribute of CL, with  $\tau$  governing the degree of skewness of the weight distribution. A smaller  $\tau$  suggests a more skewed distribution and intensified hard-mining, while a larger  $\tau$  signifies a more uniform distribution and attenuated hard-mining. Our research yields similar conclusions to those found in [16; 28], but we arrive at this finding from a novel perspective.
3.4 Empirical Analysis
In this section, we conduct the following experiments to verify the above conclusions.
Figure 2:  $\tau$  controls the variance of negative samples score. We alter the value of  $\tau$  and record corresponding variance of negative samples and mean value of positive samples.
Figure 1:  $\tau$  varies with the level of sampling bias. We report the relative top1 accuracy of SimCLR w.r.t. different  $\tau$  across different rate  $r$  of false negative samples.
$\tau$  varies with the level of sampling bias. To study the effect of  $\tau$  on model performance, we conduct experiments on the CIFAR10 dataset and manipulate the ratio of false negative samples based on the ground-truth labels. A ratio of 1 implies retaining all false negative instances in the data, while a ratio of 0 implies eliminating all such instances. The detailed experimental setup can refer to the Appendix B. From Figure 1, we make two observations: 1) There is an evident trade-off in the selection of  $\tau$ . As  $\tau$  increases from 0.2 to 1, the model's performance initially improves, then declines, aligning with our theoretical analysis. 2) As the ratio of false negative instances increases ( $r$  ranges from 0 to 1), the optimal  $\tau$  decreases. This observation is intuitive — a higher ratio of false negative instances implies a larger distribution gap between the employed negative distribution and the ideal, thereby necessitating a larger robust radius  $\eta$  to satisfy the condition  $D_{KL}(Q^{\mathrm{ideal}}||Q_0)\leq \eta$ . As Corollary 3.4, a larger robust radius corresponds to a smaller  $\tau$ .