SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chenlu Ye*
The Hong Kong University of Science and Technology
cyeab@connect.ust.hk
Rui Yang*
ryangam@connect.ust.hk
Quanquan Gu
qgu@cs.ucla.edu
tongzhang@ust.hk
We investigate the problem of corruption robustness in offline reinforcement learning (RL) with general function approximation, where an adversary can corrupt each sample in the offline dataset, and the corruption level  $\zeta \geq 0$  quantifies the cumulative corruption amount over  $n$  episodes and  $H$  steps. Our goal is to find a policy that is robust to such corruption and minimizes the suboptimality gap with respect to the optimal policy for the uncorrupted Markov decision processes (MDPs). Drawing inspiration from the uncertainty-weighting technique from the robust online RL setting [18, 55], we design a new uncertainty weight iteration procedure to efficiently compute on batched samples and propose a corruption-robust algorithm for offline RL. Notably, under the assumption of single policy coverage and the knowledge of  $\zeta$ , our proposed algorithm achieves a suboptimality bound that is worsened by an additive factor of  $\mathcal{O}(\zeta \cdot (\mathrm{CC}(\lambda, \widehat{\mathcal{F}}, \mathcal{Z}_n^H))^{1/2}(C(\widehat{\mathcal{F}}, \mu))^{-1/2}n^{-1})$  due to the corruption. Here  $\mathrm{CC}(\lambda, \widehat{\mathcal{F}}, \mathcal{Z}_n^H)$  is the coverage coefficient that depends on the regularization parameter  $\lambda$ , the confidence set  $\widehat{\mathcal{F}}$ , and the dataset  $\mathcal{Z}_n^H$ , and  $C(\widehat{\mathcal{F}}, \mu)$  is a coefficient that depends on  $\widehat{\mathcal{F}}$  and the underlying data distribution  $\mu$ . When specialized to linear MDPs, the corruption-dependent error term reduces to  $\mathcal{O}(\zeta dn^{-1})$  with  $d$  being the dimension of the feature map, which matches the existing lower bound for corrupted linear MDPs. This suggests that our analysis is tight in terms of the corruption-dependent term.
Offline reinforcement learning (RL) has received tremendous attention recently because it can tackle the limitations of online RL in real-world applications, e.g., healthcare [45] and autonomous driving [35], where collecting online data is risky, expensive and even infeasible. By leveraging a batch of pre-collected datasets, offline RL aims to find the optimal policy that is covered by the dataset without further interaction with the environment. Due to the restriction of the offline dataset, the utilization of pessimism in the face of uncertainty is widespread [1, 2, 53, 40, 16] and plays a central role in
providing theoretical guarantees for efficient learning [24, 36, 44, 48, 51, 56, 57, 63, 52]. Notably, these theoretical works demonstrate that a single-policy coverage is sufficient to guarantee sample efficiency.
In this paper, we study offline RL under adversarial corruption and with general function approximation. Adversarial corruption refers to adversarial attacks on the reward functions and transition dynamics on the data at each step before the learner can access the dataset. The learner only knows the cumulative corruption level and cannot tell whether the corruption occurs at each data point. Our corruption formulation subsumes the model misspecification [22] and the fixed fraction of data contamination [60] as special cases. Various real-world problems are under the threat of adversarial corruption, such as chatbots misled by discriminative or unethical conversations [32, 61], and autonomous vehicles tricked by hacked navigation instructions or deliberately contaminated traffic signs [12]. On the other hand, general function approximation (approximating the value function with a nonlinear function class, such as deep neural networks) plays a pivotal role in modern large-scale RL problems, such as large language model [10], robotics [26] and medical treatment [28]. Recent works have established different frameworks to explore the minimal structure condition for the function class that enables sample efficiency [19, 47, 9, 21, 14, 6, 64]. In particular, Wang et al. [47] leverage the concept of eluder dimension [39] and construct the least squares value iteration (LSVI) framework, which establishes optimism at each step for online RL.
For adversarial corruption and general function approximation, a significant amount of research has focused on the online setting. However, offline RL in this setting is still understudied due to restricted coverage conditions and lack of adaptivity. One notable exception is Zhang et al. [60], which assumes  $\epsilon$ -fraction of the offline dataset is corrupted, and their algorithm suffers from a suboptimal bound on the corruption term. Our work moves a step further and achieves corruption robustness under the LSVI framework [47] in offline RL with general function approximation by generalizing the uncertainty weighting technique [18, 55]. We also propose an algorithm robust to an additional known distribution shift. Due to space limit, we defer it to Appendix C. We summarize our contributions as follows:
- We formally define the corruption level in offline RL. With knowledge of the corruption level, we design an algorithm that draws inspiration from the PEVI algorithm [24] and the uncertainty-weighting technique. The uncertainty for each data point, serving as the bonus function, is quantified by its informativeness with respect to the whole dataset. We propose the uncertainty weight iteration algorithm to calculate the weights efficiently and prove that the output, an approximation of the uncertainty, is sufficient to control the corruption term.
- Theoretically, our proposed algorithm enjoys a suboptimality bound of  $\tilde{\mathcal{O}}(H \cdot \mathrm{CC}^{1/4}(\ln N)^{1/2}(C(\widehat{\mathcal{F}},\mu))^{-1/4}n^{-1/2} + \zeta \cdot \mathrm{CC}^{1/2}(C(\widehat{\mathcal{F}},\mu))^{-1/2}n^{-1})$ , where  $H$  is the episode length,  $n$  is the number of episodes,  $N$  is the covering number, CC is the coverage coefficient, and  $C(\widehat{\mathcal{F}},\mu)$  is the coefficient depicting how well the underlying data distribution  $\mu$  explores the feature space, and  $\widehat{\mathcal{F}}$  is the confidence set. The corruption-dependent term reduces to  $\mathcal{O}(\zeta dn^{-1})$  in the linear model of dimension  $d$ , thus matching the lower bound for corrupted linear MDPs. It is worth highlighting that our novel analysis enables us to eliminate the uncertainty-related weights from the coverage condition.
- Motivated by our theoretical findings, we present a practical offline RL algorithm with uncertainty weighting and demonstrate its efficacy under diverse data corruption scenarios. Our practical implementation achieves a  $104\%$  improvement over the previous state-of-the-art uncertainty-based offline RL algorithm under data corruption, demonstrating its potential for effective deployment in real-world applications.
Corruption-Robust Bandits and RL. There is an emerging body of theoretical literature on bandits and online RL with corruption. The adversarial corruption is first formulated in the multi-armed bandit problem by Lykouris et al. [30], where an adversary corrupts the reward in each round  $t$  by  $\zeta_t$  and the corruption level is measured by  $\zeta = \sum_{t=1}^{T} |\zeta_t|$ . Then, a lower bound with a linear dependence on  $\zeta$  is constructed by Gupta et al. [17], indicating that the ideal regret bound should achieve a "parallel" relationship:  $\text{Regret}(T) = o(T) + \mathcal{O}(\zeta)$ , and the corruption-independent term approximates the non-corrupted bound. When extending to linear contextual bandits, a line of
work [4,8,13,27,62,25] propose various methods but either derive sub-optimal regrets or require particular assumptions. The gap is later closed by He et al. [18], which achieves the minimax lower bound using a novel sample-dependent weighting technique. Specifically, the weight for each sample is adaptive to its confidence, which is also called uncertainty. Beyond bandits, earlier works on MDPs [5,20,23,29,33,37,38] consider the setting where only the rewards are corrupted, and the transitions remain intact. Wu et al. [50] begin to handle corruption on both rewards and transitions for tabular MDPs. Wei et al. [48] establish a unified framework for RL with unknown corruption under a weak adversary, where the corruption happens before the decision is made in each round. Later, Ye et al. [55] extend the weighting technique [18] to corrupted RL with general function approximation and achieve a linear dependence on the cumulative corruption level  $\zeta$ . Particularly, Wei et al. [48], Ye et al. [55] both impose corruption on the Bellman operator, which is the same as the corruption model considered in this paper.
Offline RL Against Attacks. The emergence of poisoning attacks in real-world scenarios poses new challenges for offline RL and necessitates improved defenses [49]. There are generally two types of attacks [3], namely test-time attacks and training-time attacks. In test-time attacks, the training data is clean, and the learned policy must contend with an attacker during test time. For example, Yang et al. [53] propose learning conservative and smooth policies robust to different test-time attacks. In contrast, our paper focuses on the training-time attack as another line of work [31, 49, 60], where part of the training data is corrupted maliciously. Wu et al. [49] propose two certification criteria and a new aggregation-based method to improve the learned policy from corrupted data. To the best of our knowledge, [60] is the only theoretical work on corrupted offline RL, which considers that an  $\epsilon$ -fraction  $(\epsilon = \zeta / nH)$  of samples are corrupted on both rewards and transitions for linear MDPs and achieves an  $\mathcal{O}(\zeta^{1/2} dn^{-1/2})$  suboptimality bound. Notably, distinct from the setting in Zhang et al. [60] that clean data is first collected and then corrupted by an adversary, we consider the setting that data collection and corruption occur at the same time (thus corruption at one step affects the subsequent trajectory). Therefore, our setting is different from that of Zhang et al. [60].
In this section, we formulate the episodic Markov decision process (MDP) with adversarial corruption and under general (nonlinear) function approximation. Before the formal introduction, we introduce some notations to facilitate our presentation.
Notations. Let  $[n]$  denote the set  $\{1, \ldots, n\}$ . For spaces  $\mathcal{X}$  and  $\mathcal{A}$  and a function  $f: \mathcal{X} \times \mathcal{A} \to \mathbb{R}$ , let  $f(x) = \max_{a \in \mathcal{A}} f(x, a)$ . Given a semi-definite matrix  $M$  and a vector  $v$ , we define  $\|v\|_M = \sqrt{v^\top M v}$ . For two positive sequences  $\{f(n)\}_{n=1}^{\infty}$ ,  $\{g(n)\}_{n=1}^{\infty}$ , let  $f(n) = \mathcal{O}(g(n))$  if there exists a constant  $C > 0$  such that  $f(n) \leq C g(n)$  for all  $n \geq 1$ , and  $f(n) = \Omega(g(n))$  if there exists a constant  $C > 0$  such that  $f(n) \geq C g(n)$  for all  $n \geq 1$ . We use  $\tilde{\mathcal{O}}(\cdot)$  to omit polylogarithmic factors. Sometimes we use the shorthand notation  $z = (x, a)$ .
2.1 Episodic MDPs
We consider an episodic MDP  $(\mathcal{X},\mathcal{A},H,\mathbb{P},r)$  with the state space  $\mathcal{X}$ , action space  $\mathcal{A}$ , episode length  $H$ , transition kernel  $\mathbb{P} = \{\mathbb{P}^h\}_{h\in [H]}$ , and reward function  $r = \{r^h\}_{h\in [H]}$ . Suppose that the rewards are bounded:  $r^h\geq 0$  for any  $h\in [H]$ , and  $\sum_{h = 1}^{H}r^{h}(x^{h},a^{h})\leq 1$  almost surely. Given any policy  $\pi = \{\pi^h:\mathcal{X}\to \mathcal{A}\}_{h\in [H]}$ , we define the Q-value and V-value functions starting from step  $h$  as
$$
Q _ {\pi} ^ {h} \left(x ^ {h}, a ^ {h}\right) = \sum_ {h ^ {\prime} = h} ^ {H} \mathbb {E} _ {\pi} \left[ r ^ {h ^ {\prime}} \left(x ^ {h ^ {\prime}}, a ^ {h ^ {\prime}}\right) \mid x ^ {h}, a ^ {h} \right], \quad V _ {\pi} ^ {h} \left(x ^ {h}\right) = \sum_ {h ^ {\prime} = h} ^ {H} \mathbb {E} _ {\pi} \left[ r ^ {h ^ {\prime}} \left(x ^ {h ^ {\prime}}, a ^ {h ^ {\prime}}\right) \mid x ^ {h} \right]. \tag {1}
$$
where the expectation  $\mathbb{E}_{\pi}$  is taken with respect to the trajectory under the policy  $\pi$ . There exists an optimal policy  $\pi_*$  and optimal value functions  $V_{*}^{h}(x)\coloneqq V_{\pi^{*}}^{h}(x) = \sup_{\pi}V_{\pi}^{h}(x)$  and  $Q_{*}^{h}(x,a)\coloneqq Q_{\pi^{*}}^{h}(x,a) = \sup_{\pi}Q_{\pi}^{h}(x,a)$  that satisfy the Bellman optimality equation:
$$
Q _ {*} ^ {h} (x, a) = \mathbb {E} _ {r ^ {h}, x ^ {h + 1}} \left[ r ^ {h} (s, a) + \max  _ {a ^ {\prime} \in \mathcal {A}} Q _ {*} ^ {h + 1} \left(x ^ {h + 1}, a ^ {\prime}\right) \mid x, a \right] := \left(\mathcal {T} ^ {h} Q _ {*} ^ {h + 1}\right) (x, a), \tag {2}
$$
where  $\mathcal{T}^h$  is called the Bellman operator. Then we define the Bellman residual as
$$
\mathcal {E} ^ {h} \left(f, x ^ {h}, a ^ {h}\right) = f ^ {h} \left(x ^ {h}, a ^ {h}\right) - \left(\mathcal {T} ^ {h} f ^ {h + 1}\right) \left(x ^ {h}, a ^ {h}\right). \tag {3}
$$
2.2 General Function Approximation
We approximate the Q-value functions by a function class  $\mathcal{F} = \mathcal{F}_1 \times \dots \times \mathcal{F}_H$  where  $\mathcal{F}_h: \mathcal{X} \times \mathcal{A} \to [0, 1]$  for  $h \in [H]$ , and  $f_{H+1} \equiv 0$  since no reward is generated at step  $H+1$ . Generally, the following assumption is common for the approximation function class.
Assumption 2.1 (Realizability and Completeness). For all  $h \in [H]$ ,  $Q_{*}^{h} \in \mathcal{F}^{h}$ . Additionally, for all  $g^{h+1}(x^{h+1}) \in [0,1]$ ,  $(\mathcal{T}^{h}g^{h+1})(x^{h},a^{h}) \in \mathcal{F}^{h}$ .
The realizability assumption [21] ensures the possibility of learning the true Q-value function by considering the function class  $\mathcal{F}$ . The Bellman completeness (adopted from Wang et al. [47] and Assumption 18.22 of Zhang [59]) is stronger than that in Jin et al. [21]. The former applies the least squares value iteration (LSVI) algorithm that establishes optimism at each step, while the latter proposes the GOLF algorithm that only establishes optimism at the first step. We use the standard covering number to depict the scale of the function class  $\mathcal{F}$ .
Definition 2.1 ( $\epsilon$ -Covering Number). The  $\epsilon$ -covering number  $N(\epsilon, \mathcal{F}, \rho)$  of a set  $\mathcal{F}$  under metric  $\rho$  is the smallest cardinality of a subset  $\mathcal{F}_0 \subseteq \mathcal{F}$  such that for any  $f \in \mathcal{F}$ , there exists a  $g \in \mathcal{F}_0$  satisfying that  $\rho(f, g) \leq \epsilon$ . We say  $\mathcal{F}_0$  is an  $(\epsilon, \rho)$  cover of  $\mathcal{F}$ .
2.3 Offline Data Collection Process
Offline Clean Data. Consider an offline clean dataset with  $n$  trajectories  $\mathcal{D} = \{(x_i^h, a_i^h, r_i^h)\}_{i,h=1}^{n,H}$ . We assume the dataset  $\mathcal{D}$  is compliant with an MDP  $(\mathcal{X}, \mathcal{A}, H, \mathbb{P}, r)$  with the value functions  $Q, V$  and the Bellman operator  $\mathcal{T}$ : for any policy  $\pi$ ,
$$
\begin{array}{l} \mathbb {P} \big ((\mathcal {T} ^ {h} Q _ {\pi} ^ {h + 1}) (x _ {i} ^ {h}, a _ {i} ^ {h}) = r ^ {\prime} + V _ {\pi} ^ {h + 1} (x ^ {\prime}) \big | \{(x _ {j} ^ {h}, a _ {j} ^ {h}) \} _ {j \in [ i ]}, \{r _ {j} ^ {h}, x _ {j} ^ {h + 1}) \} _ {j \in [ i - 1 ]}, Q _ {\pi} ^ {h + 1} \big) \\ = \mathbb {P} \left(r ^ {h} \left(x ^ {h}, a ^ {h}\right) = r ^ {\prime}, x ^ {h + 1} = x ^ {\prime} \mid x ^ {h} = x _ {i} ^ {h}, a ^ {h} = a _ {i} ^ {h}\right), \tag {4} \\ \end{array}
$$
where the realizability and completeness in Assumption 2.1 hold. The compliance assumption (4) is also made in Jin et al. [20], Zhong et al. [63], which means that  $\mathcal{D}$  remains the Markov property and allows  $\mathcal{D}$  to be collected by an adaptive behavior policy. The induced distribution of the state-action pair is denoted by  $\mu = \{\mu^h\}_{h\in [H]}$ .
Adversarial Corruption. During the offline dataset collection process, after observing the state-action pair  $(x^{h},a^{h})$  chosen by the data collector, an adversary corrupts  $r^h$  and  $x^{h + 1}$  at each step  $h$  before they are revealed to the collector. For the corrupted dataset  $\mathcal{D}$ , we define the corrupted value function  $Q_{\mathcal{D}}$ ,  $V_{\mathcal{D}}$ , and the Bellman operator  $\mathcal{T}_{\mathcal{D}}$  satisfying (4). To measure the corruption level, we notice that characterizing the specific modification on each tuple  $(s,a,s',r)$  is hard and unnecessary since once one modifies a tuple, the subsequent trajectory changes. Therefore, it is difficult to tell whether the change is caused by the corruption at the current step or a previous step. In fact, we only care about the part of the change that violates the Bellman completeness. Therefore, following the online setting [55, 48], we measure the corruption level by the gap between  $\mathcal{T}_{\mathcal{D}}$  and  $\mathcal{T}$  as follows.
Definition 2.2 (Cumulative Corruption). The cumulative corruption is  $\zeta$  if at any step  $h\in [H]$ , for a sequence  $\{(x_i^h,a_i^h)\}_{i,h = 1}^{n,H}\subset \mathcal{X}\times \mathcal{A}$  chosen by the data collector and a sequence of functions  $\{g^h:\mathcal{X}\to [0,1]\}_{h = 1}^H$ , we have for all  $h\in [H]$
$$
\sum_ {i = 1} ^ {n} \left| \zeta_ {i} ^ {h} \right| \leq \zeta^ {h}, \quad \sum_ {h = 1} ^ {H} \zeta^ {h} := \zeta ,
$$
where  $\zeta_i^h = (\mathcal{T}^h g^{h + 1} - \mathcal{T}_{\mathcal{D}}^h g^{h + 1})(x_i^h,a_i^h)$
The learning objective is to find a policy  $\pi$  that minimizes the suboptimality of  $\pi$  given any initial state  $x^{1} = x$ :  $\mathrm{SubOpt}(\pi, x) = V_{*}^{1}(x) - V_{\pi}^{1}(x)$ , where  $V(\cdot)$  is the value function induced by the uncorrupted MDP.
In this section, we first highlight the pivotal role that uncertainty weighting plays in controlling the corruption-related bound. To extend the uncertainty weighting technique to the offline setting, we propose an iteration algorithm. With the proposed algorithm, the theoretical result for the suboptimality is presented.
3.1 Uncertainty-Related Weights
In this subsection, we discuss the choice of weight for a simplified model without state transition  $(H = 1)$  and use the notation  $z_{i} = (x_{i},a_{i})$ . Given a dataset  $\{(z_i,y_i)\}_{i\in [n]}$ , we have  $y_{i} = \bar{f} (z_{i}) + \zeta_{i} + \epsilon_{i}$  for  $i\in [n]$ , where  $\bar{f}\in \mathcal{F}$  is the uncorrupted true value, the noise  $\epsilon_{i}$  is zero-mean and conditional  $\eta$ -subGaussian, and the corruption level is  $\zeta = \sum_{i = 1}^{n}|\zeta_{i}|$ .
We begin with delineating the consequence caused by the adversarial corruption for the traditional least-square regression:  $\hat{f} = \min_{f\in \mathcal{F}}\sum_{i = 1}^{n}\left(f(z_i) - y_i\right)^2$ . Some calculations lead to the following decomposition:
$$
\sum_{i = 1}^{n}(\hat{f} (z_{i}) - \bar{f} (z_{i}))^{2} = \underbrace{\sum_{i = 1}^{n}\left[(\hat{f} (z_{i}) - y_{i})^{2} - (\bar{f} (z_{i}) - y_{i})^{2}\right]}_{I_{1}\leq 0} + 2\underbrace{\sum_{i = 1}^{n}(\hat{f} (z_{i}) - \bar{f} (z_{i}))\epsilon_{i}}_{I_{2}:\text{Noise term}} + 2\underbrace{\sum_{i = 1}^{n}(\hat{f} (z_{i}) - \bar{f} (z_{i}))\zeta_{i}}_{I_{3}:\text{Corruption term}}.
$$
The term  $I_1 \leq 0$  since  $\hat{f}$  is the solution to the least-square regression. The term  $I_2$  is bounded by  $\tilde{\mathcal{O}}(\ln N)$  because of the  $\eta$ -subGaussainity of  $\epsilon_i$ , where  $N$  is the covering number of  $\mathcal{F}$ . The term  $I_3$  is ruined by corruption:  $I_3 \leq 2\sum_{i=1}^n |\zeta_i| = \mathcal{O}(\zeta)$ . Hence, the confidence radius  $(\sum_{i=1}^n (\hat{f}(z_i) - \bar{f}(z_i))^2)^{1/2} = \tilde{\mathcal{O}}(\sqrt{\zeta + \ln N})$  will explode whenever the corruption level  $\zeta$  grows with  $n$ .
To control the corruption term, motivated by the uncertainty-weighting technique from online settings [55, 18, 65], we apply the weighted regression:  $\hat{f} = \min_{f\in \mathcal{F}}\sum_{i = 1}^{n}\left(f(z_i) - y_i\right)^2 /\sigma_i^2$  , where ideally, we desire the following uncertainty-related weights:
$$
\sigma_ {i} ^ {2} = \max  \left(1, \frac {1}{\alpha} \underbrace {\sup  _ {f , f ^ {\prime} \in \mathcal {F}} \frac {| f (z _ {i}) - f ^ {\prime} (z _ {i}) |}{\sqrt {\lambda + \sum_ {j = 1} ^ {n} (f (z _ {j}) - f ^ {\prime} (z _ {j})) ^ {2} / \sigma_ {j} ^ {2}}}} _ {\text {U n c e r t a i n t y}}\right), \quad i = 1, \dots , n, \tag {5}
$$
where  $\alpha, \lambda > 0$  are pre-determined parameters. The uncertainty quantity in the above equation is the supremum of the ratio between the prediction error  $|f(z_i) - f'(z_i)|$  and the training error  $\sqrt{\sum_{j=1}^{n}(f(z_j) - f'(z_j))^2 / \sigma_j^2}$  over  $f, f' \in \mathcal{F}$ . Intuitively, the quantity depicts the relative information of a sample  $z_i$  against the whole training set  $\{z_1, \ldots, z_n\}$ . We can use the linear function class as a special example to explain it. When the function space  $\mathcal{F}^h$  is embedded into a  $d$ -dimensional vector space:  $\mathcal{F}^h = \{\langle w(f), \phi(\cdot) \rangle : z \to \mathcal{R}\}$ , the uncertainty quantity becomes
$$
\begin{array}{l} \sup  _ {f, f ^ {\prime} \in \mathcal {F}} \frac {| \langle w (f) - w \left(f ^ {\prime}\right) , \phi \left(z _ {i}\right) \rangle |}{\sqrt {\lambda + \sum_ {j = 1} ^ {n} \left(\langle w (f) - w \left(f ^ {\prime}\right) , \phi \left(z _ {j}\right) \rangle\right) ^ {2} / \sigma_ {j} ^ {2}}} \leq \sup  _ {f, f ^ {\prime} \in \mathcal {F}} \frac {| \langle w (f) - w \left(f ^ {\prime}\right) , \phi \left(z _ {i}\right) \rangle |}{\sqrt {\left(w (f) - w \left(f ^ {\prime}\right)\right) ^ {\top} \Lambda \left(w (f) - w \left(f ^ {\prime}\right)\right)}} \\ \leq \sqrt {\phi^ {\top} (z _ {i}) \Lambda^ {- 1} \phi (z _ {i})}, \\ \end{array}
$$
where  $\Lambda = \sum_{j=1}^{n} \phi(z_j) \phi^\top(z_j) / \sigma_j^2$ . Moreover,  $\left(\phi^\top(z_i) \Lambda^{-1} \phi(z_i)\right)^{-1}$  represents the effective number of samples in the  $\{z_i\}_{i=1}^n$  along the  $\phi(z_i)$ 's direction. We discuss in Lemma B.3 that under mild conditions the linear and nonlinear uncertainty quantities are almost equivalent.
However, since the uncertainty also depends on weights, it is impossible to determine all the weights  $\{\sigma_i\}_{i\in [n]}$  simultaneously. Compared with the online setting where the weight in each round can be determined sequentially (iteratively in rounds), we face two challenges in the offline setting: (a) how to compute uncertainty-related weights iteratively? (b) will an approximate solution to the uncertainty play an equivalent role in controlling the corruption term?
To solve the first challenge, we propose the weight iteration algorithm in Algorithm 1. Moreover, we demonstrate the convergence of this algorithm by the monotone convergence theorem in the following lemma, which ensures that the output weights are sufficiently close to desired ones (5). The proof is provided in Appendix B.1.
Lemma 3.1. There exists a  $T$  such that the output of Algorithm  $\boxed{I}\{\sigma_{i}:=\sigma_{i}^{T+1}\}_{i=1}^{n}$  satisfies:
$$
\sigma_ {i} ^ {2} \geq \max  (1, \psi (z _ {i}) / 2), \quad \sigma_ {i} ^ {2} \leq \max  (1, \psi (z _ {i})), \tag {6}
$$
where  $\psi (z_i) = \sup_{f,f'\in \mathcal{F}}\frac{|f(z_i) - f'(z_i)| / \alpha}{\sqrt{\lambda + \sum_{j = 1}^n(f(z_j) - f'(z_j))^2 / \sigma_j^2}}.$
Algorithm 1 Uncertainty Weight Iteration
1: Input:  $\{(x_{i},a_{i})\}_{i = 1}^{n},\mathcal{F},\alpha >0$
2: Initialization:  $t = 0$ ,  $\sigma_i^0 = 1$ ,  $i = 1, \dots, n$
3: repeat
4:  $t\gets t + 1$
5:  $(\sigma_i^t)^2\gets \max \Big(1,\sup_{f,f'\in \mathcal{F}}\frac{|f(x_i,a_i) - f'(x_i,a_i)| / \alpha}{\sqrt{\lambda + \sum_{j = 1}^n(f(x_j,a_j) - f'(x_j,a_j))^2 / (\sigma_j^{t - 1})^2}}\Big),i = 1,\ldots ,n$
6: until  $\max_{i\in [n]}\left(\sigma_i^t /\sigma_i^{t - 1}\right)^2\leq 2$
7: Output:  $\{\sigma_i^t\}_{i = 1}^n$