SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For the second challenge, the weighted version  $L_{n} \coloneqq \sum_{i=1}^{n} (\hat{f}(z_{i}) - \bar{f}(z_{i}))^{2} / \sigma_{i}^{2}$  can also be decomposed into three terms correspondingly. We can demonstrate that an approximate choice of weights satisfying (6) is sufficient to control the corruption term as
$$
\sum_ {i = 1} ^ {n} \frac {(\hat {f} (z _ {i}) - \bar {f} (z _ {i})) \zeta_ {i}}{\sigma_ {i} ^ {2}} = \sum_ {i = 1} ^ {n} \frac {| \hat {f} (z _ {i}) - \bar {f} (z _ {i}) | \zeta_ {i} \cdot \sqrt {\lambda + L _ {n}}}{\sigma_ {i} ^ {2} \sqrt {\lambda + \sum_ {j = 1} ^ {n} (\hat {f} (z _ {j}) - \bar {f} (z _ {j})) ^ {2} / \sigma_ {j} ^ {2}}} \leq 2 \alpha \zeta \sqrt {\lambda + L _ {n}}.
$$
Since the corruption-unrelated terms (corresponding to  $I_1, I_2$ ) can still be bounded by  $\tilde{\mathcal{O}}(\ln N)$ , we have  $L_n = \tilde{\mathcal{O}}(\ln N + 2\alpha \zeta \sqrt{L_n})$ , leading to an  $\tilde{\mathcal{O}}(\alpha \zeta + \sqrt{\ln N})$  confidence radius. Therefore, with a sufficiently small  $\alpha$ , the effect of corruption can be countered.
3.2 Corruption-Robust Algorithm
Algorithm 2 CR-PEVI
1: Input:  $\mathcal{D} = \{(x_i^h, a_i^h, r_i^h)\}_{i,h=1}^{n,H}$ ,
2: Initialization: Set  $f_{n}^{H + 1}(\cdot) \gets 0$
3: for step  $h = H, H - 1, \ldots, 1$  do
4: Choosing weights  $\{\sigma_i^h\}_{i = 1}^n$  by proceeding Algorithm  $\sqcup$  with inputs  $\{(x_i^h,a_i^h)\}_{i = 1}^n,\mathcal{F}^h,\alpha$
5: Find the weighted least-squares solution in (7)
6: Find  $\beta^n$  and construct confidence set
$$
\widehat {\mathcal {F}} ^ {h} = \left\{f \in \mathcal {F} ^ {h}: \lambda + \sum_ {i = 1} ^ {n} \left(f \left(x _ {i} ^ {h}, a _ {i} ^ {h}\right) - \hat {f} ^ {h} \left(x _ {i} ^ {h}, a _ {i} ^ {h}\right)\right) ^ {2} / \left(\sigma_ {i} ^ {h}\right) ^ {2} \leq \left(\beta^ {h}\right) ^ {2} \right\}
$$
7: Construct bonus function as (8)
8: Let  $f_{n}^{h}(\cdot ,\cdot) = \max \left(0,\hat{f}^{h}(\cdot ,\cdot) - \beta^{h}b^{h}(\cdot ,\cdot)\right)$
9: Set  $\hat{\pi}^h (\cdot) = \operatorname{argmax}_{a\in \mathcal{A}}f_n^h (\cdot ,a)$
10: end for
11: Output:  $\{\hat{\pi}^h\}_{h = 1}^H$
Now, for the offline RL with general function approximation, we integrate the uncertainty weight iteration algorithm with the pessimistic value iteration (PEVI) algorithm [24], and propose a Corruption-Robust PEVI (CR-PEVI) in Algorithm 2. Our algorithm employs backward induction from step  $H$  to 1. Set estimated value function  $f_{n}^{H + 1}(\cdot) = 0$ . At each step  $h \in [H]$ , having obtained  $f_{n}^{h + 1}$ , we calculate  $f_{n}^{h}$  by solving the following weighted least-square regression:
$$
\hat {f} ^ {h} = \underset {f ^ {h} \in \mathcal {F} ^ {h}} {\operatorname {a r g m i n}} \sum_ {i = 1} ^ {n} \frac {\left(f \left(x _ {i} ^ {h} , a _ {i} ^ {h}\right) - r _ {i} ^ {h} - f _ {n} ^ {h + 1} \left(x _ {i} ^ {h + 1}\right)\right) ^ {2}}{\left(\sigma_ {i} ^ {h}\right) ^ {2}}, \tag {7}
$$
where the weights are obtained via Algorithm [1]. As opposed to online RL where the necessity of exploration stimulates optimistic estimation, the literature on offline RL [24, 63] is more inclined to pessimism due to the limitation of offline data coverage. Hence, we construct a confidence set  $\widehat{\mathcal{F}}^h = \{f\in \widehat{\mathcal{F}}^h:\lambda +\sum_{i = 1}^n (f(x_i^h,a_i^h) - \hat{f}^h (x_i^h,a_i^h))^2 /(\sigma_i^h)^2\leq (\beta^h)^2\}$  such that the uncorrupted Bellman operator  $\mathcal{T}^h$  converts the value function  $f_{n}^{h + 1}$  into the function class  $\widehat{\mathcal{F}}^h$  (i.e.,  $\mathcal{T}^h f_n^{h + 1}\in$
$\widehat{\mathcal{F}}^h$  ) with high probability. For the bonus function, we follow [55] and choose it as
$$
b _ {h} (x, a) = \sup  _ {f, f ^ {\prime} \in \widehat {\mathcal {F}} ^ {h}} \frac {| f (x , a) - f ^ {\prime} (x , a) |}{\sqrt {\lambda + \sum_ {i = 1} ^ {n} \left(f \left(x _ {i} ^ {h} , a _ {i} ^ {h}\right) - f ^ {\prime} \left(x _ {i} ^ {h} , a _ {i} ^ {h}\right)\right) ^ {2} / \left(\sigma_ {i} ^ {h}\right) ^ {2}}}, \tag {8}
$$
which is seldom used in practical algorithms due to its unstability. Specifically, the covering number of the space containing (8) may be uncontrollable. According to Appendix E in [55], the issue of the covering number can be addressed under mild conditions by some techniques. Therefore, to maintain readability and consistency in this paper, we assume the corresponding bonus function space  $\mathcal{B}^{h+1}$  of (8) has a bounded covering number. Then we introduce pessimism by subtracting  $b^h$  from the estimated value function:  $f_n^h(x, a) = \max(0, \hat{f}^h(x, a) - \beta^h b^h(x, a))$ .
4.1 Coverage Condition
No guarantee for the suboptimality can be provided with insufficient data coverage. Based on pessimism, Jin et al. [20], Rashidinejad et al. [36] have demonstrated that the coverage over the optimal policy is sufficient for sample-efficient offline RL. The following condition covers the optimal policy under general function approximation.
Definition 4.1 (Coverage Coefficient). Consider the offline dataset  $\{x_i^h, a_i^h\}_{i,h=1}^{n,H}$ . For any initial state  $x^1 = x \in \mathcal{X}$ , the coverage coefficient is:
$$
\operatorname {C C} \left(\lambda , \widehat {\mathcal {F}}, \mathcal {Z} _ {n} ^ {H}\right) = \max  _ {h \in [ H ]} \mathbb {E} _ {\pi_ {*}} \left[ \sup  _ {f, f ^ {\prime} \in \widehat {\mathcal {F}} ^ {h}} \frac {n \left(f \left(x ^ {h} , a ^ {h}\right) - f ^ {\prime} \left(x ^ {h} , a ^ {h}\right)\right) ^ {2}}{\lambda + \sum_ {i = 1} ^ {n} \left(f \left(x _ {i} ^ {h} , a _ {i} ^ {h}\right) - f ^ {\prime} \left(x _ {i} ^ {h} , a _ {i} ^ {h}\right)\right) ^ {2}} \mid x ^ {1} = x \right], \tag {9}
$$
where  $\mathbb{E}_{\pi_*}$  is taken with respect to the trajectory induced by  $\pi_*$  in the underlying uncorrupted MDP.
In the face of corruption, we require single-policy coverage over the uncorrupted trajectory. This coefficient depicts the expected uncertainty of sample  $(x^{h},a^{h})$  induced by the optimal policy  $\pi_{*}$  compared to the  $n$  training samples. We use the linear MDP to interpret this condition, where the function space  $\mathcal{F}^h$  is embedded into a  $d$ -dimensional vector space:  $\mathcal{F}^h = \{\langle w(f),\phi (\cdot)\rangle :z\to \mathbb{R}\}$ , where  $z$  denotes the state-action pair  $(x,a)$ . With the notation  $\Lambda^h = \lambda I + \sum_{i = 1}^n\phi (z_i^h)\phi (z_i^h)^\top$ , we can demonstrate that if the sufficient "coverage" in Jin et al. [24] holds: there exists a constant  $c^{\dagger}$  such that  $\Lambda^h\succeq I + c^\dagger n\mathbb{E}_{\pi_*}[\phi (z^h)\phi (z^h)^\top |x^1 = x]$  for all  $h\in [H]$ , our coverage coefficient is bounded:  $\mathrm{CC}(\lambda ,\widehat{\mathcal{F}},\mathcal{Z}_n^H)\leq d / c^\dagger < \infty$ . Therefore, the sufficient "coverage" condition implies our coverage condition in the linear setting. We will discuss the connection formally in Lemma B.1
4.2 Main Result
Before presenting the main theorem, we introduce a new general version of the well-explored dataset condition, which is the key to eliminating the uncertainty-related weights from the suboptimality bound and deriving the final result.
Assumption 4.1 (Well-Explied Dataset). There exists a constant  $C(\widehat{\mathcal{F}}, \mu) > 8n^{-1}$  such that for a  $(n^{-1}, \| \cdot \|_{\infty})$  cover of  $\widehat{\mathcal{F}}^h$ , denoted by  $\bar{\mathcal{F}}^h$ , and for any two distinct  $f, f' \in \bar{\mathcal{F}}^h$ ,
$$
\min  _ {h \in [ H ]} \sum_ {z ^ {h} \in \mathcal {X} \times \mathcal {A}} \mu^ {h} \left(z ^ {h}\right) \left(f \left(z ^ {h}\right) - f ^ {\prime} \left(z ^ {h}\right)\right) ^ {2} \geq C (\widehat {\mathcal {F}}, \mu). \tag {10}
$$
We interpret this condition with the linear model, where the condition (10) becomes: for any two distinct  $f, f' \in \bar{\mathcal{F}}^h$ ,  $\min_{h \in [H]} \left( w(f) - w(f') \right)^{\top} \bar{\Lambda}^h \left( w(f) - w(f') \right) \geq C(\widehat{\mathcal{F}}, \mu)$ . As proved in Lemma B.2 with high probability, the above condition holds with  $C(\widehat{\mathcal{F}}, \mu) = \Theta(d^{-1})$  when the  $n$  trajectories of  $\mathcal{D}$  are independent, and the data distribution  $\mu^h$  satisfies the minimum eigenvalue condition:
$$
\sigma_ {\min } \left(\mathbb {E} _ {z ^ {h} \sim \mu^ {h}} \left[ \phi \left(z ^ {h}\right) \phi \left(z ^ {h}\right) ^ {\top} \right]\right) = \bar {c} / d, \tag {11}
$$
where  $\bar{c} > 0$  is an absolute constant. This is a widely-adopted assumption in the literature [11,46,63]. Note that  $\Theta(d^{-1})$  is the largest possible minimum eigenvalue since for any data distribution  $\tilde{\mu}^h$ ,  $\sigma_{\min}(\mathbb{E}_{z^h \sim \mu^h}[\phi(z^h)\phi(z^h)^\top]) \leq d^{-1}$  by using  $\| \phi(z^h) \| \leq 1$  for any  $z^h \in \mathcal{X} \times \mathcal{A}$ .
Theorem 1. If the coverage coefficient in Definition 4.1 is finite, under Assumption 4.1 for corruption  $\zeta = \sum_{h=1}^{H} \zeta^h$  and  $\delta > 0$ , we choose the covering parameter  $\gamma = 1 / (n \max_h \beta^h \zeta^h)$ ,  $\lambda = \ln(N_n(\gamma))$ , the weighting parameter  $\alpha = H \sqrt{\ln N_n(\gamma)} / \zeta$ , and the confidence radius
$$
\beta^ {h} = c _ {\beta} \big (\alpha \zeta^ {h} + \sqrt {\ln (H N _ {n} (\gamma) / \delta)} \big), \quad f o r h = H, \ldots , 1,
$$
where  $N_{n}(\gamma) = \max_{h} N(\gamma / n, \mathcal{F}^{h}) \cdot N(\gamma / n, \mathcal{F}^{h + 1}) \cdot N(\gamma / n, \mathcal{B}^{h + 1}(\lambda))$ . Then, with probability at least  $1 - 2\delta$ , the sub-optimality of Algorithm  $\boxed{2}$  is bounded by
$$
\mathrm {S u b O p t} (\hat {\pi}, x) = \tilde {\mathcal {O}} \bigg (\frac {H (\mathrm {C C} (\lambda , \widehat {\mathcal {F}} , \mathcal {Z} _ {n} ^ {H})) ^ {1 / 4} \cdot (\ln N _ {n} (\gamma)) ^ {1 / 2}}{n ^ {1 / 2} (C (\widehat {\mathcal {F}}, \mu)) ^ {1 / 4}} + \frac {\zeta (\mathrm {C C} (\lambda , \widehat {\mathcal {F}} , \mathcal {Z} _ {n} ^ {H})) ^ {1 / 2}}{n (C (\widehat {\mathcal {F}}, \mu)) ^ {1 / 2}} \bigg).
$$
When  $\zeta = \mathcal{O}(\sqrt{n})$ , our algorithm achieves the same order of suboptimality as the uncorrupted case. Whenever  $\zeta = o(n)$ , our algorithm is sample-efficient. Moreover, when specialized to the linear MDP with dimension  $d$ , where  $\mathrm{CC}(\lambda, \widehat{\mathcal{F}}, \mathcal{Z}_n^H) = \mathcal{O}(d/c^\dagger)$ ,  $C(\widehat{\mathcal{F}}, \mu) = \Theta(d^{-1})$  and  $\ln N_n(\gamma) = \tilde{\mathcal{O}}(d^2)$ , the suboptimality bound in Theorem 1 becomes  $\tilde{\mathcal{O}}(Hd^{3/2}n^{-1/2} + d\zeta n^{-1})$ . The corruption-independent term  $\tilde{\mathcal{O}}(Hd^{3/2}n^{-1/2})$  matches that of PEVI [24]. The corruption-dependent term nearly matches the lower bound, as will be discussed later.
Remark 4.1. Although the theory requires a known corruption level  $\zeta$ , in the experiments, we treat the uncertainty ratio  $\alpha = O(1 / \zeta)$  as a tuning hyperparameter. The use of independent and identically distributed (i.i.d.) trajectories in our experiments renders the hyperparameter tuning process straightforward and conducive to optimizing the performance. Additionally, we can offer a choice of  $\alpha = \Theta(1 / \sqrt{n})$ . This choice finds support in the online setting [55, 18], where this specific choice of  $\alpha$  ensures that suboptimality remains in the order of uncorrupted error bound, even when  $\zeta = O(\sqrt{n})$ .
Proof sketch. The detailed proof of Theorem  $\boxed{1}$  is provided in Appendix A. Here we present a brief proof sketch for the suboptimality bound, which is accomplished by three steps: (1) by Lemma A.1 if the uncorrupted Bellman backup  $\mathcal{T}^h f_n\in \widehat{\mathcal{F}}^h$  for each  $h\in [H]$ , we can bound the suboptimality by the sum of the bonus  $\sum_{h = 1}^{H}\beta^{h}\mathbb{E}_{\pi_{*}}[b^{h}(x^{h},a^{h})|x^{1} = x]$ ; (2) by Lemma A.2, we demonstrate that an approximate uncertainty weight satisfying (6) is the key to bound the weighted Bellman error  $(\sum_{i = 1}^{n}((\hat{f}^{h} - (\mathcal{T}^{h}f_{n}^{h + 1}))(x_{i}^{h},a_{i}^{h}))^{2} / (\sigma_{i}^{h})^{2})^{1 / 2}$  by  $\beta^{h} = c_{\beta}(\alpha \zeta^{h} + \sqrt{\ln(HN_{n}(\gamma) / \delta)})$ ; and (3) combining the results in the first two steps, we can obtain the suboptimality bounded by:
$\mathrm{SubOpt}(\hat{\pi},x) = \tilde{\mathcal{O}}\big(H(\mathrm{CC}^{\sigma}(\lambda ,\widehat{\mathcal{F}},\mathcal{Z}_n^H))^{1 / 2}\cdot (\ln N_n(\gamma))^{1 / 2}\cdot n^{-1 / 2} + \zeta \cdot \mathrm{CC}^{\sigma}(\lambda ,\widehat{\mathcal{F}},\mathcal{Z}_n^H)\cdot n^{-1}\big).$
Here  $\mathrm{CC}^{\sigma}(\lambda, \widehat{\mathcal{F}}, \mathcal{Z}_n^H)$  denotes the weighted coverage coefficient as follows
$$
\mathrm {C C} ^ {\sigma} (\lambda , \widehat {\mathcal {F}}, \mathcal {Z} _ {n} ^ {H}) = \max  _ {h \in [ H ]} \mathbb {E} _ {\pi_ {*}} \left[ \sup  _ {f, f ^ {\prime} \in \widehat {\mathcal {F}} ^ {h}} \frac {n \left(f \left(x ^ {h} , a ^ {h}\right) - f ^ {\prime} \left(x ^ {h} , a ^ {h}\right)\right) ^ {2} / \sigma^ {h} \left(x ^ {h} , a ^ {h}\right) ^ {2}}{\lambda + \sum_ {i = 1} ^ {n} \left(f \left(x _ {i} ^ {h} , a _ {i} ^ {h}\right) - f ^ {\prime} \left(x _ {i} ^ {h} , a _ {i} ^ {h}\right)\right) ^ {2} / \left(\sigma_ {i} ^ {h}\right) ^ {2}} \mid x ^ {1} = x \right], \tag {12}
$$
where the weight for the trajectory induced by the optimal policy  $\pi_{*}$  is
$$
\left(\sigma^ {h} \left(x ^ {h}, a ^ {h}\right)\right) ^ {2} = \max  \left(1, \sup  _ {f, f ^ {\prime} \in \widehat {\mathcal {F}} ^ {h}} \frac {\left| f \left(x ^ {h} , a ^ {h}\right) - f ^ {\prime} \left(x ^ {h} , a ^ {h}\right) \right| / \alpha}{\sqrt {\lambda + \sum_ {i = 1} ^ {n} \left(f \left(x _ {i} ^ {h} , a _ {i} ^ {h}\right) - f ^ {\prime} \left(x _ {i} ^ {h} , a _ {i} ^ {h}\right)\right) ^ {2} / \left(\sigma_ {i} ^ {h}\right) ^ {2}}}\right). \tag {13}
$$
To control the weighted coverage coefficient  $\mathrm{CC}^{\sigma}$  by its unweighted counterpart CC, which is a challenging task due to the intricate form of uncertainty-related weights, a novel technique has been employed. The main idea behind this technique is to transform  $\mathrm{CC}^{\sigma}$  into a modified version of CC multiplied by a term that can be bounded by leveraging a cover of the function class. The connection between CC and  $\mathrm{CC}^{\sigma}$  is shown in the following lemma.
Lemma 4.1. Under Assumption 4.1 and choose  $\beta^h = C_\beta \sqrt{\ln N}$  (where  $C_\beta > 0$  contains the logarithmic terms that are omitted) and  $\lambda$  given in Theorem 7 we have
$$
\mathrm {C C} ^ {\sigma} (\lambda , \widehat {\mathcal {F}}, \mathcal {Z} _ {n} ^ {H}) = \tilde {\mathcal {O}} \big ((\mathrm {C C} (\lambda , \widehat {\mathcal {F}}, \mathcal {Z} _ {n} ^ {H})) ^ {1 / 2} \cdot (C (\widehat {\mathcal {F}}, \mu)) ^ {- 1 / 2} \big).
$$
Intuitively, because the weights themselves are closely related to the uncertainty, the weighted coverage coefficient  $\mathrm{CC}^{\sigma}$  where the high-uncertain samples are down-weighted can be upper-bounded by the square root of the unweighted coefficient CC. See Appendix B.3 for details.
(b)
Figure 1: Performance on the Walker2d and the Halfcheetah tasks under (a) random reward, (b) random dynamics, (c) adversarial reward, and (d) adversarial dynamics attacks.
Lower Bound. We construct a lower bound for linear MDPs with adversarial corruption  $\zeta$  to show that the  $\mathcal{O}(d\zeta n^{-1})$  corruption term of our suboptimality is optimal. The construction of the lower bound is adapted from Zhang et al. [60], where an  $\epsilon$ -constant fraction of the dataset is contaminated. We present the proof of Theorem 2 in Appendix B.4.
Theorem 2 (Minimax Lower Bound for Linear MDPs). Under linear MDPs with corruption (Definition [2.2]), for any fixed data-collecting distribution  $\nu$ , any algorithm  $L: \mathcal{D} \to \Pi$  with the knowledge of  $\zeta$  cannot find a better policy than  $\mathcal{O}(d\zeta n^{-1})$ -optimal policy with probability more than  $1/4$ :
$$
\min_{L,\nu}\max_{MDP,\mathcal{D}}\operatorname {SubOpt}(\hat{\pi}_{L}) = \Omega \big(d\zeta /n\big),
$$
where  $\mathcal{D}$  is the corrupted dataset initially generated from the MDP and then corrupted by an adversary, and  $\hat{\pi}_L$  is the policy generated by the algorithm  $L$ .
Based on our theoretical results, we propose a practical implementation for CR-PEVI and verify its effectiveness on simulation tasks with corrupted offline data.
Practical Implementation. To make our algorithm more practical, we use neural networks to estimate the  $Q$  function (i.e.,  $f$  in our theory) and the weight function  $\sigma$ . In linear MDPs, under a sufficiently broad function approximation class  $\widehat{\mathcal{F}}^h$  and sufficiently small parameter  $\lambda$  in Eq. (8), the bonus function can be simplified as the bootstrapped uncertainty of  $Q$  functions, which in turn can be estimated via the standard deviation of an ensemble of  $Q$  networks. We defer the detailed discussion to Appendix B.5. Following the state-of-the-art uncertainty-based offline RL algorithm Model Standard-deviation Gradients (MSG) [16], we learn a group of  $Q$  networks  $Q_{w_i}$ ,  $i = 1,\dots,K$  with independent targets and optimize a policy  $\pi_{\theta}$  with a lower-confidence bound (LCB) objective [16,2]. Specifically,  $Q_{w_i}$  is learned to minimize a weighted regression objective similar to Eq.(7). The weight function  $\sigma$  is estimated via bootstrapped uncertainty:  $\sigma(x,a) = \max(\sqrt{\mathbb{V}_{i=1,\dots,K}[Q_{w_i}(x,a)]}, 1)$ , where  $\mathbb{V}_{i=1,\dots,K}[Q_{w_i}]$  is the variance between the group of  $Q$  functions. This uncertainty estimation method has also been adopted by prior works [2, 16, 54]. We refer to our practical algorithm as Uncertainty Weighted MSG (UWMSG) and defer details to Appendix D.
Experimental Setup. We assess the performance of our approach using continuous control tasks from [15] and introduce both random and adversarial attacks on either the rewards or dynamics for the offline datasets. Details about the four types of data corruption and their cumulative corruption levels are deferred to Appendix D. The ensemble size  $K$  is set to 10 for all experiments. For evaluation, we report average returns with standard deviations over 10 random seeds. More implementation details are also provided in Appendix D.
Experimental Results. We compare UWMSG with the state-of-the-art uncertainty-base offline RL methods, MSG [16], EDAC [1], and SAC-N [1] under four types of data corruption. In particular, MSG can be considered as UWMSG with a constant weighting function  $\sigma(s, a) = 1$ . As demonstrated in Table [1] and Figure [1], our empirical results find that (1) current offline RL methods are susceptible to data corruption, e.g., MSG, EDAC, SAC-N achieve poor performance under adversarial attacks, and (2) our proposed UWMSG significantly improves performance under different data corruption scenarios, with an average improvement of  $104\%$  over MSG. More results can be found in Appendix F.
Table 1: Comparison of different offline RL algorithms under different environments and different data corruption types.
Environment	Attack Type	UWMSG	MSG	EDAC	SAC-N
Halfcheetah	Random Reward	7299.9 ± 169.0	4339.5 ± 3958.7	7128.2 ± 120.5	7357.5 ± 165.2
Random Dynamics	1425.0 ± 1659.5	212.9 ± 793.3	-12.3 ± 131.8	-66.2 ± 169.9
Adversarial Reward	1016.7 ± 503.9	243.1 ± 338.6	-127.2 ± 30.3	-55.7 ± 24.0
Adversarial Dynamics	4144.3 ± 1437.6	-87.3 ± 1055.6	374.0 ± 589.5	-246.8 ± 104.9
Walker2d	Random Reward	2189.7 ± 603.0	539.1 ± 534.6	-3.7 ± 1.2	-3.8 ± 1.4
Random Dynamics	2278.9 ± 706.4	2122.8 ± 821.5	-3.6 ± 0.8	-3.1 ± 0.7
Adversarial Reward	1433.4 ± 592.1	605.8 ± 310.7	61.0 ± 120.6	9.5 ± 21.6
Adversarial Dynamics	946.0 ± 300.4	506.0 ± 175.5	-4.8 ± 0.3	-5.1 ± 0.7
Hopper	Random Reward	2021.1 ± 888.6	1599.6 ± 814.5	107.7 ± 65.5	178.8 ± 114.0
Random Dynamics	2116.2 ± 618.6	1552.7 ± 532.8	5.9 ± 1.2	3.9 ± 2.2
Adversarial Reward	751.4 ± 72.5	651.0 ± 58.0	29.2 ± 18.7	111.8 ± 62.7
Adversarial Dynamics	931.2 ± 227.8	717.7 ± 204.3	5.9 ± 1.2	3.9 ± 2.2
Average	2212.8	1083.6	630.0	607.1
In summary, the experimental results validate the theoretical impact of data corruption for value-based offline RL algorithms. Our practical implementation algorithm demonstrates superior efficacy under different data corruption types, thereby highlighting its potential for real-world applications.
This work investigates the adversarially corrupted offline RL with general function approximation. We propose the uncertainty weight iteration and a weighted version of PEVI. Under a partial coverage condition and a well-explored dataset, our algorithm achieves a suboptimality bound, where the corruption-independent term recovers the uncorrupted bound, and the corruption-related term nearly matches the lower bound in linear models. Furthermore, our experiments demonstrate promising results showing that our practical implementation, UWMSG, significantly enhances the performance of the state-of-the-art offline RL algorithm under reward and dynamics data corruptions.
Our work suggests several potential future directions. First, it remains unsolved whether one can design robust algorithms to handle additional distribution shifts of the state-action pairs. Second, when the corruption level is unknown, how to design a theoretically robust offline RL algorithm and overcome the lack of adaptivity in the offline setting requires further research. Finally, we hope that our work sheds light on applying uncertainty weights to improve robustness in deep offline RL works and even practical applications.
We thank Wei Xiong for valuable discussions and feedback on an early draft of this work. We also thank the anonymous reviewers for their helpful comments.
[1] An, G., Moon, S., Kim, J.-H., and Song, H. O. (2021). Uncertainty-based offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436-7447.
[2] Bai, C., Wang, L., Yang, Z., Deng, Z., Garg, A., Liu, P., and Wang, Z. (2022). Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. arXiv preprint arXiv:2202.11566.
[3] Behzadan, V. and Munir, A. (2018). Mitigation of policy manipulation attacks on deep q-networks with parameter-space noise. In Computer Safety, Reliability, and Security: SAFECOMP 2018 Workshops, ASSURE, DECSO, SASSUR, STRIVE, and WAISE, Västerås, Sweden, September 18, 2018, Proceedings 37, pages 406-417. Springer.
[4] Bogunovic, I., Losalka, A., Krause, A., and Scarlett, J. (2021). Stochastic linear bandits robust to adversarial attacks. In International Conference on Artificial Intelligence and Statistics, pages 991-999. PMLR.
[60] Zhang, X., Chen, Y., Zhu, X., and Sun, W. (2022). Corruption-robust offline reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pages 5757-5773. PMLR.
[61] Zhang, X., Ma, Y., Singla, A., and Zhu, X. (2020b). Adaptive reward-poisoning attacks against reinforcement learning. In International Conference on Machine Learning, pages 11225-11234. PMLR.
[62] Zhao, H., Zhou, D., and Gu, Q. (2021). Linear contextual bandits with adversarial corruptions. arXiv preprint arXiv:2110.12615.
[63] Zhong, H., Xiong, W., Tan, J., Wang, L., Zhang, T., Wang, Z., and Yang, Z. (2022a). Pessimistic minimax value iteration: Provably efficient equilibrium learning from offline datasets. In International Conference on Machine Learning, pages 27117-27142. PMLR.
[64] Zhong, H., Xiong, W., Zheng, S., Wang, L., Wang, Z., Yang, Z., and Zhang, T. (2022b). A posterior sampling framework for interactive decision making. arXiv preprint arXiv:2211.01962.
[65] Zhou, D. and Gu, Q. (2022). Computationally efficient horizon-free reinforcement learning for linear mixture mdps. Advances in neural information processing systems, 35:36337-36349.