SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Another class of approaches employs an adversarial training framework, where offline RL is posed a two player game between an adversary that chooses the worst-case hypothesis (e.g., a value function or an MDP model) from a hypothesis class, and a policy player that tried to maximize the adversarially chosen hypothesis. Xie et al. (2021a) propose the concept of Bellman-consistent pessimism to constrain the class of value functions to be Bellman consistent on the data. Cheng et al. (2022) extend this framework by introducing a relative pessimism objective which allows for robust policy improvement over the data collection policy  $\mu$  for a wide range of hyper-parameters. Our approach can be interpreted as a model-based extension of Cheng et al. (2022). These approaches provide strong theoretical guarantees even with general function approximators while making minimal assumptions about the function class (realizability and Bellman completeness). Chen et al. (2022) provide an adversarial model learning method that uses an adversarial policy to generate a data-distribution where the model performs poorly and iteratively updating the model on the generated distribution. There also exist model-based approaches based on the same principle (Uehara and Sun, 2021; Rigter et al., 2022) for optimizing the absolute performance. Of these, Rigter et al. (2022) is the closest to our approach, as they also aim to find an adversarial MDP model that minimizes policy performance. They use a policy gradient approach to train the model, and demonstrate great empirical performance. However, their approach is based on absolute pessimism and does not enjoy the same RPI property as ARMOR.
D A Deeper Discussion of Robust Policy Improvement
D.1 How to formally define RPI?
Improving over some reference policy has been long studied in the literature. To highlight the advantage of ARMOR, we formally give the definition of different policy improvement properties.
Definition 2 (Robust policy improvement). Suppose  $\widehat{\pi}$  is the learned policy from an algorithm. We say the algorithm has the policy improvement (PI) guarantee if  $J(\pi_{\mathrm{ref}}) - J(\widehat{\pi}) \leq {}^{o(N)} / N$  is guaranteed for some reference policy  $\pi_{\mathrm{ref}}$  with offline data  $\mathcal{D} \sim \mu$ , where  $N = |\mathcal{D}|$ . We use the following two criteria w.r.t.  $\pi_{\mathrm{ref}}$  and  $\mu$  to define different kinds PI:
(i) The  $PI$  is **strong** if  $\pi_{\mathrm{ref}}$  can be selected arbitrarily from policy class  $\Pi$  regardless of the choice data-collection policy  $\mu$ ; otherwise,  $PI$  is **weak** (i.e.,  $\pi_{\mathrm{ref}} \equiv \mu$  is required).
(ii) The  $PI$  is robust if it can be achieved by a range of hyperparameters with a known subset.
Weak policy improvement is also known as safe policy improvement in the literature (Fujimoto et al., 2019; Laroche et al., 2019). It requires the reference policy to be also the behavior policy that collects the offline data. In comparison, strong policy improvement imposes a stricter requirement, which requires policy improvement regardless of how the data were collected. This condition is motivated by the common situation where the reference policy is not the data collection policy. Finally, since we are learning policies offline, without online interactions, it is not straightforward to tune the hyperparameter directly. Therefore, it is desirable that we can design algorithms with these properties in a robust manner in terms of hyperparameter selection. Formally, Definition 2 requires the policy improvement to be achievable by a set of hyperparameters that is known before learning.
Theorem 3 indicates the robust strong policy improvement of ARMOR. On the other hand, algorithms with robust weak policy improvement are available in the literature (Fujimoto et al., 2019; Kumar et al., 2019; Wu et al., 2019; Laroche et al., 2019; Fujimoto and Gu, 2021; Cheng et al., 2022); this is usually achieved by designing the algorithm to behave like IL for a known set of hyperparameter (e.g., behavior regularization algorithms have a weight that can turn off the RL behavior and regress to IL). However, deriving guarantees of achieving the best data-covered policy of the IL-like algorithm is challenging due to its imitating nature. To our best knowledge, ATAC (Cheng et al., 2022) is the only algorithm that achieves both robust (weak) policy improvement as well as guarantees absolute performance.
D.2 When RPI actually improves?
Given ARMOR's ability to improve over an arbitrary policy, the following questions naturally arise: Can ARMOR nontrivially improve the output policy of other algorithms (e.g., such as those based on absolute pessimism (Xie et al., 2021a)), including itself? Note that outputting  $\pi_{\mathrm{ref}}$  itself always satisfies RPI, but such result is trivial. By "nontrivially" we mean a non-zero worst-case improvement. If the statement were true, we would be able to repeatedly run ARMOR to improve over itself and then obtain the best policy any algorithm can learn offline.
Unfortunately, the answer is negative. Not only ARMOR cannot improve over itself, but it also cannot improve over a variety of algorithms. In fact, the optimal policy of an arbitrary model in the version space is unimprovable (see Corollary 10)! Our discussion reveals some interesting observations (e.g., how equivalent performance metrics for online RL can behave very differently in the offline setting) and their implications (e.g., how we should choose  $\pi_{\mathrm{ref}}$  for ARMOR). Despite their simplicity, we feel that many in the offline RL community are not actively aware of these facts (and the unawareness has led to some confusion), which we hope to clarify below.
Setup We consider an abstract setup where the learner is given a version space  $\mathcal{M}_{\alpha}$  that contains the true model and needs to choose a policy  $\pi \in \Pi$  based on  $\mathcal{M}_{\alpha}$ . We use the same notation  $\mathcal{M}_{\alpha}$  as before, but emphasize that it does not have to be constructed as in Eqs. (2) and (3). In fact, for the purpose of this discussion, the data distribution, sample size, data randomness, and estimation procedure for constructing  $\mathcal{M}_{\alpha}$  are all irrelevant, as our focus here is how decisions should be made with a given  $\mathcal{M}_{\alpha}$ . This makes our setup very generic and the conclusions widely applicable.
To facilitate discussion, we define the fixed point of ARMOR's relative pessimism step:
Definition 3. Consider Eq. (1) as an operator that maps an arbitrary policy  $\pi_{\mathrm{ref}}$  to  $\widehat{\pi}$ . A fixed point of this relative pessimism operator is, therefore, any policy  $\pi \in \Pi$  such that  $\pi \in \operatorname{argmax}_{\pi' \in \Pi} \min_{M \in \mathcal{M}_{\alpha}} J_M(\pi') - J_M(\pi)$ .
Given the definition, relative pessimism cannot improve over a policy if it is already a fixed point. Below we show a sufficient and necessary condition for being a fixed point, and show a number of concrete examples (some of which may be surprising) that are fixed points and thus unimprovable.
Lemma 9 (Fixed-point Lemma). For any  $\mathcal{M} \subseteq \mathcal{M}_{\alpha}$  and any  $\psi : \mathcal{M} \to \mathbb{R}$ , consider the policy
$$
\pi \in \operatorname {a r g m a x} _ {\pi^ {\prime} \in \Pi} \min  _ {M \in \mathcal {M}} J _ {M} \left(\pi^ {\prime}\right) + \psi (M) \tag {19}
$$
Then  $\pi$  is a fixed point in Definition 3. Conversely, for any fixed point  $\pi$  in Definition 3, there is a  $\psi : \mathcal{M} \to \mathbb{R}$  such that  $\pi$  is a solution to Eq. (19).
Corollary 10. The following are fixed points of relative pessimism (Definition 3):
1. Absolute-pessimism policy, i.e.,  $\psi (M) = 0$
2. Relative-pessimism policy for any reference policy, i.e.,  $\psi (M) = -J_{M}(\pi_{\mathrm{ref}})$
3. Regret-minimization policy, i.e.,  $\psi (M) = -J_M(\pi_M^\star)$ , where  $\pi_M^\star \in \mathrm{argmax}_{\pi \in \Pi}J_M(\pi)$ .
4. Optimal policy of an arbitrary model  $M \in \mathcal{M}_{\alpha}$ ,  $\pi_M^\star$ , i.e.,  $\mathcal{M} = \{M\}$ . This would include the optimistic policy, that is,  $\operatorname{argmax}_{\pi \in \Pi, M \in \mathcal{M}_\alpha} J_M(\pi)$
Return maximization and regret minimization are different in offline RL We first note that these four examples generally produce different policies, even though some of them optimize for
objectives that are traditionally viewed as equivalent in online RL (the "worst-case over  $\mathcal{M}_{\alpha}$ " part of the definition does not matter in online RL), e.g., absolute pessimism optimizes for  $J_{M}(\pi)$ , which is the same as minimizing the regret  $J_{M}(\pi_{M}^{\star}) - J_{M}(\pi)$  for a fixed  $M$ . However, their equivalence in online RL relies on the fact that online exploration can eventually resolve any model uncertainty when needed, so we only need to consider the performance metrics w.r.t. the true model  $M = M^{\star}$ . In offline RL with an arbitrary data distribution (since we do not make any coverage assumptions), there will generally be model uncertainty that cannot be resolved, and worst-case reasoning over such model uncertainty (i.e.,  $\mathcal{M}_{\alpha}$ ) separates apart the definitions that are once equivalent.
Moreover, it is impossible to compare return maximization and regret minimization and make a claim about which one is better. They are not simply an algorithm design choice, but are definitions of the learning goals and the guarantees themselves—thus incomparable: if we care about obtaining a guarantee for the worst-case return, the return maximization is optimal by definition; if we are more interested in obtaining a guarantee for the worst-case regret, then again, regret minimization is trivially optimal. We also note that analyzing algorithms under a metric that is different from the one they are designed for can lead to unusual conclusions. For example, Xiao et al. (2021) show that optimistic/neutral/pessimistic algorithms<sup>9</sup> are equally minimax-optimal in terms of their regret guarantees in offline multi-armed bandits. However, the algorithms they consider are optimistic/pessimistic w.r.t. the return—as commonly considered in the offline RL literature—not w.r.t. the regret which is the performance metric they are interested in analyzing.
$\pi_{\mathrm{ref}}$  is more than a hyperparameter—it defines the performance metric and learning goal. Corollary 10 shows that ARMOR (with relative pessimism) has many different fixed points, some of which may seem quite unreasonable for offline learning, such as greedy w.r.t. an arbitrary model or even optimism (#4). From the above discussion, we can see that this is not a defect of the algorithm. Rather, in the offline setting with unresolvable model uncertainty, there are many different performance metrics/learning goals that are generally incompatible/incomparable with each other, and the agent designer must make a choice among them and convey the choice to the algorithm. In ARMOR, such a choice is explicitly conveyed by the choice of  $\pi_{\mathrm{ref}}$ , which subsumes return maximization and regret minimization as special cases (#2 and #3 in Corollary 10)
E A More Comprehensive Toy Example for RPI
We illustrate with a simple toy example why ARMOR intuitively demonstrates the RPI property even when  $\pi_{\mathrm{ref}}$  is not covered by the data  $\mathcal{D}$ . ARMOR achieves this by 1) learning an MDP Model, and 2) adversarially training this MDP model to minimize the relative performance difference to  $\pi_{\mathrm{ref}}$  during policy optimization. Consider a one-dimensional discrete MDP with five possible states as shown in Figure 4. The dynamics is deterministic, and the agent always starts in the center cell. The agent receives a lower reward of 0.1 in the left-most state and a high reward of 1.0 upon visiting the right-most state. Say, the agent only has access to a dataset from a sub-optimal policy that always takes the left action to receive the 0.1 reward. Further, let's say we have access to a reference policy that demonstrates optimal behavior on the true MDP by always choosing the right action to visit the right-most state. However, it is unknown a priori that the reference policy is optimal. In such a case, typical offline RL methods can only recover the sub-optimal policy from the dataset as it is the best-covered policy in the data. Now, for the sake of clarity, consider the current learner policy is same as the behavior policy, i.e it always takes the left action.
ARMOR can learn to recover the expert reference policy in this example by performing rollouts with the adversarially trained MDP model. From the realizability assumption we know that the version space of models contains the true model (i.e.,  $M^{\star} \in \mathcal{M}_{\alpha}$ ). The adversary can then choose a model from this version space where the reference policy  $\pi_{\mathrm{ref}}$  maximally outperforms the learner. Note, that ARMOR does not require the true reward function to be known. In this toy example, the model selected by the adversary would be the one that not only allows the expert policy to reach the rightmost state, but also predicts the highest reward for doing so. Now, optimizing to maximize relative performance difference with respect to this model will ensure that the learner can recover the expert behavior, since the only way for the learner to stay competitive with the reference policy is to mimic the reference policy in the region outside data support. In other words, the reason why ARMOR has RPI to  $\pi_{\mathrm{ref}}$  is that its adversarial model training procedure can augment the original offline data with
9 Incidentally, optimistic/neutral policies correspond to #4 in Corollary 10.
True MDP
Dataset Transitions
Figure 4: A toy MDP illustrating the RPI property of ARMOR. (Top) The true MDP has deterministic dynamics where taking the left  $(a_{l})$  or right  $(a_{r})$  actions takes the agent to corresponding states; start state is in yellow. The suboptimal behavior policy only visits only the left part of the state space, and the reference policy demonstrates optimal behavior by always choosing  $a_{r}$ . (Bottom) A subset of possible data-consistent MDP models (dynamics + rewards) in the version space. The adversary always chooses the MDP that makes the reference maximally outperform the learner. In response, the learner will learn to mimic the reference outside data support to be competitive.
Possible Dynamics and Rewards in Unknown Region
new states and actions that would cover those generated by running the reference policy in the true environment, even though ARMOR does not have knowledge of  $M^{\star}$ .
F Further Experimental Details
F.1 Experimental Setup and Hyper-parameters
We represent our policy  $\pi$ , Q-functions  $f_{1}, f_{2}$  and MDP model  $M$  as standard fully connected neural networks. The policy is parameterized as a Gaussian with a state-dependent covariance, and we use a tanh transform to limit the actions to the action space bound similar to Haarnoja et al. (2018). The MDP model learns to predict the next state distribution, rewards and terminal states, where the reward and next-state distributions part are parameterized as Gaussians with state-dependent covariances. The model fitting loss consists of negative log-likelihood for the next-state and reward and binary cross entropy for the terminal flags. In all our experiments we use the same model architecture and a fixed value of  $\lambda$ . We use Adam optimizer (Kingma and Ba, 2015) with fixed learning rates  $\eta_{fast}$  and  $\eta_{slow}$  similar to Cheng et al. (2022). Also similar to prior work (Kidambi et al., 2020), we let the MDP model network predict delta differences to the current state. The rollout horizon is always set to the maximum episode steps per environment. A complete list of hyper-parameters can be found in Table 3.
Compute: Each run of ARMOR has access to 4 CPUs with 28GB RAM and a single Nvidia T4 GPU with 16GB memory. With these resources each run tasks around 6-7 hours to complete. Including all runs for 4 seeds, and ablations this amounts to approximately 2500 hours of GPU compute.
Hyperparameter	Value
model_num_layers	3
model_hidend_size	512
model_nonlinearity	swish
policy_num_layers	3
policy_hidend_size	256
policy_nonlinearity	relu
f_num_layers	3
f_hidend_size	256
f_nonlinearity	relu
Table 2: Model Architecture Details
Hyperparameter	Value
critic learning rate ηfast	5e-4
policy learning rate ηslow	5e-7
discount factor	0.99
rollout horizon	max episode steps
model buffer size	10^6
batch size	125
model batch size	125
num warmstart steps	10^5
τ	5e-3
F.2 Detailed Performance Comparison and RPI Ablations
In Table 4 we show the performance of ARMOR compared to model-free and model-based offline RL baselines with associate standard deviations over 8 seeds. For ablation, here we also include  $\mathsf{ARMOR}^{\dagger}$ , which is running ARMOR in Algorithm 1 but without the model optimizing for the Bellman error (that is, the model is not adversarial). Although  $\mathsf{ARMOR}^{\dagger}$  does not have any theoretical guarantees (and indeed in the worst case its performance can be arbitrarily bad), we found that  $\mathsf{ARMOR}^{\dagger}$  in these experiments is performing surprisingly well. Compared with ARMOR,  $\mathsf{ARMOR}^{\dagger}$  has less stable performance when the dataset is diverse (e.g. -med-replay datasets) and larger learning variance. Nonetheless,  $\mathsf{ARMOR}^{\dagger}$  using a single model is already pretty competitive with other algorithms. We conjecture that this is due to that Algorithm 1 also benefits from pessimism due to adversarily trained critics. Since the model buffer would not cover all states and actions (they are continuous in these problems), the adversially trained critic still controls the pessimism for actions not in the model buffer, as a safe guard. As a result, the algorithm can tolerate the model quality more.
Table 3: List of Hyperparameters.
Dataset	ARMOR	\( ARMOR^† \)	\( ARMOR^{re} \)	MoREL	MOPO	RAMBO	COMBO	ATAC	CQL	IQL	BC
hopper-med	101.4 ± 0.3	100.4 ± 1.7	65.3 ± 4.8	95.4	28.0 ± 12.4	92.8 ± 6.0	97.2 ± 2.2	85.6	86.6	66.3	29.0
walker2d-med	90.7 ± 4.4	91.0 ± 10.4	79.0 ± 2.2	77.8	17.8 ± 19.3	86.9 ± 2.7	81.9 ± 2.8	89.6	74.5	78.3	6.6
halfcheetah-med	54.2 ± 2.4	56.3 ± 0.5	45.2 ± 0.2	42.1	42.3 ± 1.6	77.6 ± 1.5	54.2 ± 1.5	53.3	44.4	47.4	36.1
hopper-med-reply	97.1 ± 4.8	82.7 ± 23.1	68.4 ± 5.2	93.6	67.5 ± 24.7	96.6 ± 7.0	89.5 ± 1.8	102.5	48.6	94.7	11.8
walker2d-med-reply	85.6 ± 7.5	78.4 ± 1.9	50.3 ± 5.7	49.8	39.0 ± 9.6	85.0 ± 15.0	56.0 ± 8.6	92.5	32.6	73.9	11.3
halfcheetah-med-reply	50.5 ± 0.9	49.5 ± 0.9	36.8 ± 1.5	40.2	53.1 ± 2.0	68.9 ± 2.3	55.1 ± 1.0	48.0	46.2	44.2	38.4
hopper-med-exp	103.4 ± 5.9	100.1 ± 10.0	89.3 ± 3.2	108.7	23.7 ± 6.0	83.3 ± 9.1	111.1 ± 2.9	111.9	111.0	91.5	111.9
walker2d-med-exp	112.2 ± 1.7	110.5 ± 1.4	105.8 ± 1.4	95.6	44.6 ± 12.9	68.3 ± 15.0	103.3 ± 5.6	114.2	98.7	109.6	6.4
halfcheetah-med-exp	93.5 ± 0.5	93.4 ± 0.3	61.8 ± 3.75	53.3	63.3 ± 38.0	93.7 ± 10.5	90.0 ± 5.6	94.8	62.4	86.7	35.8
pen-human	72.8 ± 13.9	50.0 ± 15.6	62.3 ± 8.35	-	-	-	-	53.1	37.5	71.5	34.4
hammer-human	1.9 ± 1.6	1.1 ± 1.4	3.1 ± 1.9	-	-	-	-	1.5	4.4	1.4	1.5
door-human	6.3 ± 6.0	3.9 ± 2.4	5.9 ± 2.75	-	-	-	-	2.5	9.9	4.3	0.5
relocate-human	0.4 ± 0.4	0.4 ± 0.6	0.3 ± 0.25	-	-	-	-	0.1	0.2	0.1	0.0
pen-cloned	51.4 ± 15.5	45.2 ± 15.8	40.0 ± 8.25	-	-	-	-	43.7	39.2	37.3	56.9
hammer-cloned	0.7 ± 0.6	0.3 ± 0.0	2.7 ± 0.15	-	-	-	-	1.1	2.1	2.1	0.8
door-cloned	-0.1 ± 0.0	-0.1 ± 0.1	0.5 ± 0.4	-	-	-	-	3.7	0.4	1.6	-0.1
relocate-cloned	-0.0 ± 0.0	-0.0 ± 0.0	-0.0 ± 0.0	-	-	-	-	0.2	-0.1	-0.2	-0.1
pen-exp	112.2 ± 6.3	113.0 ± 11.8	92.8 ± 9.25	-	-	-	-	136.2	107.0	-	85.1
hammer-exp	118.8 ± 5.6	115.3 ± 9.3	51.0 ± 11.05	-	-	-	-	126.9	86.7	-	125.6
door-exp	98.7 ± 4.1	97.1 ± 4.9	88.4 ± 3.05	-	-	-	-	99.3	101.5	-	34.9
relocate-exp	96.0 ± 6.8	90.7 ± 6.3	64.2 ± 7.3	-	-	-	-	99.4	95.0	-	101.3
Table 4: Performance comparison of ARMOR against baselines on the D4RL datasets. The values for ARMOR denote last iteration performance averaged over 4 random seeds along with standard deviations, and baseline values were taken from their respective papers. Boldface denotes performance within  $10\%$  of the best performing algorithm.
F.3 Effect of Residual Policy
In Figure 5, we show the effect on RPI of different schemes for initializing the learner for several D4RL datasets. Specifically, we compare using a residual policy (Section 5) versus behavior cloning the reference policy on the provided offline dataset for learner initialization. Note that this offline dataset is the suboptimal one used in offline RL and is different from the expert-level dataset used to train and produce the reference policy. We observe that using a residual policy (purple) consistently shows RPI across all datasets. However, with behavior cloning initialization (pink), there is a large variation in performance across datasets. While RPI is achieved with behavior cloning initialization on hopper, walker2d and hammer datasets, performance can be arbitrarily bad compared to the reference on other problems. As an ablation, we also study the effect of using a residual policy in
(g) Relocate
Figure 5: Comparison of different policy initializations for RPI with varying pessimism hyperparameter  $\beta$ . ORL denotes the performance of offline RL with ARMOR (Table 1), and REF is the performance of reference policy. Purple represents residual policy initialization and pink is initialization using behavior cloning of the reference on the suboptimal offline RL dataset.
the offline RL case where no explicit reference is provided, and the behavior cloning policy is used as the reference similar to Section 5.1. We include the results in Table 4 as  $\mathrm{ARMOR}^{re}$ , where we observe that using a residual policy overall leads to worse performance across all datasets. This lends evidence to the fact that using a residual policy is a compromise in instances where initializing the learner exactly to the reference policy is not possible.
F.4 Connection to Imitation Learning
Dataset	ARMOR-IL	BC
hopper-exp	111.6	111.7
walker2d-exp	108.1	108.5
halfcheetah-exp	93.9	94.7
Table 5: ARMOR-IL on expert datasets. By setting  $\lambda  = 0,\beta  > 0$  we recover IL.
As mentioned in Section 4.2, IL is a special case of ARMOR with  $\lambda = 0$ . In this setting, the Q-function can fully affect the adversarial MDP model, so the best strategy of the policy is to mimic the reference. We test this on the expert versions of the D4RL locomotion tasks in Table 5, and observe that ARMOR can indeed perform IL to match expert performance.
F.5 Ablation Study: RPI for Different Reference Policies
Here we provide ablation study results for robust policy improvement under different reference policies for a wide range of  $\beta$  values (pessimism hyper-parameter). For all the considered reference policies we present average normalized scroes for ARMOR and reference (REF) over multiple random seeds, and observe that ARMOR can consistently outperform the reference for a large range of  $\beta$  values.
Random Dataset Reference We use a reference policy obtained by running behavior cloning on the RANDOM versions of different datasets. This is equivalent to using a randomly initialized neural network as the reference.
Dataset	0.01	0.05	0.1	0.5	1.0	10.0	100.0	200.0	500.0	1000.0	REF
hopper-med	1.3	1.5	4.7	9.6	20.4	34.8	25.8	40.8	25.6	28.9	1.2
walker2d-med	0.0	0.0	0.2	1.5	4.0	17.4	23.6	12.4	12.1	20.1	0.0
halfcheetah-med	0.0	0.1	0.1	0.7	1.2	-0.1	-0.7	0.1	1.1	-0.3	-0.1
hopper-med-replay	1.3	3.0	8.2	13.3	39.5	57.6	48.0	34.4	51.0	32.9	1.2
walker2d-med-replay	0.0	0.0	0.1	4.0	5.9	13.1	10.8	14.1	16.5	16.6	0.0
halfcheetah-med-replay	-0.2	0.4	0.3	0.7	0.9	7.7	5.8	8.2	4.9	6.1	-0.2
Hand-designed Reference In this experiment we use a hand-designed reference policy called RAN-DOMBANGBANG, that selects either the minimum of maximum action in the data with 0.5 probability each.