How to Solve Contextual Goal-Oriented Problems with Offline Datasets?
Authors: Ying Fan
Jingling Li
Adith Swaminathan
Aditya Modi
Ching-An Cheng
Venue: NeurIPS 2024 poster (2024)
Keywords: offline RL
goal-conditioned RL
Primary area: reinforcement_learning

Abstract: We present a novel method, Contextual goal-Oriented Data Augmentation (CODA), which uses commonly available unlabeled trajectories and context-goal pairs to solve Contextual Goal-Oriented (CGO) problems. By carefully constructing an action-augmented MDP that is equivalent to the original MDP, CODA creates a fully labeled transition dataset under training contexts without additional approximation error. We conduct a novel theoretical analysis to demonstrate CODA's capability to solve CGO problems in the offline data setup. Empirical results also showcase the effectiveness of CODA, which outperforms other baseline methods across various context-goal relationships of CGO problem. This approach offers a promising direction to solving CGO problems using offline datasets.

How to Solve Contextual Goal-Oriented Problems with Offline Datasets?
Ying Fan', Jingling Li?, Adith Swaminathan", Aditya Modi Ching-An Cheng? University of Wisconsin-Madison *ByteDance Research
Miicrosoft Research
Abstract
We present novel method, Contextual goal-Oriented Data Augmentation (CODA), which uses commonly available unlabeled trajectories and context-goa pairs solve Contextual Goal-Oriented (CGO) problems. By carefully constructing an aetion-augmented MDP that is equivalent to the original MDP, CODA creates a fully labeled transition dataset under training contexts without additional approx- imation error. We conduct a novel theoretical analysis to demonstrate CODA's capability to solve CGO problems in the offline data setup. Empirical results also showcase the effectiveness of CODA, which outperforms other baseline methods across various context- goal relationships of CGO problem. This approach offers a promising direction to solving CGO problems using offline datasets.
1 Introduction
Goal-oriented problems [16] are an important class of sequential decision- making problems with widespread applications, ranging from robotics [39], game-playing [12], to logistics [24]. In particular, many real-world goal oriented problems are contextual, where the objective of the agent is to reach goal set communicated by context. For example, consider instructing a truck operator with the context "Deliver goods warehouse in the Bay area". Given such a context and an initial state, is acceptable to reach any feasible goal (a reachable warehouse location) in the goal set (warehouse locations including non-reachable ones). We call such problems Contextual Goal-Oriented (CGO) problems, which form an important special case of contextual Markov Decision Process (MDP) [10].
CGO is a practical setup that includes goal-conditioned reinforcement learning (GCRL) as a special case (the context in GCRL is just the target goal), but in general contexts in CGO problem can be more abstract (like high-level task instructions in the above example) and the relationship between contexts and goals are not known beforehand. CGO problems are challenging because 1) the rewards are sparse as in GCRL and 2) the contexts can be difficult to map into feasible goals. Nevertheless, CGO problem has an important structure that the transition dynamics (e.g., navigating city road network) are independent of the contexts that specify tasks. Therefore, efficient multitask learning can be achieved by sharing dynamics data across tasks.
In this paper, we study solving for CGO problems in an offline setup. We suppose access two datasets an (unlabeled) dynamics dataset of trajectories, and (labeled) context-goal dataset containing pairs of contexts and goal examples. Such datasets are commonly available in practice. The typical contextual datasets for imitation learning (IL) (which has pairs of contexts and expert trajectories) is one example, since can convert the contextual IL data into dynamics data and context-gon -oal pairs. Generally, this setup also covers scenarios where expert trajectories are not accessible (e.g., because of diverse contexts and initial states), since does not assume goal examples to appear in the trajectories or the contexts are readily paired with transitions in expert trajectories. Instead, it allows the dynamics datasets and the context-goal datasets be independently collected. For example, in robotics, task-agnostic play data can be obtained at scale [22, 34] in an unsupervised manner whereas instruction datasets (e.g., (25)) can provide context-goal pairs. navigation, self-
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

tonottmminall states
Terminal state
Real transitions with real actions, unlabeled
Fictitious transitions given context with fictitious action Fictiious transittions given context 3, with ficiiiiuss action
meoeg Fictitious transitions ghen context 1, with fictibous action
Figure I:
Illustration of CODA: We create fictitious transitions from goal examples to terminal states under the given context in the action-augmented MDP with reward 1, which enables the supervised signal to propagate back to unsupervised transitions via Bellman equation.'
driving car trajectories (e.g., [35, 32J) also allow us to learn dynamics whereas landmarks datasets (e.g. [24, 9I) provide context-goal pairs.
While offline CGO problems as described above are common in practical scenarios, no algorithms have been specifically designed to solve such problems and CGO has not been formally studied yet. Some baseline methods could be easily conceptualized from the literature, but their drawbacks are equally apparent. One intuitive approach is to extend the goal prediction methods in GCRL [26, 27]: given a test context, we can predict a goal and navigate to using goal-conditioned policy, where the goal prediction model can be learned from the context-goal dataset and the goal- conditioned policy can be learned from the trajectory dataset. However, the predicted goal might not always be feasible given the initial state since our context-goal dataset is not necessarily paired with transitions. Alternatively, the offline problem could be formulated as a special case of missing label problems [41] and we can learn a context-conditioned reward model to label the unsupervised
knowledge,
transitions when paired with contexts as in [14]. However, this approach ignores the gral-oriented nature of the problem and the fact that here only positive data (i.e. goal examples) are available for reward learning, which poses extra significant challenges. CGO can be framed as an offline reinforcement learning (RL) problem with missing labels; However, existing algorithms [42, 14, 21] in family assume access to both positive data (contexts-goal pairs) and negative data (contexts and non-goal examples), whereas only positive data are available here.
In this work, we present the first precise formalization of the CGO setting, and propose novel Contextual
Data Augmentation (CODA) technique that can provably solve CGO problems subject to natural assumptions on the datasets' quality. The core idea is convert the context-goal dataset and the unsupervised dynamics dataset to a fully labeled transition dataset of an equivalent action-augmented MDP, which circumvents the drawbacks in other baseline methods by fully making use of the CGO structure of the problem. We give high-level illustration of this idea in Figure 1. In Figure 1, given a randomly sampled context-goal pair from the context-goal dataset, we create fictitious transitions from the corresponding goal example to a fictitious terminal state with a fictitious action and reward I, and pair with the corresponding context. Also, we label all unsupervised transitions with reward 0 and non-terminal, and pair with the contexts randomly. Combining the two, we then have a fully labeled dataset (of an action- autiione contextual MDP, which this data augmentation and relabeling process effectively creates), making it possible to propagate supervision signals from the context-goal dataset to unsupervised transitions via the Bellman equation. We can then apply any offline RL algorithm based Bellman updates like CQL [19]. IQL [18], PSPI [37], ATAC [4] etc. In comparison with the baseline methods discussed earlier, our method naturally cireumvents their intrinsic challenges: 1) CODA directly learns context- conditioned policy and avoids the need to predict goals; 2) CODA effectively uses a fully labeled dataset, avoiding the need to learn reward model and extra costs from inaccurate reward modeling.
2 Related Work
Offline RL. Offline RL methods have proven to be effective in goal-oriented problems as also
allows learning common set of sub- sub-goals/skills [3, 23, 38). A variety of approaches are used to mitigate the distribution shift between the collected datasets and the trajectories likely to be generated
by learned policies: 1) constrain target policies to be close to the dataset distribution [8, 36. 71, 2) incorporate value pessimism for low-coverage or -ff--tr-itiiittio states and actions [19, 40, 15] and 3) adversarial training via two-player game [37, 4].

---

Offline RL with unlabeled data. Our CGO setting is a special case of offline RL with unlabeled data, or more broadly the offline policy learning from observations paradigm [21]: There is only a subset of the offline data labeled with rewards (in our setting, that is the contexts dataset, as we don't know which samples in the dynamics dataset are goals.). However, the MAHALO scheme in [21] is much more general than necessary for CGO problems, and we show instead that our CODA scheme has better theoretical guarantees than MAHALO in Section 5. In our experiments, we compare CGO with several offline RL algorithms designed for unlabeled data: UDS [42] where unlabeled data is assigned zero rewards and PDS [14] where a pessimistic reward function is learned from labeled dataset.
cond-cond RL (GCRL), GCRL special case of our CGO setting, which has been extensively studied since [16]. There are two critical aspects of GCRL: 1) data relabeling to make better use of available data and 2) learning reusable skills to solve long-horizon problems by chaining sub-goals or skills. On the one hand, hindsight relabeling methods [1. 20] are effective by reusing visited states in the trajectories as successful goal examples. For 2), hierarchical methods for determining sub-goals, and training goal reaching policies have been effective in long-horizon problems [28. 30, 33. Another key objective of GCRL is goal generalization. Popular strategies include universal value function approximators [29], unsupervised representation learning [26, 28, 1 I1, and iissimism-inducee generalization in offline GCRL formulations [38]. Our CGO framing enables both data reuse and goal generalization, by using contextual representations and reduction to offline RL to combine dynamics and context-godd goal datasets.
Data-sharing in RL Sharing information across multiple tasks is a promising approach to accelerate learning and to identify transferable features across tasks. In RI., both multi-task and transfer learning settings have been studied under varying assumption on the shared properties and structures of different tasks [43, 33, 2. 5]. For data sharing in CGO, we adopt the contextual MDP formulation [10, 31J, which enables knowledge transfer via high-level contextual cues. Prior work on offline RL has also shown the utility of sharing data across tasks; hindsight relabeling and manual skill grouping [17], inverse RL [20], sharing Q-value estimates [41, 30] and reward labeling [42, 14].
3 Preliminaries
In this section, we introduce the setup of CGO problems, infinite-horizon formulation for CGO, and the offline learning setup with basic assumptions for our offline dataset.
CGO Setup A Contextual Goal-Oriented (CGO) problem describes a multi-task gral-oriented setting with a shared transition kernel. We consider Markovian CGO problem, defined by the tuple M = (S, P, R. 7, C, do), where S is the state space, A is the action space, P: x A + ASS) is the transition kernel, R: S x C + {0, 1} is the sparse reward function, E 10, is the discount factor, C is the context space, and denotes the space of distributions.
Each context c E C specifies a goal-reaching task with goal set G. C S, and reaching any goal in the goal set Ge is regarded as successful, inducing the reward function R(s, c) = 1(s E Ge). An episode of a CGO problem starts from initial state 8o and context e sampled from do(so, c). and terminates when the agent reaches the goal set G.. e does not change during the transition; only 8t changes according to P(s' (s, a) and the transition kernel is context-independen
infinite-horizon Formulation for CGO setup A fictitious zero-reward absorbing state s+ & S can translate termination after reaching the goal to an infinite horizon formulation: whenever the agent enters G. it transits to s+ in the next step (for all actions) and stays at s+ forever. This is a standard technique to convert a goal-reaching problem (with a random problem horizon) to an infinite horizon problem. This translation does not change the problem, but allows cleaner analyses. We adopt this formulation in the following.
We give details of this infinite-horizon conversion in the following. First, we extend the reward and the dynamics: Let S SU(s+), X = S x C, and X = S C. Define X+ = (x x = (s, c), = st ce E C). With abuse of notation, we define the reward and transition on X as R(x)) = 1(s Go)
3

---

where r = (s, The transition kernel P(z' xr a) = P(s)|s,c,a)I(( = c), where
P( 1(ss' st) if Ge or =st
(( otherwise.
Given a policy X - (AA), the state-action value function (i.e., Q function) is Q (x; Ex.P 2 R = I, ag = a). (*) = Q" (ar, T) is the value function given n, where Q(I,)) Lova [Q(z, a]]
[0.1]. The return ( (*(o)) Q" (do, T). x² is the optimal policy that maximized J(T) and Q* = Q V V*" Let G represent the goal set X' that is, x f
G= fz
(s, c). s E Ge)-
Offline Learning for CGO We aim solve CGO problems using offline datasets without additional online environment interactions, namely, by offine RL. We identify two types of data that are commonly available: Dayn a(s,a,s') is unsupervised dynamics dataset of agent trajectories collected from P(s" 8, a). and Dgeal == {(c, s) E Ge} is supervised dataset of context-goal pairs, which be easier collect than expert trajectories. We suppose that there two distributions Mayn (8, 0, 8') and (ggoa( c), where Maya (sls,, |s, a) = P[s'|s,00 a) and Mgeal has support within Ge i.e., s E Ge. We assume that Dayn and goal are i.i.d. samples drawn from the distributions Jdyn and Mgoal. i.c.,
Mgoal (s|c) 0 +
Dayn [(s;, ai, s()) (dya), Dgeal {(s}, ey) )goal).
Notice that we do not assume the goal states in D goal to be in Daya- thus cannot always naively pair transitions in Day with contexts in Doaa and assign them with reward 1. To our knowledge, no existing algorithm can provably learn near-optimal π using only the positive D goal data (i.e., without non-goal examples) when combined with Dayn data.
Contextual Goal-Oriented Data Augmentation (CODA)
The key idea of CODA is the construction an actior-augmented MDP with which the dynamics and context-gon datasets can be combined into fully labeled offline RL dataset. In the following, we first describe this action-augmented MDP (Section 4.1) I) and show that preserves the optimal policies of the original MDP (Appendix A.1). Then outline practical algorithm to convert the two datasets of an offline CGO problem into a dataset for this augmented MDP (Section 4.2) such that any generic offline RI algorithm based on Bellman equation can be used as solver.
4.1 Action-Augmentedd MDP
We propose an action- augmeened MDP (shown in Figure 1), which augments the action space of the contextual MDP in Section 3 with a fictitious action a* & A.
Ā Aatt]. We define the reward of this action-augmented MDP be action-dependent: for & = () E X, a(x,a)== == 1ss E Ga) = a+), ), which means the reward is only if a is taken in the goal set, otherwise
We also extend the transition upon taking action at: P(z'lz,at) =(z 1 st), and maintain the transition with real actions: P(x'|z,a) a) = P(s'ls, a)1(c' c), which means whenever taking at. the agent would always transit to 5*, and the transition remains the same as in the original MDP given real actions. Further, implement st terminal True.
We define this augmented MDP as M = (R,A,R,,,1))
Policy conversion. For a policy x : X - A(A) in the original MDP, define its extension on M:
x #&G.
<(llz)
(1)
at otherwise.
Regret equivalence. An observation that comes with the construction is that a policy is optimal in the original MDP, we can easily use the extension above to create an optimal policy in the augmented one. If policy is optimal in the augmented MDP, it must take a only when æ E G (otherwise the return is lower, due to entering s- too early), thus we can revert this optimal policy of the augmented MDP find an optimal policy in the original MDP without changing its behavior and performance. We stated this property below; details can be found as Lemma A.3 in Appendix A.I.

---

Theorem 4.1 (Informal). The regret of a policy extended to the augmented MDP is equal to the regret of the policy in the original MDP, and any policy defined in the augmented MDP can be converted into that in the original MDP without increasing the regret. Thus, solving the augmented MDP can yield correspondingly optimal policies for the original problem.
Remark 4.2. The benefit of using the equivalent M is to avoid missing labels: given contexts in D gool- the rewards in M are known from our dataset setup in Section 3, whereas the rewards of the original MDP M are missing.
4.2 Method
CODA is designed based on the observation on regret relationship in Theorem 4.1: As described in Figure 1, given context-goal pair (s, c) from the dataset D Doaal we create a fictitious transition from to It with action at, reward 1 under context We also label all unsupervised transitions in the dataset Dayn with the original action and reward under c. In this way, we can have fully labeled transition dataset in the augmented MDP given any c from the context-goal dataset and then run offline algorithms (based on the Bellman equation) on this dataset. This CODA algorithm is formally stated in Algorithm 1. It takes two datasets D Dayn and goat as input, and produces labeled transition dataset Dayn U goal that is suitable for use by any offline RL algorithm based on Bellman equation like CQL [19], IQL [18], PSPI [37], ATAC [4], etc.
Interpretation. Why would our action augmentation make sense? We consider dynamic program- ming on the created dataset. Imagine we have fictious transition from to s with a under context When we calculate (x) via Bellman equation where r = (s, c). will choose the action with the highest Q" value in the augmented action space. The fictitious action would be the optimal action since it induces the highest Q" value?, meaning is already in Ge, and no further action is needed. Then the value of (x(x) would naturally propagate to some state Tprer = (Sprev, c) via Bellman equation if x is reachable starting from Tprev as shown Figure 1, so Fprev would still have meaningful values even with the intermediate reward O. For to be reachable starting from xpev we do not require the exact s to appear in the trajectory dataset due to the generalization ability of the value function (details in Section 5). For non-goal states, such fictitious action never appears in the dataset, thus would not be the optimal action in Bellman equation in pessimistic offline RL. For example, the fictitious action never appears the candidate in argmax in algorithms like IQL, and would be punished as OOD actions in algorithms like CQL. We will prove this insight formally below in Section 5.
Algorithm 1 CODA for CGO
Input: Dynamics dataset Ddyn: context-goal dataset Deeu
for each sample (s, c) ~ D goal do
Create transition" (x, a aa,tt,,*" 11,+t), where x = (s, e) and x+ = (st, c), add it to D D
end for
for each as,0,'') ~ Dayn do for each (1) ~ D goal do
Create transition (Ir, at 0, 2'), where x (s,c) and x' (s), add it to Daym
end for
end for Output: Dayn and geoa
Remark 4.3, We do not need to learn to perform at for the policy in practice since it is only for fictitious transitions which is already inside the goal set in the original MDP. (From the proof of Lemma A.3, we know taking at is always strictly worse than taking actions in the original action space A.) Therefore, we simply use the original action space for policy modeling and only use the fictitious transitions in value learning. We note that in practice Algorithu I can be implemented as a preprocessing step the minibatch sampling of a deep offline RL. algorithm (as opposed to computing the full Dåy and Daoat once before learning).
Forr all a + at Q', Q' (z, a) < Q2 at) when < 1. If 1. the agent might also learn to travel to other goal states starting from with some probability, which is also acceptable in CGO.
5

---

5 CGO is Learnable with Positive Data Only
In Section 4, we show that fully labeled dataset can be created in the augmented MDP without inducing extra approximation emrors. But we still have access to negative data, i.c., context and non-goal pairs. A natural question arises: Can we learn to solve CGO problems with positive data only? What conditions needed for CGO to be learnable with offline datasets?
We show in theory that we do not need negative data to solve CGO problems by conducting a formal analysis for our method, instantiated with PSPI [37] as an example of the base algorithm. We present the detailed algorithm CODA+ in Appendix A.3. This algorithm uses function classes F S x A + IR and G S - IR to model value functions and optimizes the policy given a policy class II based on absolute pessimism defined on initial states.
We present our assumptions and the main theoretical result as follows.
Assumption 5.1 (Realizability). We assume for any т E II, Q and R E G, where F,g are the function classes for action-value and reward respectively.
Assumption 5.2 (Completeness). We assume: For any 1 E F.g E 9 and a E II. max(g(æ), (a, x)) E F; And for any f E T ((r,a) Bellman backup operator with respect to P(s T fz, = P(dsaas (f .)).
where T is a zero-reward
These two assumptions mean that the function classes F and G are expressive enough, which are standard assumptions in offline RL based on Bellman equation [37]. For deriving our main result, we define the coverage assumption needed below.
Definition 5.3. We define the generalized concentrability coefficients:
||S -'''
l1g RIpP
Can(T))= max S.''ET 11f - TT* ff
and (xou(TT) == max
(2)
||g Rlli
ET.P where |/bl = Evry ( Pec Er ET
7/1 a, a)). pēc(x)
F 1 = =)], and T is the first time the agent enters the goal set.
Concentrability coefficients is generalization notion of density ratio: It describes how much the (unnormalized) distribution in the numerator is "covered" by that in the denominator in terms of the generalization ability of function approximators [37]. If Cdyn(T), Cgoal (T) are finite given Mggoal Pdyn, F, and u. then we say π is covered by the data distributions, and conceptually offline RL can learn policy to be no worse than π
We state theoretical result, which is proven by a careful reformulation of the Bellman equation of the aegten- MDP, and construct augmented value function and policy classes in the analysis using the CGO structures (see Appendix A).
Theorem 5.4. Let n² denote the learned policy of CODA PSPI with datasets Dāy and D goat, using value function classes F = {X x A + [0, 1]} and G = (X + [0, 1]}. Under Assumption 5. 1, 5.2 and 5.3, with probability 1 6, it holds, for any E II,
((x) J(xt) S. Cam((T) (v log( F 9 11188)
log( F (1
log((91/8) DRoaal
(goa
[Dann]
D
where Cdm((T) and (T) are concentrability coefficients.
Interpretation. We interpret Theorem 5.4 follows: The statistical errors in value function estimation would decrease as we have more data from Mgeal and Maya; For any comparator π with finite coefficients (T)) (T)) the final regret upper bound would also decrease. Taking as an example. For the coefficients (ayu(T)) (g (T) to be finite, it indicates 1) the state-action distribution from the dynamics data "covers" the trajectories generated by x²' which includes the case of stitching 2) the support of /ggoal "covers" the goals x² would reach. We note that these conditions not any stronger than general requirements solve offline algorithms: The "coverage"
*We state a more general result for non-finit function classes Theorem A. in the appendix 'This does not mean the dynamics data have to be generated by the optimal policy; they can be generated by highly suboptimal policies so long they together provide sufficient coverage.

---

6333
(a)
(b)
(c)
Figure 2: Illustration of the context-goal relationship with increasing complexity (Each red boundary
defines goal set with its center location as context). (a) Contexts and goal sets are very similar such that it could be approximately solved by context-agnostic policy. (b) Contexts are finite, and different contexts map to distinct goal sets, which requires context-dependent policies. (c) Contexts are continuous and infinite. The context-goal mapping is neither one-to-many to-many nor many-to-one, creating CGO problem with full complexity.
above is measured based the generalization ability of f and g respectively as in Definition 5.3; e.g.. if f(11 and f222 are similar for #1 # Ø2, then 22 is within the coverage of μ so long as ø1 can be generated by z in terms of the generalization ability of f. Such coverage condition is weaker than coverage conditions based on density ratios. Besides, Theorem 5.4 simultaneously apply to all II not just π² Therefore, as long the above "coverage" conditions hold for any policy that can reach the goal set, the agent can learn to reach the goal set. Thus, we show that CODA with PSPI can provably solve CGO without the need for additional non-goal samples, i.e., CGO is learnable with positive data only.
Remark 5.5. Here we only require function approximation assumptions made the original MDP, without relying on functions defined on the fictitious action or completeness assumptions based on the fictitious transition. As result, our theoretical results are comparable with those of other approaches.
Remark 5.6. MAHALO [21] is a SOTA offline RL algorithm that can provably learn from unlabeled data. One version of MAHALO is realized top of PSPI in theory; however, their theoretical result (Theorem D.I) requires a stronger version concentrability, maxgeg ig-riliza/llla-- /larrlllr?? u to be small. In other words, it needs negative examples of (contexi, non goal state) tuples for learning.
Intuition for other base algorithms. Notice that PSPI is just one instantiation. Conceptually, the coverage conditions above also make sense for other pessimistic offline RL instantiations based on the Bellman equation (like IQL), since the key ideas used in the above analyses are that the regret relationship (Theorem 4.1) between the original MDP and the action augmented MDP (which is algorithm agnostic) and that pessimism together with Bellman equations can effectively propagate information from the context-god dataset (without the need for negative data). However, performing complete theoretical analyses CODA for all different offline RL algorithms out of the scope of this paper.
6 Experiments
In this section, we present the experimental setup and results for CODA. Code is publicly available at: https: //tgtubbtbu/ttt///// com/yingfan- coda.
For a comprehensive empirical study, we first introduce the diverse spectrum of practical CGO setups.
Diverse spectrum of practical CGO problems. The main challenge of the CGO problem compared with traditional goal-conditioned RL is the potential complexity in the context-goal relationship. Therefore, to showcase the efficacy of different methods, we construct three levels with increasing difficulty as shown in Figure 2: (a) has a similar complexity as a single-task problem where the context does not play significant role; (b) requires (c) has infinite continuous context, requiring a context-dependent policy and generalization ability to contexts outside the offline data set. We aim to answer the following questions: 1) Does our method work under the data assumptions in Section 3, with different levels of context-goal complexity? 2) Is there any empirical benefit from using CODA, compared with baseline methods including reward learning, goal prediction, etc?
dependet policy but only has finite contexts;
7

---

6.1 Environments and Datasets
Dynamics dataset. For all experiments, we use the original AntMaze-v2 datasets different mazes and 6 offline datasets) of D4RL [6] as dynamics datasets Däyn: removing all rewards and terminals.
Context-goal dataset. We construct three levels of context and goal relationships shown in Figure 2. For each setup, we first define the context set, and then sample a fixed set of states from the offline trajectory dataset that satisfies the context-goal relationship, and then randomly perturb the states such that there would be no way to directly match goal examples to some states in the trajectories given contexts. Notice that this context-goal relationship only used for dataset construction and is not accessible to the learning algorithm." 6 The specific context-goal relationship are discussed in Section 6.3 with the construction/evaluution details Appendix B.2.
6.2 Method and Baselines
For controlled experiments, we use IQL [18] the same backbone offline algorithm for all the meth- ods with the same set of thperparameters. Our choice of IQL is motivated by both its benchmarked performance several RL domains and its structural similarity to PSPI (use of value/policy function classes along with pessimism). Please see Appendix B.1 for hyperparameters.
We describe the algorithms compared in the experiments.
CODA. We apply CODA in Algorithm 1 with IQL as the offline RL algorithm solve the augmented MDP defined in Section 4.1 More specifically, we set a to be an extra dimension in the action space of the action-value function, and model the policy with the original action space. Empirically, found that equally balancing the samples Dayn and Deoal generates the best result". Then we apply IQL on this labeled dataset.
Reward prediction, For this family of baselines, we need to use the learned reward to predict the label of context-goal samples in the randomly sampled context-transition pairs during training, so we need to pre-train a reward model using the context-gov goal dataset. We use PDS [14] for reward modeling, and learn pessimistic reward function using ensembles of models on the context- goal dataset. Then we apply the reward model to label the transitions with contexts, run IQL on this labeled dataset, and get a context-dependent policy. Besides PDS, also test naive reward prediction (RP, which follows the same setup of PDS but without ensembles) and UDS [42] +RP in Section 6.3 (See details in Appendix B.1). Additionally, we add results from training with the oracle reward (marked as "Oracle Reward") where we provide the oracle reward for any query context-goal pairs, as reference of the performance upper bound for reward prediction methods.
Goal prediction. We consider another GCRL-based baseline. Notice that the relationship between contexts and goals is unknown in CGO, we cannot directly apply traditional GCRL methods to CGO problems. Therefore, we adopt a workaround to use GCRL methods: We learn a conditional generative model as the goal predictor using classifier-free diffusion guidance [13], where the contexts serve as the condition, and the goal examples are used to train the generative model. We also learn a general goal-conditioned policy with the dynamics-only dataset using HER [I]+IQL. Given test context, the goal predictor samples the goal given the context, which is then passed as the condition to the policy.
6.3 Results
Original AntMaze: Figure 2(a). In the original AntMaze, 2D goal locations (contexts) are limited to small area as Figure (a). To make it CGO problem, we make the test context visible to the agent. This setting in Figure is approximately single-task problem.
CODA generally achieves better performance than reward learning and goal prediction methods. Comparing the normalized return in each AntMaze environment for all methods, our method consis- tently achieves equivalent or better performance each environment compared to other baselines
'Also note that the state space Antmaze not only includes the 2D location; it also includes data from robotic arms, etc. We define the context-gon goal relationship only on the 2D location and ignore other information. We study the effect of this sampling ratio CODA's performance in Table in Appendix B.

---

(Table 1). 8 Moreover, the performance of Goal Prediction is rather poor, which mainly comes from enough goal examples to learn from in this setup due limited goal area.
Table Average success rate (%) in AntMaze-v2 from all environments.
Env/Method
CODA (Ours)
PDS
Goal Prediction
RP
UDS+RP
Oracle Reward
umaze
94.8+1.3
93.0+1.3
46.4+6.0
50.5+2.1
54,3+6.3
94,4=0.61
umaze diverse
72.8+7.7
50.6+7.8
42.8+4.4
72.8+2.6
71,5+4.3
76.8+5.44
medium play
75,8+1.9
66.8+4.9
43.8+4.7
0.5+0.3
0.3+0.3
80,6+1.56
medium diverse
84.5+5.2
22.8+2.4
28.6+3.9
0,5+0.5
0.8+0.5
72.4+4.26
large play
60.0=7.6
39.644.9
13.0+4.0
0+0
0+0
41.2+3.58
large diverse
36.8#6.9
30.0+5.3
12.6+2.7
34.2+2.59
average
70.8
50.5
31.2
20.7
21.2
66.6
Four Rooms: Figure 2(b). We partition the maze into four rooms as in Figure 2(b), where the discrete room numbers (1,2,3,4) serve as contexts and we uniformly select test contexts. A context-dependent policy is needed, but there is no generalization required for unseen contexts in this setup.
We show the normalized return (average success rate in percentage) in each modified Four Rooms environment for our method and baseline methods in Table 2, where our method consistently outperforms the performances of baseline methods.
Table Average scores from Four Rooms with perturbation. The score for each run the average success rate (%) of the other three rooms.
Env/Method medium-pla play medium--iverse
CODA (Ours)
PDS
Goal Prediction
Oracle Reward
78.720.9
46.024.47
59.342.6
77.742.0
83.6+1.9
51.3+3.6
66.7+2.4
87.4+1.2
large-play
65.5+2.5
13.9+2.4
41.4+3.6
67.242.7
large-diverse
72.242.9
11.143.8
42.0+3.0
69.643.1
average
75.0
30.6
52.4
75.5
Figure 2(c). We use diverse distribution of contexts as shown in Figure 2(c), where the contexts are randomly sampled from non-wall states. For test contexts, we have two settings: sampling from the training distribution; 2) sampling from a far-away area from the start states.
Random Cells:
Overall, CODA outperforms the baselines under the setup in Figure 2(c). We show the normalized return (average success rate in percentage) each modified Random Cells environment in Table 3, which also shows the generalization ability of our method in the context space. CODA also generalizes to different test context distribution: We also test with distribution shift of the contexts in Table 4. We can observe that when tested with this different context distribution, CODA still generates better overall results compared to reward learning and goal prediction baselines.
Table 3: Average scores from Random Cells. The score for each run is the average success rate (%)
of random test contexts from the same training distribution.
Env/Method medium-pla.
CODA (Ours)
PDS
Goal Prediction
Oracle Reward
76,8+6.1
52.0+8.8
66. 7.27
71.9+0.1
medium-diverse
78.2+6.5
60.9+11.3
69.7+8.7
79.3x6.1
large-play
57.6+12.4
50.6+6.4
42.4+8.2
49.4+9.3
large-diverse
54.748.8
58,3+9.2
44.248.1
58,2+3.4
average
66.8
55.5
55.8
64.7
Reference to training with oracle reward. Notice that training with oracle reward is the skyline performance. From the results, training with oracle reward does not generally improve the perfor- mance much compared to CODA, though it generally outperforms PDS and Goal Prediction. This is mainly due to the sparsity of the positive samples the randomly sampled context-transine pairs. On the other hand, CODA easily uses these positive examples via our augmentation, which is another advantage our method over reward prediction baselines.
*We find umaze is too easy: even if the reward labeling is bad it still has relatively high reward, so we also omit it in other experiments. We also find UDS and RP are not very effective in our data setup, we also omit them in other experiments.

---

Table 4: Average from Random Cells with perturbation. The for each run is the average success rate (%) random test contexts with a distribution shift.
Env/Method medium-ples play
CODA (Ours)
PDS
Goal Prediction
Oracle Reward
67.918.2
50.1
70.5+1.9
67 .27.2
medium-diverse
72.5+6.5
57.5#14.8
63.0+7.2
68.7+7.9
large-play
60.2+4.8
48.1+8.0
44.3+4.1
59.8+4.4
large-diverse
58.045.8
44.129.9
55.425.7
57.627.6
average
64.7
49.9
58.3
63.3
Evaluation of the Reward Model. We also visualize the learned reward model from reward learning baselines in Appendix B.3: PDS is consistently better at separating positive and negative datasets than UDS and naive RP, but PDS can still fail at fully separating positive and negative examples. Intuitively, our method does not require reward learning thanks to the construction of the augmented MDP, which avoids the extra errors in reward prediction and leads to better performance.
6.4 Discussion and Limitation
Our experiments are limited to low-dimensional simulations. Nevertheless, the success of our method with diverse context-goal relationships serves first milestone to showcase its effectiveness, and we believe CODA would be useful in real-world scttings (c.g., learning visual-language robot policies) for its simplicity and theoretical guarantees. Potential scaling up by incorporating features from large pretrained models would be an exciting future direction, which can make our method generalizable to the real world.
7 Conclusion
We propose CODA for offline CGO problems, and prove CODA can learn near-optimal policies without the need for negative labels with natural assumptions. We also validate the efficacy of CODA experimentally, and find it outperforms other reward-learning and goal prediction baselines across various CGO complexities. We believe our method has the potential to generalize to real-world applications by further scaling up.
10

---

References
[1] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. In NeurIPS, 2017.
[2] André Barreto, Will Dabney, Rémi Munos, Jonathan Hunt, Tom Schaul, Hado van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. In NeurIPS, 2017.
[3] Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jacob Varley, Alex Irpan, Benjamin Eysenbach, Ryan C Julian, Chelsea Finn, et al. Actionable models: Unsupervised offline reinforcement learning of robotic skills. In ICML, 2021.
[4] Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning. In JCML, 2022.
[5] Carlo D Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, and Jan Peters. Sharing knowledge in multi-task deep reinforcement learning, In ICLR, 2020.
[6] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 2020.
[7] Scott Fujimoto and Shixiang Gu. minimalist approach to offline reinforcement learning. In NeurIPS, 2021.
[8] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In ICML., 2019.
(9] Meera Hahn, Devendra Singh Chaplot, Shubham Tulsiani, Mustafa Mukadam, James M Rehg, and Abhinav Gupta. No rl, no simulation: Learning to navigate without navigating. In NeurIPS, 2021.
[10] Assaf Hallak, Dotan Castro, and Shie Mannor. Contextual markov decision processes. arXiv preprint 1rXiv:1502.022555 2015.
[11] Beining Han, Chongyi Zheng, Harris Chan, Keiran Paster, Michael R Zhang, and Jimmy Ba. Learning domain invariant representations in goal-conditioned block mdps. In NeurIPS, 2021.
[12] Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojeiech Czarnecki, Simon Schmitt, and Hado Van Hasselt. Multi-task deep reinforcement learning with popart. In AAAI, 2019.
[13] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint
arXiv:2207. / 2298, 2022.
[14] Hao Hu, Yiqin Yang, Qianchuan Zhao, and Chongjie Zhang. The provable benefit of unsuper- vised data sharing for offline reinforcement learning. In ICLR, 2023.
[15] Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline rI? In ICML, 2021.
[16] Leslie Pack Kaelbling. Learning achieve goals. In IJCAI, 1993.
[17] Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-tass robotic reinforcement learning at scale. arXiv preprint arXiv:2104.08212. 2021.
[18] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforeement learning with implicit q-leaming. In ICLR, 2021.
[19] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine Conservative q-learning for offline reinforcement learning. In NeurlPS, 2020.
[20] Alexander C Li, Lerrel Pinto, and Pieter Abbeel. Generalized hindsight for reinforcement learning. In NeurlPS, 2020.
[21] Angi Li, Byron Boots, and Ching-An Cheng. Mahalo: Unifying offline reinforcement learning and imitation learning from observations. In ICML, 2023.
[22] Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In CORL, 2020.
[23] Yecheng Jason Ma, Jason Yan, Dinesh Jayaraman, and Osbert Bastani. Offline goal-conditioned reinforcement learning via f-dvvantage regression. In NeurIPS, 2022.

---

[24]
Piotr Mirowski, Matthew Koichi Grimes, Mateusz Malinowski, Karl Moritz Hermann, Keith Anderson, Denis Teplyashin, Karen Simonyan, Koray Kavukcuoglu, Andrew Zisserman, and Raia Hadsell. Learning to navigate in cities without map. In NeurlPS, 2018.
[25]
Dipendra K Misra, Jaeyong Sung, Kevin Lee, and Ashutosh Saxena. Tell me dave: Context- sensitive grounding of natural language manipulation instructions. International Journal of Robotics Research, 35(1-3):2811
2016.
[26]
Ashvin Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. In NeurIPS, 2018.
[27] Ashvin Nair, Shikhar Bahl, Alexander Khazatsky, Vitchyr Pong, Glen Berseth, and Sergey Levine. Contextual imagined goals for self-supervised robotic learning. In Conference Robot Learning, pages 530-539. PMLR, 2020.
[28] Suraj Nair and Chelsea Finn. Hierarchical foresight: Self-supervised learning of long-horizon tasks via visual subgoal generation. In ICI.R, 2019.
[29] Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxi- mators. In ICML., 2015.
[30] Avi Singh, Albert Yu, Jonathan Yang. Jesse Zhang, Aviral Kumar, and Sergey Levine. Cog: Connecting new skills to past experience with offline reinforcement learning. arXiv preprint arXiv:2010. 2020.
[31] Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-task reinforcement learning with context-based representations. In ICML, 2021.
[32] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR, 2020.
[33] Yee Whye Teh, Victor Bapst, Wojeiech Marian Czarnekki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: robust multitask reinforcement learning. In NeurIPS, 2017.
[34] Homer Rich Walke, Kevin Black, Tony Z. Zhao, Quan Vuong, Chongyi Zheng, Philippe Hansen- Estruch, Andre Wang He, Vivek Myers, Moo Jin Kim, Max Du, Abraham Lee, Kuan Fang, Chelsea Finn, and Sergey Levine. Bridgedata v2: A dataset for robot learning at scale. In CORL, 2023.
[35] Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, Deva Ramanan, Peter Carr, and James Hays. Argoverse 2: Next generation datasets for self-driving perception and forecasting. In NeurIPS, 2021.
[36] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint
2019.
[37] Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman- consistent pessimism for offline reinforcement learning. In NeurlPS, 2021.
[38] Rui Yang, Lin Yong, Xiaoteng Ma, Hao Hu, Chongjie Zhang, and Tong Zhang. What is essential for unseen goal generalization of offline goal-conditioned rl? In ICML, 2023.
[39] Albert Yu and Ray Mooney. Using both demonstrations and language instructions to efficiently learn robotic tasks. In ICLR, 2023.
[40] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: model-based offline policy optimization. In NeuriPS, 2020.
[41] Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Sergey Levine, and Chelsea Finn. Conservative data sharing for multi-task offline reinforcement learning. In NeurlPS, 2021.
[42] Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Chelsea Finn, and Sergey Levine. How to leverage unlabeled data in offline reinforcement learning. In ICML, 2022.
[43] Zhuangdi Zhu, Kaixiang Lin, Anil K Jain, and Jiayu Zhou. Transfer learning in deep reinforce- ment learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
12

---

A Theoretical Analysis
In this section, we provide a detailed analysis for the instantiation of CODA using PSPI [37]. We follow the same notation for the value functions, augmented MDP, and extended function classes as stated in Section and Section the main text.
A.1 Equivalence Relations between Original and Augmented MDP
We begin by showing that the optimal policy and any value function in the augmented MDP can be expressed using their analog in the original MDP. With the augmented MDP defined as M == (X, A, RG 7) in Section 4.1, I, we first define the value function in the augmented MDP. For a policy X A, we define the Q function for the augmented MDP as
00
(x,
En'
od R(, a) z, 00
Notice that we don't 't have a reaching time random variable in this definition; instead the agent would enter absorbing state st after taking at in the augmented MDP. We can define similarly (*(() = Q (x, R).
Remark A.1. Let Qh be the extension of Q based on R. have, for x & G. Qh(1, = Q (xx Va E and for x E G. Qh(x, a) Q² (z, a+)======= =1 Va A.
By the construction of the augmented MDP, it is obvious that the following is true.
Lemma A.2. Given X A(A), let ग be its extension. For any h X A R, it holds
Σ Enhhlz, = ET. -E.
00 En'
Eg.P
(x, & x+
1==
where T the goul-reaching time (random variable) and we define he (x. at) 11x,, m).
We relate the value functions between the two MDPs.
Proposition A.3. For policy X A(A), let ग. be its extension (defined above). We have for all z RE A.
(= a) 2 2* (z, (=(x)= (*(x)
Conversely, for a policy &: & A(A), define its restriction X and A by translating probability of originally on at be uniform over A. Then we have for all E S. A
Q=(x, a) 2 (5(x,a) a) VŠ(x) 2 (E(x)
Proof. The first direction follows from Lemma A.2. For the latter, whenever & takes a at some x & G. it has (<(x)= = 0 but V5(x) 2 0 since there no negative reward in the original MDP. By performing telescoping argument, we can derive the second claim.
By this lemma, we know the extension of x (i.e., .*) is also optimal to the augmented MDP and V*(x) V (x) for X. Furthermore, we have reduction that solve for the optimal policy in the original MDP by the solving augmented MDP, since
VÉ(()) - (do) V (d(o))
for all & : & + A(A). In particular,
Regret(s) == V* - V* - = (dn)==V+ (d)) - V- Regret
(3)
13

---

Since the augmented MDP replaces the random reaching time construction with an absorbing-stale version, the function Q" of the extended policy satisfies the Bellman equation
(0 R(2, a) PL (a', T)]
(x,
(4)
For X and E A, we show how the above equation can rewritten in Q" and R.
Proposition A... For X and E A
(x, = - (maa)(mmmaxa)))))maaa (c',,)))
For a = at, (*(xxat) = R(x, at) R((x). For x E Xt, Qz () 0.
Proof. The proof follows from Lemma A.5 and the definition of P.
Lemma A.5. F E X, (z, F) max ((()) (=, (I, x))
Proof. F
(= at), (x,
if E G otherwise
(x)
(Because of definition of πt)
(x,
i E G otherwise
(Because of Proposition A.3)
(x)
R(x,a'
x E G otherwise
(Definition of augmented MDP)
(T)
[(ar),
if G otherwise
(x, x),
max([l()), (2,
where in the last step we use R(x) 1 for x E G and R(x)) 0 otherwise.
A.2 Function Approximator Assumptions
In Theorem 5.4, we assume access to a policy class IT : A(A)). We also assume access
to function class == ( X' x A - [0, 1]} and a function class = [g X' - [0, 1)). We can think of them as approximators for the Q function and the reward function of the original MDP.
For an action value function f X x A [], define its extension:
g(x)); O, x xEX+
a= and X
a)
(5)
f(x,a)
otherwise.
The extension of is based state value function g XX' [0, 1] which determines the action value of z only at at. One could also view g(x) as a goal indicator: after taking at the agent would always transit the zero-reward absorbing state st, so g(x) R(I, at) which is the indicator of whether & E Ge.
Recall the zero-reware Bellman backup operator T with respect to P(s'ls, a) as defined in Assump- tion 5.2:
(", = FE m)]
where Pa(a' x, a) = P(s'|s, a)1(c' = c). Note this definition is different from the one with absorbing state s in Section 3. Using this modified backup operator, show that the following realizability assumption is true for the augmented MDP:
Proposition A.6 (Realizability). By Assumption 5.1 and Assumption 5.2, there is I E F and E G such that Q3 fg-
14

---

Proof. By Assumption 5.2, there is h E F such that h(x,a) max(R(x), (z( By Proposi-
tion A.4, have for E X. # a+
(x = 0 -' (maxa) |r (x))
=T E
For = a", we have Qa (a, (x) R(xx R(x) E g. Finally Q" 0 for x- E dt.. Therefore, Qa = for some E and g E g.
L
A.3 CODA+PSPI Algorithm
In this section, we describe the instantiation of PSPI with CODA in detail along with the necessary notation. The main theoretical result and its proof is then given in Section A.4. As discussed in Section 5, our algorithm is based on the idea of reduction, which turns the offline CGO problem into standard offline RI problem in the augmented MDP. To this end, construct augmented datasets Dayn and Deoal in Algorithm 1 as follows:
Dayn {Tin Ons nny 2 Hr 0, 2n (5i, Cj), (8 (y), -ai (8i, Dayn: (:Ccg) D goaa D goal {(In, at Tn, Fn
1n (Sn)Cn): Cn), xxt (s+ en), (Sn, Cn) Dooal
With this construction, we have: Daya ~ Moya(s, a, ('ga(a))()) and Deoal Mgoaa (c, 3)1 a = 8+). We use the notation, Muyy a, a, Mayn (8, 0, 8 (goa((c) and /' (I, a, a. = Agoal (c, s) = attasss = s+). We will also use the notation Tij = (5;, T'y = (+ 0ÿ) in the above construction. These two datasets have the standard tuple format, so we can run offline RL on Dayn p Also, note that Daynl = Dayn D Doaal and |geaal [Doaa|.
PSPI. We consider the information theoretic version of PSPI [37] which can be summarized as follows: For an MDP (X, R, P. 7)) given a tuple dataset = {(x, a, r, x²)), a policy class II, and a value class F, it finds the policy through solving the two-player game:
max min well fEF
(()..
S.t.
e(f. D)
min fi,, S PeF
(6)
1bb E (leanane The term fi D) ming' {(f', fin, D) in the constraint is an empirical estimation of the Bellman error on f with respect to т on the data distribution μ4, i.e. (f (r, a) T f(x R.. constrains the Bellman error to small, since Ex,a~n (I, a) Q² (r. a)) 0.
where f(do, TT Exg (z T)),
f' ('(x,,)))²
x))
CODA+PSPI. Below we show how to run PSPI to solve the augmented MDP with offline dataset
Dayn D To this end, we extend the policy class from to II, and the value class from F to Fg using the function class G based on the extensions defined in Section 4.1. One natural attempt is to implement equation with the extended policy and value classes II and F and D Day D goal- This would lead to the two player game:
max min REll f
(d),
s.t.
[ign D) min P.EFo
Lgg D) Se
However, equation 7 is not well-defined algorithm, because its usage of the extended policy स in the constraint requires knowledge of G. which is unknown to the agent.
Fortunately, we show that equation 7 can be slightly modified so that the implementation does not actually require knowing G. Here we use a property (Proposition A.4) that the Bellman equation of the augmented MDP:
@at (z, a) R((,, a) + - 1 (x', (x), =0 =0+ mmax R(x') Q" (x', x))]
for x E X and a + at, and (I, = for G and = a
We rewrite the squared Bellman error these two data distributions, Dayn and Deeat using the Bellman backup defined on the augmented MDP (see eq.4) as below:
Epon [(2" (x, -
-=*** (xx a)) = F [(@ (Ir, a) - 7E
P(1l,,0) - [max (R(x), Q=(x,, x)) =))))
15

---

Eg (l0 ( = Ex (2,at) - 1)
We can construct approximator fz, a) for Q (z, Substituting the estimator fz a) for Q* (az a) in the squared Bellman errors above and approximating them by finite samples, we derive the empirical losses below.
Layn (gi = Dayal (a,a,,,"') Dan
Σ
(f(2, a) (( ()))
(8)
Lyoal (s))
Σ
(g(x)) - 1)2
(9)
Dgoaa
(2,a,,, Dant
where we have (2, = f(x, (a * at) (a at) for x & X+
Using this loss, we define the two-player game of PSPI for the augmented MDP:
max min fo(do:
(10)
EE FoEF
s.t. (ayn(Jo: fgi min
Pg: S cdyn
Es
Egoal (fg) 0
Notice (do. #) = f'do, T). Therefore, this problem can be solved using samples from D without knowing G.
A.4 Analysis of CODA+PSPI
Covering number. We first define the covering number on the function classes F, G. and II", For F and G. we use the Loo metric. use (F, and Nox (9, to denote the their e-covering numbers. For II, we use the - metric, i.e., || T2lloo,1 suPzEx ( 72 (1()))l We use No (II, e) to denote its e-covering number.
High-probability events. In CODA+PSPI (eq. 10), choose the policy in class which has the best pessimistic value function estimate. In order to show this, we will need two high probability results (we defer their proofs to Section A.4.1). To that end, we will use the following notation for the expected value of the empirical losses:
paa (F Sgg (igitt)) R) = E(x,a,"") (f(x,
E gay (Fg) = E .et Pe (g(()) 1)²
First, we show that for any policy II, the true value function Q* satisfies the empirical constraints specified in eq. equation 10. Lemma A.7. With probability at least holds for all ı E II.
((V Dam
La Samn(Q"" (²))) min Pa (²)) eF
D
Dooacl
ggoat Q 50
where 10 log (M /9.1/ D Dow DN (0..
(Dwl
Forr finite function classes, the resulting performance guarantee will depend on [F], [9] and [1I] instead the covering numbers as stated in Theorem 5.4. "Technicalyy, we can remove Noc presentation.
G, | P in the upper bound, but we include it here for cleaner
16

---

(V
We use the notation Edyn
for the first upper bound in Lemma A.7.
Payal
[Padll
Next, show that for every pair of value function La F and policy & E II which satisfies the constraints eq. equation 10, the empirical estimates provide a bound on the population error with high probability.
Lemma A.8. For all E F9 E and π E II satisfying
(S fg:
min F
lamilgg, so: #) S edyr
gooal (Fg) S 0.
with probability at least 1 5, have:
()) 7E ((11,)()))))) [max(g(a') 5 f(x', T))l
00 Eddyn)
log Na (G. (/
11((x) 11l 50
V
Egoal
D Dognal
Pessimistic estimate. Our next step is show that the solution of the constrained optimization problem in equation 10 is pessimistic and that the amount of pessimism is bounded.
Lemma A.9. Given n, let fg denote the minimizer in equation 10. With high probability, Fg (do, R) S (da X)
Proof. By Lemma A.7, for any policy II, we know that O satisfies the constraints in equation equation 10. Therefore, we have
(d R) S Qh( #) =Q (do: T)-
We will now bound the amount of underestimation for the minimizer f7 in the above lemma. Lemma A.10. Suppose zo do is not in G almost surely. For any I1,
(do; R) -fg (do,
< E,
r max((" (1141), f (I)41 f (x,, (z), a.) + (R(())))
Note that in trajectory xv E G whereas in & for L < T by definition of
Proof.
Let Fr = (fr, ( be the empirical minimizer. By performance difference lemma, we can
write
= (1 (1- - (1 - (1- (1 (1-)))=111)p))))))))) म) (1
= E.z [R(z, a) + 7 (1', (x,
where with abuse of notation define d' (z, x') d' a) P a)) where d= (z. a) is the average state-action distribution of स in the augmented MDP.
In the above expectation, for x E G, we have = a and + (st c) after taking a at x = (s. c).
which leads to
M(I, a) + 7fg (2 T) Fg (z, R(x, a) 75. 7f
(x .) (x,aa) R(xx) (x)
For G and & x+, we have + at and x' $ x therefore
R(x,a) + tf 7) (x, a) = R(x) + 75g (x' 7) - (x,
S 7 ma (x'), f m) - (2,
17

---

where the last step is because of the definition of fg. For x' E X we have E X and the reward zero, so
f az) =0
Therefore, we can derive
(1-7)Q- (Io, m) - (1- ())
S E h max(g" ( x)) - (x, & G + (R(x) - (x)) E
Finally, using Lemma A.2 can have the final upper bound.
Main Result: Performance Bound. Let of be the learned policy and let f be the learned function approximators. For any comparator policy n, let fg (f g²) be the estimator of x on the data. We have.
v (do) - v= (do)
(do)
= (" π- Q" (do, π3
("o )
1z (do, #t) + (do,
(do, (d), - (do, Qd x1)
g
(d )
J (do: =t)
s (do, x) f (do; R)
< E.
"(y(" (xx()" (f (((41,) f (Iz, + + (B((T - 9" (IT))
< En
hh max(g" (2141), ft (I)+1; - f" (z ail --- R(IT) 900 (IT))
< Cdya (T)) Ib max(y"
(xx, + Egoal (T) Esoallll 11]
I Cayn (Eay + (T) VEgoll
where (aya(T) (T) and (T) are the concentrability coefficients defined in Definition 5.3.
Theorem A.11. Let nt denote the learned policy of CODA PSPI with datasets Dayn and D goal- {X' x [0, and G {X [0, 1]}. Under realizability
using value function classes and completeness assumptions as stated in Assumption 5. and Assumption 5.2 respectively, with probability 8, it holds, for any x II,
log (1.
(v
D Davel
( - ((r)) S Cayn
Egone (T)
D goaat
Dzout
log N=. (G (I.. (I.1/ DemillDaal) and Cdy((T)) and
where 0
(T) are concentrability coefficients which decrease the data coverage increases.
A.4.1 Proof of Lemmas A.12 and A.13
We first show the following complementary lemma where we use a concentration bound on the constructed datasets Däyn and D Lemmas A.7 and A.8 will follow deterministically from this main auxiliary result.
Lemma A.12. With probability least 5, anyaarayy/ any f. f. f2 E F and G, we have: Eeoe (f1, fg. F) Eam (f2.fa- laya(fi fg. л) + layn ft.
llfi f2ll
+
+
D goat
|Dayll
goal
D Deam
D poot
Dapall
(Nr.(F:1/1Dpmall) Do ( (Dmml)Ar..... (11.1/ (
where
log
18

---

Proof. Our proof is similar to proof of corresponding results in Xie et al. [37] (Lemma A.4) and Cheng al. [4] (Lemma 10) but we derive the result for the product distribution Payn = Mayn Myoal and its empirical approximation using Daya- Throughout this proof, we omit the bar on क as ldyn does not use the extended definition of the policy and further use M, N for the dataset sizes gDaall Dayal For any observed context (cy. E Dgoals we define the following quantity:
Pago (J, T) = )' ( m))) ((d)))((())))))))))
For conciseness, we use notation Taj for (s, Cj) and d'a for (s', (.) where (8, 8') is sampled from dynamies distribution and Cj D Dgoat. We first start with the following:
(Jr, Ja. Eaaga (S2, () Edyn(Si, Sgy + (2y
M
1 M
I pag (f1, Fg: x) - lagg fg:
1
Σ
ed
fgr
Σ M
Maya (f2, fg.
(11)
M
j=1
3=1
M
M Σ 1=1
+
Σ 1=1
Mam (fi. Fo.
Clann fg. Sg: + layalfa fg.
(12)
We will derive the final deviation bound by bounding each of these two empirical deviations in lines equation 11.equation 12. First, we will bound the line equation 11:
M
Ga fg.
Σ j=1
Maya (f2, fg. T)
1=1
Σ
Paya (J1, (g. - ej
(f2, fg.
Maya
M
F Paps
- [ m))) - (J2(Hoj,
max(o(z')),, x)))
M
Σ
Fam [[Ji a) f2(#os, ()) + f2( -2 [ +)))
M
F F(eaa))~uon
- + f2( a) 27 (g)os,)))
==
(13)
M
Σ
(felEoj, a) 170 {T))
Using similar argument, we can show that:
Lim (f1, fg, - Cana (f2, x) Ep (() 7 (s) -( ((),)(())))
(14)
Let Fe, G. be e-cover of F and G. and II, be e-cover of II, i... 3f1, E Fer 9 E GG. and
# II. such that || flloo. Ilfi filloo Ilfe felbo S and |
19

---

Then, for any f.f1,f2 E F. E II and their corresponding f,f, E Fo Ge, t E ll::
M
Lagul 1 fg.
Lag
1 M a=1
Σ
(Pi Mr (f2. Jz.
- - - (f2((s,c), (T
(a((o,, ( 0) f2( a) 27 1n)
M
4V 4V (M
2og
M
3M
where the first equation follows from eqs. equation 13 and equation 14, and the last inequality follows from Bernstein's 's inequality with a union bound over the classes Fe, Ges II, where is the variance term follows:
Var coeptaat
- [ckc:a/eiofiiisooooooooooooo00o0
E Eemppane
- (taeayywwlllllooooooodd/00000000 ((
< 4E
- (f((s,,))))-_((ss)))))__))))
where used that fact that f.g E [0. 1].
Thus, with probability
Lap (1i, lass (F2, Jg
Σ
e Mann (fi.
lg (F2. S (f2.
M
NNo (Ge.F.ooNs
NNo (F (II,))
< Fellasn log
2log
M
3M
Using the property of the set covers of F, F, II, we can easily conclude that:
Pap Fg. epp (fe. Fg.
1
Σ
el Aaga (f1. E (f2, Fg.
M
log
log
(NaolF.hNNas (G.CWWWWWW. (11.c)
$ 11 - f2llãsm"
M
M
(.. (G.e (II.))
log
+
+
(15)
M
Now, we bound the second deviation term in eq. line equation 12:
M
T)
Ep (f2. Fg. - (eayn(fi, - (syn(fesfg
M
1=1
M Epeg (Fi(Z))) - f(zleg m)))² (fz(2oj, a) y-1-mmmaaa max(g(4()))) feleg x)))²
1
.
-maax((la)))))) (falzey, a) max(o(=l))) f(1 x)) "] (16)
N
20

---

For any fixed Cj. using the same strategy as we used for bounding the first term in eq. line equation 11, for any f. fI. f2 F. 9, II and their corresponding f, f1 f2 Fer Ges f II with probability at least 1 6:
( fg.
P (f2, (g.
(fam
NN (Na(Fo)aalGeNN
log (Na( (Gre) (0 (I..e)
log
- Ilfi
fal fallmaxxtes
N
N
(F, (G.e) (II,e)
log
+
N
We can now consider the sum in the second term in eq. line equation 12 for f. f1, S2, G. ā as:
M
e Maga (o. -
Cj (e Fo. - (Eaya fo. - fsyn(f2.f..
M
(F, (c.
log
Ballasxxtest
M
N
j=1
(No(Fio (G.
log
log
+
+el
+
N
N
(9.
M
log
1
fellmonx(es) - 11 Fellas
M
N
1=1
No. (Gr (TT.))
log
+
11
F2llpom
NNa(FiN (G.
No. (G.N
log
log
+
+e
(G. ((T,))
(G.
log
log
2 11 - S2llagn1
VNM
log (No(F (N (o)
Ma
log
+
tc
+
(17)
where the last inequality follows from Hoeffding's inequality. We can now bound the in eq. line equation 12 as:
M
Auga (Ji. (g. pnga (J2,
( fo: - (Ji layn (J.
M
jml
(Noo MosFFeNeelGeeNW.nntt
log
log
$ 1
- Fzllagn"
N
VNM
Nar(F.cWa (G..N.
No. (G
log
log
+
+
+
(18)
N
N
21

---

Combining eqs. equation 15 and equation 18 with e = O MN ) we get the final result.
Lemma A.13. With probability at least - 6, for any 9. 92 E 9 and f & F, we have: Pao (f)) ( -goo (f & (Fo2)
(foz)
log NN. D 8
/
log
00 I191 92 gellpowr
D Dgoat
[ Dooa
Proof. This result can be proven using the same arguments as used in Lemma A.12 using covering argument just over g.
Using these two main concentration results, can now prove Lemmas A.7 and A.8.
Proof of Lemma A.7. Note Q' S, for some f E F and g E g (Proposition A.6) and
E E = E م (1) - (0( (x', +T))
a) - (I,
The lemma can now be proved by following similar proof of Theorem A.I of Xie al. [37]. The key difference is the use of our concentration bounds in Lemmas A. and A.13 instead of Lemma A.4 in the proof of Xie al. [37]. On the other hand, (goa((g)
because the reward R(x) is
deterministic which results in the second inequality.
Proof of Lemma A.8. This result can again be proved using the same steps as in Lemma A.5 from Xie et al. [37] based on the concentration bound in Lemmas A.12 and A.13.
B Experimental Details
B.1 Hyperparameters and Experimental Settings
IQL. For IQL,, we keep the hyperparameter and tune other hyperparameters on the antmaze-mediung antmaze-mediun--lly--22 environment and choose batch size 1024 from candidate choices (256, 512, 1024, 2046), learning rate 10-1 from candidate choices (5 10-5 10 10-4,3 10 10=4) and 3 layer MLP with RuLU activating and 256 hidden units for all networks. We use the same set of IQL hyperparameters for both our methods and all the baseline methods included in Section 6.2 and apply to all environments. In the experiments, we follow the convention of the -1/0 reward in the IQL implementation for Antmaze, which can be shown to be the same as the 0/1 reward notion in terms of ranking policies under the discounted MDP setting.
0.99, = 0.9, 10.0, and a 0.005 in [18],
Reward Prediction (RP). For naive reward prediction, we use the full context-goal dataset as positive data, and train reward model with 3-layer MIP and ReLU activations, learning rate 10-1 batch size = 1024, and training for 100 epochs for convergence. To label the transition dataset, we need to find some appropriate threshold to label states predicted as goals given contexts. We choose the percentile as 5% in the reward distribution evaluated by the context-goal set as the threshold to label goals (if reward is larger than the threshold than it is labeled terminal), from candidate choices (0%, 5%, 10%). J. Then we apply it to all environments. Another trick we apply for the reward prediction is that instead of predicting 0 for the context-goal dataset, we let it predict but shift the reward prediction by -1 during reward evaluation, which prevents the model from learning all weights. Similar tricks are also used in other reward learning baselines.
UDS+RP. We use the same structure and training procedure for the reward model as RP, except that we also randomly sample minibatch of "negative" contextual transitions with the same batch size for balanced distribution, which is constructed by randomly sampling combinations of a state in the trajectory-oni distribution of positive and negative samples, we sample from each dataset with equal probability. For the threshold, we choose the percentile as 5% in the reward distribution evaluated by the context-goal set as the threshold label goals the antmaze-medium maddiuu--lly-2222 environment, from candidate choices (0%. 5%, 10%). ). Then apply it to all environments.
dataset and context from the context-goal dataset. To create a balanced
22

---

PDS. We use the same structure and training procedure for the reward model as RP, except that we train an ensemble of 10 networks as in [14]. To select the threshold percentile and the pessimistic weight k; we choose the percentile as 15% in the reward distribution evaluated by the context-goal set as the threshold label goals from candidate choices (0%, 5%, 10%, 15%, 20% ), and k = 15 from the candidate choices (5,10,15,20) in the antmaze-meddim-playe environment. Then we apply them to all environments.
CODA (ours). We do not require extra parameters other than the possibility of sampling from the real and fake transitions. Intuitively, we should sample from both datasets with the same probability create an overall balanced distribution. We ran additional experiments to study the effect of this sampling ratio hyperparameter: ratio of samples from the context-goal dataset D Dgaal to total samples in each minibatch. Table 5 shows that CODA well long as the ratio roughly balanced in sampling from both dataset.
Compute Resources. For all methods, each training run takes about 8h on a NVIDIA T4 GPU.
Table 5: Average success rate (%) in AntMaze-v2 from all environments, with different sampling ratios from the context- goal dataset.
EnwRatio umaze
0.1
0.3
0.5
0.7
0.9
91.6+1.3
92.4#1.0
94.8+1.3
8,
84.8+3.0
umaze diverse
76.8+1.9
79.2+1.6
72.8=7.7
76.622.3
65.448.8
medium play
82.342.1
85.0+1.8
75.8+1.9
7..8
76,621.3
medium diverse
79.4+1.6
76.6+3.0
84.5=5.2
75.6+2.0
72.0+3.5
large play
50.8+2.0
45.2+3.7
60.0+7.6
43.6+2.3
46.6+2.3
large diverse
35.8+5.7
37.4+4.7
36.8+6.9
34.4+2.4
27.0+2.1
average
69.5
68.9
70.8
64.9
62.1
B.2 Context-Goal dataset Construction and Environmental Evaluation.
Here we introduce the context-goal dataset in the three levels of context-goal setup mentioned in Section 6 and how to evaluate in each setup. We also include our code implementation for reference.
Original Antmaze. We extract the 2D locations from the states in the trajectory dataset with terminal=True the context (in original antmaze, it suffices to reach the Lg ball with radius 0.5 around the center), where the contexts are distributed very closely as visualized Figure 2(a), and the corresponding states serve the goal examples with Gaussian perturbations ((, 0.05) on the dimensions other than the 2D) location.
Four Rooms. For each maze map, we partition rooms like Figure 2(b) and use the room number as the context. To construct goal examples, we create copy of all states in the trajectory dataset, perturb the states in the copy by [0, 0.05) on each dimension, and then randomly select the states (up to 20K) according to the room partition.
Random Cells. For each maze map, we construct range of non-wall 2D locations in the maze map and uniformly sample from it to get the training contexts. To construct the goal set given context, we randomly sample up to 20K states with the 2D locations within the Le ball with radius 2. Figure 2(C) a intuitive visualization of the corresponding context-goal sets. For test distributions, we have two settings: 1) the same as the training distribution; 2) test contexts are drawn from limited area that is far away from the starting point of the agent.
Evaluation. We follow the conventional evaluation procedure [18), where the success rate normalized to be 0-100 and evaluated with 100 trajectories. We report the result with standard ervor across 5 random seeds. The oracle condition we define in each context-goal setup is used to evaluate whether the agent has successfully reached the goal and also defines the termination of episode.
23

---

B.3 Reward Model Evaluation
For reward learning baselines, we evaluate the learned reward model to showcase whether the learned reward function can successfully capture context-goad relationships.
Evaluation dataset construction. We construct the positive dataset from context-goal examples, and the negative dataset from the combination of the context set and all states in the trajectory-only data, then use the oracle context-goal definition in each setup filter out the positive ones. We then evaluate the predicted reward on both positive and negative datasets, generating boxplots to visualize the distributions the predicted reward for both datasets.
Results. Here present boxplots for reward models with experimental setups in Section 6.3. Overall we observe that PDS+RP is consistently better at separating positive and negative distributions than UDS and naive reward prediction. However, PDS still fail fully separating positive and negative examples.
-34
(a) PDS
UDS+RP
(c) RP
Figure 3: Reward model evaluation for the large-diverse dataset for original AntMaze environment. Green dots are outliers.
14
010
-35
(a) PDS
UDS+RP
RP
Figure 4: Reward model evaluation for the medium-divedd diverse dataset for the original AntMaze environ- ment. Green dots are outliers.
-32
-34
(a) PDS
(b) UDS+RP
(c) RP
Figure 5: Reward model evaluation for the umaze-diverse dataset for the original AntMaze environ- ment. Green dots are outliers.

---

-10
(a) Large, PDS
(b) Large. UDS+RP
(c) Medium, PDS
(d) Medium, UDS+RP
Figure Reward model evaluation for the Four Rooms environment. Green dots are outliers.
(a) Large, PDS
(b) Large, UDS+RP
(c) Medium, PDS
(d) Medium, UDS+RP
Figure 7: Reward evaluation for Random Cells environment (the test context distribution is the same as training). Green dots outliers.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All the theoretical results and experimental findings referenced in the abstract and introduction are included in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have added limitations section (see Section 6.4).
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
26

---

Justification:
We include the full detailed proof of our theoretical result in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Our experiments are based on standard RL domains and we discuss all the modifications that we consider for our setup in detail. Further, all algorithmic details and hyperparameters are also discussed.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
27

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provide code in supplementary files.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips) cc/
public/ /wwic/ggg/ees//eee/eee Sod.edee///odSuuimiiic sssisssinnsn sii.oooiiisooooiiyyyccocoomoommc for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / is.ps. cc/pcc/pll....ccccccccccc c/gu des / /CdeeSuddnnn ss onPol ccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We provide all the necessary details in the main paper (Section 6) with additional information in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report the standard errors across multiple runs for all our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
28

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide compute resource in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttpa:////uriippe cc http://ccccccpcctccccccccaaaaaaa ines?
Answer: [Yes]
Justification: Based on the NeurIPS code of ethics, we do not see any direct ethical or societal impact considerations for our work.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Based on the NeurIPS code of ethics, we do not see any direct ethical or societal impact considerations for our work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
29

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: These safeguard concerns do not apply to our experimental domains.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We include the information in the supplementary files.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use package should be provided. For popular datasets, papersw's thoode. com / datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
30

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Not applicable this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing or human subjects involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human subjects crowd-soureing involved in experiments for this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "Sw1qnArC87", "reviewer": "Reviewer_CzDa", "summary": "This paper proposes a simple action-augmented MDP formulation for contextual goal-oriented problems in an offline RL setting. They show that their action-augmented MDP has a regret that is equivalent to the original MDP and any policy can be converted interchangeably without changing the regret. Along with theoretical justification, they also show that better performance compared to other reward prediction and goal-oriented RL methods in the experiments.", "questions": "- I am curious as to why is there a wide performance variance in Tables 1,2,3 between different env or method. Specifically, why is CODA significantly outperforming baselines in few settings while only providing marginal benefits in few others.\n\n- The poor performance of goal-oriented RL methods in basic setting in Table 1 is also surprising, I request the authors to add more insights into this either in the rebuttal or in the final version.", "limitations": "The significant limitations have been delineated well in Sec 6.4.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces and solves an interesting and challenging problem of context-defined goal-oriented RL policy learning, and do so without having access to labeled samples for the task.\n- The authors propose a very effective and clever solution for converting existing dynamics and context-goal datasets to first create learning data, and then create augmented MDP policies while establishing learnable guarantees using only positive data (Th 5.4). \n- The empirical performance is strong and generally outperforms other baselines.", "weaknesses": "- I am unsure how well the theoretical assumptions and setting of the paper transfers to more real world and larger-scale environments.\n\n- The empirical evaluation has been only performed using a single environment, so the generalizability of the method to more diverse settings is unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable comments. We found them extremely helpful in improving our draft. We address each comment in detail below.\n\n### 1. “How well the theoretical assumptions and setting of the paper transfers to real world”:\n\nFor the assumptions, our assumption **5.1** and **5.2** are general assumptions about the expressiveness of the value functions which are standard in offline RL theories. In plain words, we need the value function approximators to be expressive enough to effectively perform approximate dynamic programming in offline RL.\n\nFor the offline CGO setup, we assume a dataset of dynamics data and a dataset of context-goal examples, which are **commonly available**. Notice the dynamics data is task agnostic and the context-goal examples do not require expert demonstrations. For example, in the example in Introduction where we instruct the truck to feasible warehouses, we only need the logged data of truck driving without any labels (dynamics data), and a set of instructions (contexts) and the corresponding warehouses (goals). We don’t need the expert trajectories of how the trucks can be navigated under different instructions (contexts) which allows commonly available dataset to be used in our setup.\n\n### 2. “The generalizability of the method to more diverse settings is unclear”: \nWhile all the experiments are done using the AntMaze simulators, we specifically design each experiment to reflect the **diverse scenarios that offline CGO problems can cover**. In **Section 6 (L265)** and **Fig 2**, we propose a taxonomy of offline CGO problems based on context-goal relationship with an increasing complexity, and we design experiments to cover all these scenarios. Moreover, for each scenario, we use various dynamics datasets and different mazes. The experimental results show the proposed algorithm can consistently solve all these different scenarios.\nWe used the same AntMaze domain throughout these experiments so that we can study – through controlled experiments – the effects due to different data relationships in different CGO settings. If we were to implement one setup from Fig 2 in one domain and another setup from Fig 2 in another one, there may be additional confounders due to the domain change. We hope these results provide good evidence to show the generalizability of our method to different CGO settings. Nonetheless, we acknowledge experiments with more realistic datasets and environments would be interesting future work.\n\n### 3. “Why is there a wide performance variance”: \nWe want to first highlight that **Table 1, 2, 3** are from **different context-goal relationships**. Despite being implemented via the same simulator (of the same dynamics), they mean totally different learning problems, so they should not be directly compared. We should only compare algorithms within the same table, not across tables. \nFor each table, we provide the results of training with oracle reward, which is the skyline reference performance with respect to each dynamics dataset. The difference between that skyline and the other baselines is the room for improvement. Sometimes the headroom can be small, e.g., some coverage assumption is met like all goals in the dataset are feasible or the context-state reward can be easily learned from the dataset. In these tables we see that our proposed approach gives consistent improvement over baselines whenever possible. We will better clarify how the tables should be interpreted in the revision.\n\n\n### 4. “Poor performance of goal prediction baseline in Table 1”: \nThe reason is that as shown in **Figure 2(a)**, the goal area for this setup is very small, thus the number of goal examples in the context-goal dataset is very limited, thus providing difficulty for learning the goal distribution given context for goal prediction methods since it needs to learn to generate the goals first and then sample the goal as the condition.\n\nWe hope our response resolves all the concerns in your review and please feel free to let us know if you have any other questions in the discussion period.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y5osA0nWPR", "reviewer": "Reviewer_ek4e", "summary": "The paper combines an (unlabeled) dynamics dataset of trajectories, and a (labeled) context-goal dataset in an offline setting to create a combined MDP(Markov Decision Process). They do it by augmenting the dynamic dataset to have fallacious action to the terminal state with reward 1 on goal states given context c. All other states have a reward of 0. They also give a theoretical proof of their methodology.", "questions": "See the weakness section", "limitations": "NA", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "The paper presents a general solution that can be applied to a wide range of problems within the field; the findings and methods have broad applicability, increasing their relevance and potential impact across various contexts.\n\nThe authors have validated their methodology through comprehensive proofs and empirical evidence (to some extent). This thorough validation provides a solid foundation.", "weaknesses": "-Limited novelty, the paper provides a way to combine two types of datasets, but the methodology it provides is trivial\n\n-Paper does provide a mathematical proof of the technique, but the technique itself is straightforward enough that the proof is trivial given that it builds on baseline proof\n\n-Not adequate experimentation\n\n-Experimentation on a very simple problem set\n\t\n-The paper needs more polishing (e.g., MDP acronym used in the abstract without initializing,  Line 8: \"outperform\" should be \"outperforms\"; Line 177: \"fictious\" should be \"fictitious\". etc.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. We address each comment below.\n\n### 1. Limited novelty: \nCould you provide specific reasons why the novelty is limited? We respectfully disagree that our method is trivial since we carefully design the augmented MDP structure and the data augmentation method such that 1) we do not need to deal with the challenges of predicting the feasible goals in this augmented MDP and 2) we do not need to handle the missing context label problem in this augmented MDP. This is why this augmentation method is effective despite its simplicity: **it fully makes use of the CGO structure of the problem to circumvent the drawbacks in other baseline methods. Simplicity is not a drawback and does not equal lack of novelty.**\n\n### 2. Proof is trivial: \nA major aspect of our proposed algorithm is the reduction of the CGO problem to an offline RL problem. While the overall template of the proof will naturally follow the proof structure of the underlying base offline RL analysis, our proof is based on the **novelly designed augmented MDP** (which is a goal-conditioned MDP). This step is critical, as **no existing proof can show the effectiveness of the offline goal-conditioned RL if only positive data (context-goal pairs in our case) is available**, which is our offline CGO setup. In order to prove our results, **we use a careful reformulation of the Bellman equation of the augmented MDP and construct an augmented value function and policy class in the analysis using the CGO structure**. As such, we respectfully disagree that the proof is trivial.\n\n### 3. Not adequate experiments: \nCould you specify the reasons? In our environments, we use different dynamics datasets in different mazes, and for each maze we have three different context-goal setups representing the different CGO setups in the spectrum discussed in **Section 6 (L265)**. We show that our method works well and consistently outperforms the baseline methods in these different scenarios. Moreover, the main goal of this paper is to formulate the CGO setup since it is not formally studied in the literature, and as the first milestone to show it can be solved with commonly available offline datasets, which the proposed method demonstrates.\n\n### 4. Typos: \nThanks for pointing them out, and we will fix them in the revised draft.\n\nWe hope our response resolves all the concerns in your review and please feel free to let us know if you have any other questions in the discussion period. If our responses have addressed your concerns, please kindly consider raising the score. Thank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "buIriG0Fqa", "reviewer": "Reviewer_Kh9A", "summary": "This paper focuses on a new RL task: Contextual Goal-Oriented Offline RL task. This task considers an offline goal-context pair dataset and an unsupervised transition dataset. To address this task, in this paper, Contextual goal-Oriented Data Augmentation (CODA) is proposed to augment a new state $s^+$ and new action $a^+$ representing the goal within the context is achieved with reward as 1. The theoretical analysis is proposed to demonstrate CODA's capability to solve CGO problems.", "questions": "Some questions are listed in the Weaknesses.\n\nThe final score will be updated when these questions are answered.", "limitations": "There is no clear negative societal impact.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) This work focuses on a new task where goal and context are related. I am not familiar with the literature of Contextual Goal-Oriented (CGO).  Personally, I think this task is interesting. \n\n2) CODA methods show good performance on empirical evaluations.", "weaknesses": "1) In the Related Work section, some methods to solve Goal-conditioned RL are listed. Can these methods be used to solve this problem? If yes, how do these methods perform in the empirical studies?  What is the relation between these methods and the used baseline methods?\n\n2)  This paper proposes a data augmentation method. Can this method be incorporated with other Goal-conditioned RL approaches to achieve better performance, such as relabeling? \n\n3) The proposed data augmentation method is straightforward. It uses goal and context to label states and actions. Why such data augmentation can achieve such impressive results? More discussions are expected. \n\n4) More datasets and real-world scenarios can be used to further demonstrate the effectiveness of this method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable comments. We found them extremely helpful in improving our draft. We address each comment below.\n\n### 1. “Can goal conditioned methods be used to solve this problem”: \n\nThere is no straight-forward way to directly apply typical goal-conditioned methods to our offline CGO setup in general, since the relationship between contexts and goals is **unknown** here. On the other hand, the traditional GO setting (equivalently a CGO special case where context = goal) essentially assumes the relationship between contexts and goals is **known**; and algorithms like Hindsight Experience Replay (HER) critically relies on this assumption for the relabeling.\n\nNonetheless, we still try to show how goal-conditioned RL methods might perform in this more general setting. We do this by incorporating a goal-prediction baseline, which first learns to predict goals given a context using the context-goal dataset, and then uses the goal-conditioned policy learned from the dynamics-only dataset with HER (L45). We presented the result of the “**Goal Prediction**” baseline in **Table 1, 2, 3**. \n\n### 2. “Can this method be incorporated with other Goal-conditioned RL approaches to achieve better performance, such as relabeling”: \nAs mentioned above, due to the missing relationship between goals and contexts, goal-conditioned RL methods cannot be directly applied. We do highlight HER is used in our Goal Prediction baseline to learn the goal-conditioned policy (**L307**). \n\nThere is no way to directly relabel contexts in our offline CGO setup since the context is not available in the dynamics dataset, and we only have a fixed and limited context-goal dataset. Similarly, for other goal conditioned RL approaches, we cannot directly use that due to missing context labels for the states in the dynamics dataset.\n\n### 3. Why such data augmentation can achieve such impressive results?\n\nAs discussed in our introduction, the baseline methods all have their drawbacks (**L44**): For the goal prediction methods, the predicted goal might not always be feasible given the initial state. For the reward prediction baseline, it does not make good use of the goal-oriented nature of the CGO problems, and it might be challenging to infer reasonable reward for any context-goal pair, given only with positive data is available during training (context-goal dataset). \n\nOn the other hand, in CODA, we carefully design the **augmented MDP** structure which our data augmentation method relies on. In this way, 1) **we do not need to deal with the challenges of predicting the feasible goals** in this augmented MDP like goal prediction methods and 2) **we do not need to handle the missing context label problem** in this augmented MDP. Notice that this augmented MDP **equivalent to the original MDP**. This is why this augmentation method is effective despite its simplicity: **it fully makes use of the CGO structure of the problem to circumvent the drawbacks in other baseline methods**.\n\n### 4. More datasets and real-world scenarios: \n\nWe acknowledge that having more datasets would be better. However, our main goal of this paper is to formulate the CGO setup since it hasn’t been formally studied in the literature, and to show it can be probably solved with commonly available offline datasets, as demonstrated by the proposed method. Our controlled experiments, although limited to the same simulator, are designed to cover the spectrum of different CGO setups listed in the taxonomy given in **Section 6 (L265)**. We think this is a sufficient first milestone to showcase the effectiveness of the proposed method. Testing on real world datasets is important future work but is out of the current scope of this paper.\n\nWe hope our response resolves all the concerns in your review and please feel free to let us know if you have further questions. If our responses have addressed your concerns, please kindly consider raising the score. Thank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XGcf6zUL5g", "reviewer": "Reviewer_ug4M", "summary": "This paper proposes a novel method to solve CGO problems and proves CODA can learn near-optimal policies without the need for negative labels with natural assumptions. In addition, sufficient experiments prove the effectiveness of the proposed method.", "questions": "None", "limitations": "None", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The contribution of the proposed method is interesting.\n2. Sufficient experiments prove the effectiveness of the proposed method.\n3. Theoretical analysis is sufficient.", "weaknesses": "None", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review and the acknowledgement.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper received the following scores: R1: 7 - Accept , R2: 4 - Borderline reject, R3: 5 - Borderline accept 
R4: 7 - Accept (with R4 being the least credible and should be discarded). 

Overall, reviewers highlight the interesting and challenging problem statement, the fact that the paper presents a general solution, as well as the good performance on empirical evaluations.

On the negative side, especially R2 points out that there is limited novelty, that the technique itself is straightforward, as well as limited experimentation on a very simple problem set. 

Overall, I tend to agree with R2 for the following reasons: While the idea is interesting and designed for a general setup, the evaluation is limited to a single scenario. So, the evaluation does not really help to understand how generalizable the setup is, e.g., by evaluating a second scenario. 

Usually, I would highly recommend the authors to improve the paper in this direction and extend the evaluation, probably in supplementary, to more setups and let the paper pass. In this case, the authors clearly stated that they do not intend to do this. We therefore had an intense discussion including SACs, and the decision was made to accept the paper. I hope that authors take the advice of R2 and AC serious, and strive for an improved CR version to strengthen their submission.
Decision: Accept (poster)