Offline RL with Discrete Proxy Representations for Generalizability in POMDPs
Authors: Pengjie Gu
Xinyu Cai
Dong Xing
Xinrun Wang
Mengchen Zhao
Bo An
Venue: NeurIPS 2023 poster (2023)
Keywords: Offline RL
POMDP

Abstract: Offline Reinforcement Learning (RL) has demonstrated promising results in various applications by learning policies from previously collected datasets, reducing the need for online exploration and interactions. However, real-world scenarios usually involve partial observability, which brings crucial challenges of the deployment of offline RL methods: i) the policy trained on data with full observability is not robust against the masked observations during execution, and ii) the information of which parts of observations are masked is usually unknown during training. In order to address these challenges, we present Offline RL with DiscrEte pRoxy representations (ORDER), a probabilistic framework which leverages novel state representations to improve the robustness against diverse masked observabilities. Specifically, we propose a discrete representation of the states and use a proxy representation to recover the states from masked partial observable trajectories. The training of ORDER can be compactly described as the following three steps. i) Learning the discrete state representations on data with full observations, ii) Training the decision module based on the discrete representations, and iii) Training the proxy discrete representations on the data with various partial observations, aligning with the discrete representations. We conduct extensive experiments to evaluate ORDER, showcasing its effectiveness in offline RL for diverse partially observable scenarios and highlighting the significance of discrete proxy representations in  generalization performance.
ORDER is a flexible framework to employ any offline RL algorithms and we hope that ORDER can pave the way for the deployment of RL policy against various partial observabilities in the real world.

Offline RL with Discrete Proxy Representations for Generalizability in POMDPs
Pengjie Gu l,t, Xinyu Cai 1st, Dong Xing², Xinrun Wang 1.1, Mengehen Zhao 3.1 Bo An' School of Computer Science and Engineering, Nanyang Technological University, Singapore College of Computer Science and Technology, Zhejiang University Noah's Ark Lab, Huawei
{pengj gu, xinyu. caiy xinrun. wang, boan) (ttpl//tttuuuu....... edu.sg,
dongxing&zju. edu. edu.en, en, zhaomengchennhuuavii.. com com
Abstract
Offline Reinforcement Learning (RL) has demonstrated promising results in various applications by learning policies from previously collected datasets, reducing the need for online exploration and interactions. However, real-world scenarios usually involve partial observability, which brings crucial challenges of the deployment of offline RL methods: i) the policy trained on data with full observability not robust against the masked observations during execution, and ii) the information of which parts of observations are masked is usually unknown during training. In order to address these challenges, we present Offline RL with DiscrEte pRoxy representations (ORDER), a probabilistic framework which leverages novel state representations to improve the robustness against diverse masked observabilities. Specifically, we propose a discrete representation of the states and use proxy representation to recover the states from masked partial observable trajectories. The training of ORDER can be compactly described as the following three steps. i) Learning the discrete state representations on data with full observations, ii) Train- ing the decision module based on the discrete representations, and iii) Training the proxy discrete representations the data with various partial observations, align- ing with the discrete representations. We conduct extensive experiments to evaluate ORDER, showcasing its effectiveness in offline RL for diverse partially observable scenarios and highlighting the significance of discrete proxy representations in generalization performance. ORDER is flexible framework to employ any offline RL algorithms and we hope that ORDER can pave the way for the deployment of RL policy against various partial observabilities in the real world.
1
Introduction
The past decade witnesses the remarkable success Reinforcement Learning (RL. various domains, such as Atari games [38] Go 43 and even nuclear fusion [10. Offline RI,, particularly promising approach, focuses on learning effective policies from previously collected datasets, reducing the need for online exploration and interactions [34]. However, the various partial observabilities in the real world pose challenges when deploying the learned policy in practice. Examples include autonomous cars facing blocked or perturbed observations [1I], drones navigating through cluttered environments with occlusions [5T]. and robotic manipulators handling objects with varying degrees of visibility (5). As result, agents trained offline datasets must adapt their learned policies to cope with dynamic, uncertain, and partially observable environments, ensuring generalizability in real-world applications.
1 Equal contribution.
2 Corresponding authors.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Our research targets the situations where agents, trained on offline datasets with full state information, encounter various partial observations in real-world deployments. For instance, robot trained in a lab with complete data on obstacles and victims might face limited visibility due to occlusions, sensor noise, or environmental factors like smoke or varying light in real scenarios. This contrast between training and real-world conditions can hinder the system's 's generalizability and effectiveness.
There are two main challenges for the generalization against partial observabilities. i) During testing, the partial observability may differ substantially from the data on which the RL policy was trained, leading to significantly reduced performance. ii) The specific forms of partial observability remain unidentified during training, preventing the tailored training of agents for these observabilities. These challenges can be understood as the generalization issues within POMDPs. In this paper, our focus narrows to the generalization of a collection of POMDPs arising from an inherent MDP via diverse masked observation functions, which we refer to as a POMDP family.
Existing offline RL methods primarily target fully observable environments with comprehensive state data, causing difficulties in handling real-world scenarios involving partial observability and uncertain observation functions (30) 31 29 13, (28 8 [5
26 7 35 While POMDP methods excel in online RL, they rely on continuous environment interactions for policy adaptation, making them unsuitable 40). Furthermore, most of these methods are developed
for offline RL 14 47 for specific POMDPs and thus lack the ability to generalize across different and unknown partial observation functions. Although some attempts have been made to address partial observability in offline RL 177, their effectiveness is constrained by restrictive assumptions, such as a known observation function or a focus on theoretical analysis.
9 21 36, 20
22 32
To bridge the research gap, we unveil Offline RL with DiscrEte pRoxy representation (ORDER), a framework that leverages discrete proxy state representations for diverse observation functions. Drawing an analogy, consider mapping vast terrain: A detailed map is challenging to recall, but broadly categorizing into zones simplifies navigation. Similarly, the discrete representation in ORDER categorizes states, aiding RL algorithms. Moreover, even when part of the terrain (state) is obscured, a knowledgeable guide (proxy representation) can infer the underlying area, Our contributions are three-fold. Firstly. we propose discrete representation of states to improve the robustness against the partial observability and proxy representation to recover the discrete representation of the state from partial observable trajectory. Secondly, we propose three-stage training of ORDER: i) we first train the discrete representation on data with full observation through Vector Quantized AutoEncoder (VQ-VAE 45)), ii) we train the decision module based on the discrete state, and iii) we train the proxy discrete representation on the data with various partial observations, aligning with the discrete representation. Finally, we present a thorough evaluation of ORDER, showcasing its effectiveness in enhancing offline RL algorithms for various partially observable scenarios. Our experiments highlight the importance of discrete proxy representations in maintaining alignment under partial observability, leading improved generalization performance. ORDER is flexible framework to employ any state-of- oftth--rt (SoTA) offline RL algorithms and we hope ORDER can pave the way for the deployment of RL policy against various partial observabilities in the real world.
2
Related Works
Offfine RL. Offfine RL aims to learn policies from pre-collected transition data, eliminating the need for active data collection (34). In recent years, this area has seen significant growth in research g Both empirical and theoretical studies have identified overestimation error, caused by out- -itt----iittibutionn states and actions, as a critical challenge in offline RI [13). As a result, numerous behavioral regularizations have been proposed to keep the learned policy within the bounds of the offline data during RL training [15 26 7 335. However, the majority of cxisting methods focus on fully observable environments, limiting their applicability in real- world situations that often involve partial observability and uncertain observation functions [31 13 28 8 26). In such cases, the partial observability encountered during execution can greatly differ from the data on which the RL policy is trained, leading to a significant decline in performance. A noteworthy attempt to tackle partial observability in offline RL is the work by [17. However, this effort mainly centers on theoretical analysis and assumes a known observation function, which restricts its practical application. Our work sets itself apart from this previous effort by assuming access to full observations in the offline dataset, while the observation functions in the testing environment are unknown, potentially diverse.
30
29

---

and subject to change. Our approach is designed to generalize to uncertain observation functions, thus improving generalization performance across range of partially observable scenarios.
Online RL for POMDPs. Existing work addressing POMDPs generally falls into two main categories: i) belief state methods [27 14
21), which aim to deduce the distribution of hidden (belief) states from partial observation histories and then apply standard RL techniques to these inferred states; and ii) memory-based policies that use memory-based architectures, like recurrent neural networks (RNNs), to store observation trajectories and summarize past experiences 24 23
44 3 50 POMDPs and lack the ability to generalize across different and unknown partial observation functions. Furthermore, these online techniques hinge on the agent's 's capacity to interact with the environment and adjust its policy online, which is unsuitable for the offline RL setting. Another relevant area is asymmetric reinforcement learning, where existing works assume that agents can access privileged information (e.g., full observations) during training. but need to function in testing environments with partial observations [39
37 36
49 [9]. However, most of these methods are developed for specific
33 Nonetheless, they also assume a specific partial observation and depend on data collection through environment interaction. In contrast, our method presumes that we can access full observation information from fixed dataset without interacting with the environment. Additionally, our method does not focus on a single, fixed partial observation function; it is designed to effectively generalize across a variety of masked observation functions. This adaptability allows our approach to address real-world situations characterized by diverse and uncertain observation functions.
48
2
16
Sim-to-Real RL.
Sim-to-real RL is notable direction in related works. This approach leverages simulation training for real-world applications, adapting policies based on transferred knowledge
[52_ A prevalent strategy in this realm is domain randomization 44]. Rather than meticulously replicating real-worke parameters, the environment is extensively randomized to encapsulate the bias. Additionally, many researchers
genuine data distribution, regardless of simulation-realiy adopt domain adaptation, which capitalizes on source domain data to enhance model performance in a target domain, typically data-scarce [
18 25 A key challenge here lies in unifying the often-divergent feature spaces between source and target domains to facilitate knowledge transfer effectively. Our work shares thematic connection with the sim-to-real paradigm, as both address the transition from controlled conditions to real-world unpredictability. In our framework, agents, trained comprehensively offline datasets, must adapt to partial real-world observations. While sim-to-real focuses on bridging the gap between simulation and reality, our emphasis is on transitioning from thorough training data to fragmented real-world observations. Both aim for real- proficiency, but the challenges and remedies they explore differ substantially.
3 Preliminary
POMDPs. A Partially Observable Markov Decision Process (POMDP) is mathematical frame- work used to model peeiiiionmaaiing problems in uncertain environments [41]. Formally, a POMDP defined as a tuple (S, A. O, T, So, O, Oo, R, 7). where the underlying process Decision Process (MDP) (S, A. T, So. R, H, 7). Concretely, S represents the set of states, A denotes the set of actions, T S A x S -> [0. 1] is the transition function (dynamics), S, : - [o. 1] specifies the initial state distribution, S S:SxAA A - R is the reward function, H E N refers to the time horizon, and y E [0, 1) is the discount factor. Additionally, in the POMDP setting, O denotes the set of observations, and O: : S x O + [0, 1] represents the observation function. At the initial time step / = 0, a starting state 80 ~ So(-) and an initial observation op ~ On(-150) are sampled. At any time step / E 0, H- - 1, the policy takes an action a, E A in the environment, which updates the state according to the dynamics 8 ~ T(-+s,, The next observation is then sampled as ot+1 (-1)s(1), and the reward is computed as rt+1 R(st, ae, St+1). Let the observable 01-1. Or, at), then the memory-basee trajectory up to time step ł be denoted as To.t = (og, ag. 01, o,), conditioned on the entire history. policy, in its most general form, is defined as π
Markov
In this paper, we focus on the masked partially observable setting, where state factors are obscured or "masked" in dynamic manner, such as when sensors of robot are occluded or signals are perturbed. In this context, any state can be represented as a collection of random variables 8, = sst, ot ....... J. This implies that the state consists of various factors collected from multiple sources.
3

---

Each factor s' can differ in type and size, ranging from high-dimensional (c.g., multimedia data) low- dimensional (e.g.. tabular data). We introduce M-dimensional binary mask variable m M {0. 1) }aa* to signify the observability of each factor: for the t-th time step and the i-th factor, mi = 1 if it is observed, and 0 otherwise. Consequently, we redefine as the masked observation function: Given the mask value m, the current observation o can be get from ((-|s,) = [s]] m = 1]. In this context, different mask variables result in distinet POMDPs due to variations in their observation functions. And the mask variable my may change with the time step t. An MDP can be considered as a special case of a POMDP when the observation o is always identical to the state 8, i.e., for any time step & and factor i, mi- 1.
Only accessible during tuining
Oy(a)
02 (giea
By Ogfr)
(yts)
Diverse POMDPs
Figure 1: In general POMDP problem, the policy is trained on samples which contain state information and expected to perform well across various POMDPs with distinet observation functions.
Generalization over a POMDP Family. Prior works generally aim to address specific POMDP. where the policy is trained and tested within the same POMDP. In this paper, we aim to tackle a more general POMDP problem depicted in Figure [I In particular, during the training period, the policy has access to samples which include state information. During the testing period, the policy is expected to perform well in various POMDPs that share the same underlying MDP but have distinct observation functions. The objective for such a problem in the masked POMDP setting can be regarded finding optimal policy π [aa])oo)) to maximize the expected discounted return under different masked observation functions: max, Rsg--So.m-MM [ f- 7're+1|80)) [80]]-
To:e)
Offline RL. Our work focuses on the offline setting, where the agent learns a policy from a static dataset D (s, a, s',r) without collecting new experience data. The dataset is generated by another policy, referred to as the behavioral policy and denoted by (a(a)s) We allow full access to state information during the training phase by relaxing partial observability constraints. However, during execution, the trained policy must handle diverse partial observation functions. Our method separates representation learning and policy learning, enabling the use of state-of-the-un offline RL algorithms to train a decision head that can make decisions based on the learned representation (see Section 4). In our paper, we employ Implicit Q-learning (IQL) (28 (see details of IQL in Appendix.) as the offline RL. algorithm for training our decision module.
4 Offline RL with Discrete Proxy Representations
We introduce ORDER, a framework designed tackle challenges in partially observable settings for offline RI. Our approach uses two main strategies: i) We learn discrete proxy state representations to handle outtr--distributtin states and uncertainties in observation functions. The choice of discrete state representation is deliberate, as ensures that our policy maintains its generalizability even in situations characterized by uncertainty. By converting unseen partial observations into discrete forms akin to those present in the training data, ORDER could adapt to a variety of uncertain, partially observable situations. ii) We use an estimated proxy representation with learned discrete state representations. This allows our model to understand the state information even with limited observations, improving accuracy and making the optimization process easier. In the following subsections, we first give an overview of the training process, then explain how we learn discrete state representations and discrete proxy representations for partial observations.
connect partial observations
Overview. As illustrated in Figure 2 ORDER a three-phase paradigm for developing a policy capable of addressing the general POMDP problem, In the first phase, train state encoder & for generating discrete state representations and then convert the original dataset D = {(88, s', r)} to new dataset D.- {(2, a, z', ,r)], where (bs)) Z is the concatenation of representations from different state factors. We will introduce the details of $ in the following subsection. In the second phase, an arbitrary offline RL algorithm is applied learn decision head h((a)z) from D... This head could provide an action for given state representation 2. Combined with 4. the oracle policy
o (a]s), which can be executed on the state space, can written as (als) = h (())) However,

---

Phose Training encoder and coaverting dataset
Phose Training decision bead
Phase 3: Training proxy encoder
Original datawet
Training
eucoder + (5)
Converted dataset [00.0
Training
Deciiiaa head A(a]e)
[ss xr2
((s,0,5'r))
Consiructing data
Generated partial observations Tmiiiing
Prexy state encoder (2g For--1 0t)
Full o
Diserete state representations 2
Figure 2: The three-phase training paradigm of ORDER, encompassing (1) training a state encoder & (2) training decision head h(a)2), and (3) training a proxy state encoder o
the full state information is only available during training, During execution, the overall policy cannot be directly used, since some state factors will be missed. To address this issue, in the third phase, we train praxy state encoder to predict the discrete state representation from the past history of partial observations instead of the full state information: & ~ 4 To: 11) Here, & is the predicted state representation which conditioned the past history of observations. Combining the decision head h(a z) with the proxy state encoder o (--1To:--1 ot), we obtain a proxy policy, which could address different partially observable scenarios, ø (ax) (o-]Tox--1,0) ot) h (d10(1170x11,0)) We summarize the oracle policy and proxy policy in Figure (c-d).
4.1 Learning Discrete Representations over States
In this subsection, present a method for learning discrete state representations, drawing inspiration from the VQEVAE approach [45]. Recall that in our setting. we consider the state as a collection of M heterogeneous state factors:
[s 8² , (refer Section 3}. For each factor, define codebook e' E x 21, 2....... .). where represents the size of the discrete latent space (i.e.,a K K-way categorical), and D denotes the dimensionality of each latent embedding vector ej E IR..j E 21,2,--- ... K}. As shown in Figure (a), the factor encoder d' takes the state factor s' as input and outputs an embedding é'. The discrete latent variable z is then calculated by the nearest neighbor look-up g using the codebook ex as follows:
g e¹)
where arg min ||ēt etile.
Next, we obtain the discrete compositional representation by concatenating the discrete representa- tions of all state factors:
( CONCAT(22, ................
Here, we summarize the entire process of mapping to as = ((s). For simplicity, we refer to & as the state encoder, which actually comprises the set of all codebooks and factor encoders: (e', of 1,2, .M. And its architecture is illustrated in Figure (a).
Learning Objective for d. The representation learning objective is based on the Vector Quantized AutoEncoder (VQ-VAE) approach [45] and can be expressed as follows:
M
M
2
1
2
Σ C( = EanD [1(=) - 3 M
sg((e)
+
(3)
2
M
Here, represents the of all codebooks and factor encoders, while E denotes the state decoder and Σ(2) is the state reconstruction. The function sg(.) performs stop-gradient operation to block gradients from entering its argument. The hyperparameter 3 controls the degree which the code can change. The first term is the reconstruction loss, which ensures that the encoder captures sufficient state information. Note that for the nearest lookup operation in Equation (T), there is no actual gradient,
the -htighhtthrrugh estimator copy gradients from the decoder input z

---

to the encoder output 2", The second term is the codebook loss, which only applies to the discrete latent vector and encourages the chosen 2 be close to the encoder output 8t The third term the commitment loss, which only applies to the factor encoders and encourages Et to remain close to the selected discrete latent vector z². We set the value of / to 0.25, as in the original VQ-VAE paper 45 The training paradigm of state encoder is summarized in Appendix.
4.2 Learning Discrete Proxy Representations over Partial Observations
a
C
Dechina kecee)
d
Dediia hard
Prodution feadled
Cabbbode
moadie (s(
Pros stax moxde (ai (k)
1
Trajecary cauaer
Coddbaak
(x*)
giocn (or)
Canore 94l absrccttoo
Trajessyy patiid
Priny state ranader (Ryltas-3
(X)
Orack
Procy polley ((0Taa---92)
The architectures of (a) the state encoder (()) and (b) the proxy state encoder (3 o.). And the constructions of (c) the oracle policy <[ass) and (d) the proxy policy A(q//Tor-1,44))
Figure 3:
In this section, focus the alignment of partial observations with the learned discrete state representations derived from full state information. To achieve this, we propose a method that employs recurrent architecture and learnable mask token, allowing the model handle partial observability and estimate the underlying state of the enviromment. By aligning the estimation process with the learned representations mentioned in the previous subsection, our model can effectively generate proxy state representations from incomplete observations, enabling efficient decision- making in partially observable settings. Overall, our aim is obtain inference model named proxy state encoder, denoted as To:t-, or). which approximates the true posterior distribution p(zelTon-1,0 0z). Its architecture is illustrated in Figure (b).
4.2.1 Construction of Partial Observable Training Data
Trajectory of full states
Trajectory masked by M
Trajectory masked by M2
Figure 4: Illustration of Both Missing Scenarios. The figure showcases trajectories spanning various time steps (horizontal axis) and state factors (vertical axis). Colorful grids represent observable state factors, while grey grids signify masked state factors. The left panel presents the trajectory of the full states. In contrast, the middle and right panels display trajectories following masking by the dynamical missing scenario Mi and the factor reduction scenario M2, respectively.
The first step to training & is to get a good training dataset. However, one challenge is that we cannot directly collect partial observations from the environment in the offline RI. setting. We only have the original offline dataset D = {(x, a, rr) and the converted representaion dataset D {(2, a, ,2,,,,))) } that contains full state information. To address this issue, we construct trajectories of partial observations by utilizing the converted dataset as the training data for $. And we introduce partial observability by randomly setting portion of mask variables mi be 0, simulating partially observable scenarios. Specifically, we adopt two scenarios
Dynamical Missing Scenario Mi: For each time step we set the mask variable mf = 0 at the probability of 7) and otherwise the probability of 7). this case, the missing factors dynamically change over time with certain missing ratio.

---

Factor Reduction Scenario M2 : Given a trajectory of data whose time length is T, we set the mask variables mb.r 0 at the probability of 17, and otherwise at the probability of '/- This scenario arises when part of the sensors are removed, and the corresponding components of the observations are consistently missing over time.
We give examples of trajectories masked by both missing scenarios in Figure + And we adopted both scenarios in our training paradigm by using them iteratively, and denoted the process generating random masking variables mo:: My, which summarized in Appendix.
4.2.2 Learning Discrete Proxy Representations
After generating training data, the next step to training is to aggregate information partial observable trajectory. Concretely, given the partial observation [s 11.
+ f gf e. m.aax)
( (5t(t))
if m' 1, 0.
inakk] (0) CONCAT(2", ,,,,.... ,. where x²
(4)
if mm
Here, we denote [|nast]] the masked state encoder, which contains the trained state encoder mentioned in the previous subsection and an additional learnable mask token e mankk] E RIxD. It directly converts the observable state factor into the learned discrete factor representation, and represents the unobservable state factor as the mask token. Next, we use trajectory encoder & to capture the temporal information the trajectory of partial observations. Concretely, it takes current observation representations and previous actions as input and output continuous vector 1 E( ((-1).
Discrete State Representation Distribution Estimation. Now, we want predict the cat- egorical distribution of p(zal To:--1 For each factor i, we build set of prediction head {w'li j 1,2, ........ ..... .M} which takes the trajectory representation V as input, and output the categorical probalility over the discrete embeddings of the unobserved factors in the form of softmax output. And we only estimate the unobservable factor.
During the execution, we predict the proxy state representations as follow:
g { ~ (-(p)) if m = 0
im =
ž = CONCAT( ,,,,,,,
,MM) where
(5)
Learning Objective for क. The overall learning objective for the distribution estimation be summarized as minimizing the Cross-Entropy loss between the predicted probability distribution and the true discrete embedding associated with the input sample:
[aclaakk] -g,
L (1-mi) logw' ='1vo). (6)
The training paradigm of proxy state encoder is summarized in Appendix.
Experiments
In this section, we present a thorough empirical evaluation of ORDER, highlighting its effectiveness in improving offline RL algorithms for a variety of partially observable scenarios. We integrate ORDER with the IQL algorithm (28) and assess its performance across an array of environments in the D4RL benchmark [I2] under different partial observation situations. Our results clearly indicate that ORDER substantially enhances the generalization performance of policies trained on offline datasets in diverse partially observable conditions. To ensure the robustness of our findings, we conduct all experiments with five distinct random seeds, each consisting of ten separate runs. For detailed information on the architectures and hyperparameters, please refer to Appendix. In the following subsections,
first evaluate the generalization performance of our model and other baselines under varying degrees of partial observability. Subsequently, we carry out an empirical study analyze the significance of our learned discrete proxy representations.
7

---

HallCheetah-mediumvv
Hopper-mediunv2
Walker2d
Maze-medium-dense-W
Dynumical Missing Sccaurto
Aug. Normalize Score
40
Aug. Normalzzo Score
80
Aug. Normalized Score
80
80
60
M
30
Norman
40
40
10L/A
10
20
20
20-
0
50
30
50
70
90
100
30
50
70
50
75
Missing
Missing Ratio (%)
Missing Ratio (9%)
Missing Ratio (%)
Figure 5: Generalization performance of different models under dynamic missing scenarios.
5.1 Evaluating Generalization Performance under Diverse Partial Observation Functions
In this subsection, our primary objective is to assess the generalization performance of policies, which are trained on offline datasets containing full observations, under various partial observation scenarios. We evaluate ORDER and other baseline algorithms on gym locomotion tasks and maze navigation tasks in the D4RL benchmark [1].. In our experimental setup, we treat each dimension of the observation space as an independent state factor.
Our proposed model employs the IQL algorithm (28) to train the decision head on the representation space. By combining the trained proxy state encoder with the decision head, our model can make decisions under partial observations. We denote it as TQL_ ORDER and compare it with the following baselines: i) IQL with the recurrent architecture (IQI_R) R): We integrate IQL with state-of-the-an method for online RL in POMDPs [40] to create this baseline. This method utilizes a specialized recurrent neural network architecture to tackle partial observability and can be combined with arbitrary actor-critic algorithms. During training, we employ the same mask strategy (see Section 4 to construct partial observable training data for it. Furthernore, to indicate which state factors are missing, we use ( to replace with the value of unobserved factors and concatenate the mask variable my with the observation vector O₂ as the extended input. ii) IQL with the strategy of filling zero (IQL FZ): This baseline is trained directly using the IQL algorithm with the original offline dataset. During testing, 0 are used to represent unobservable factors. iii) IQL with the strategy of filling adjacent elements (IQI_ FA): This baseline is also trained directly using the IQL algorithm with the original offline dataset. During testing, if there is a latest observed value of an unobserved factor, it used to represent the current unobservable factors; otherwise, 0 is used.
Generalization Performance under Dynamical Missing Scenarios. We first evaluate policy performance under dynamically missing scenarios, where certain state factors may intermittently and randomly be absent over time. We conduct this evaluation across various missing ratios to gauge the models' adaptability. An intrinsic characteristic of a policy with strong generalization ability its ability to maintain performance as the missing ratio increases. The results are illustrated in Figure 5. QL_ORREER outperforms other baselines in most cases, demonstrating its superior generalization performance over dynamic missing scenarios. On the other hand, the performance of IQL_ORDER slightly decreases with increasing missing ratios in most cases. However, even in extreme missing scenarios (missing ratio greater than 70%), still maintains good performance. Additionally, IQI_Fz and IQI__A exhibit similar performance, with IQI__A FA slightly outperforming IQ._FZ. Their performance is worse than ORDER, indicating that simply padding zeros or the latest values into unobservable values cannot effecttivlyy address this task. Filling adjacent elements proves to be a better strategy than filling zeros. Finally, IQL_R R performs the worst in all cases under dynamic missing scenarios. However, it is worth noting that IQL_R R performs well in settings where observation functions are single and fixed (see Appendix). This suggests that it struggles to develop effective policies under diverse and dynamic partial observation settings. One possible reason is that the diverse partial observation function leads to enhanced -on-staiionariy of dynamics, making it challenging directly train policies on these cases and resulting in highly unstable training.
Generalization Performance under Factor Reduction Scenarios. Next, we evaluate policy performance under factor reduction scenarios, where in each test episode, some state factors are randomly removed throughout the entire episode. Notably, we exclude the IQI__A FA model in this task since we cannot obtain the latest values of the missing factors.
8

---

HalfCheetah-meede medium-v2
Hopper-medlum-v2
Waller2ddmediium -2
Maze-mediummdense--
Avg. Normalizes Score
Aug. Narmalized Score
6. oooth
Avg. Normalized Score
8
Avg. Narmalized Score
100
RL oRDen
40
IX.
10L72
60
60-
30
40
40-
40
20-
10
30
50
70
90
10
30
50
70
90
30
30
50
70
90
25
50
75
Missing Ratio (%)
Missing
Ratio (96)
Missing
Ratio (%6
Missing Ratio 1%)
Figure 6: Generalization performance of different models under factor reduction scenarios.
The results are depicted in of Figure 6 JQL_ORDER outperforms other baselines in most cases. However, when the missing ratio is high, its performance decreases more significantly than in dynamic changing scenarios. This might be attributed the increased difficulty of factor reduction scenarios compared to dynamic changing scenarios. In this setting, models cannot infer the current missing factors from their latest observed values; instead, they must rely on other observable factors for inference. Both IQL FZ and IQL R continue to perform poorly. indicating that padding zeros is not an effective strategy and that IQL cannot achieve stable training under such challenging scenarios.
5.2 Empirical Analysis of Discrete Proxy Representations
In this subsection, we aim to underscore the significance of discrete proxy representations in our framework. We conduct experiments on the Hopper-medium-Ve dataset. For these experiments, we set up two comparison mootels-Contiuuuss Reps. and Raw States-both of which employ our three-phan phase paradigm develop policies. The Continuous Reps. model uses a Variational Autoencoder ( (VAE) in the first phase to generate continuous representations, and in the third phase, it predicts Gaussian distributions over these continuous representations. On the other hand, the Raw States model applies an identity function as its state encoder in the first phase and predicts Gaussian distributions of the missing factors in the third phase.
Diccrete Mops.
Dynamical Missing Scenario
Mandin Rati
Ractor Ra decitiion Scanario
10
50
73
missing missing ratio-50%
missing ratio-711%
missing
23
40
Original representations (or states) Proxy representations for states)
Figure 7: Empirical Analysis of discrete proxy representations. (a) T-SNE visualization showing alignment of discrete, continuous, and raw state representations with original representations under varying missing ratios Factor Reduction Scenarios. (b-c) Generalization performance under dynamic missing and factor reduction scenarios. (d) Performance evaluation with varying numbers of learned discrete representations.
One challenge we would like to address is the potential misalignment between the partial observability encountered during testing and the data on which the RL policy is trained. To examine this, we first sample 10 trajectories generated by different models under various missing ratios in the Factor Reduction Scenarios. We then use t-SNE 46 to visualize their generated proxy representations (or states) and the original representations (or states), which can be regarded as the ground truth under full

---

observability. The proxy representations are intended to prevent policies from encountering unseen observations, which could lead to severe performance degradation in the offline RL setting. Therefore, suitable proxy representation should align well with the original representation, even when the missing ratio is high. shown in Figure 7 (a), we observe that discrete proxy representations align well with the original representations when the missing ratio is below 70%. In contrast, continuous proxy representations begin to mismatch with the original representations when the missing ratio exceeds 10%. Meanwhile, the proxy raw states show mismatch at missing ratio of 10%. Furthermore, Figure (b- illustrates the generalization performance under both missing scenarios. We find positive correlation between alignment ability and generalization performance: discrete representations outperform continuous representations, which in turn perform better than raw states. Additionally, discrete representations maintain good performance when the missing ratio is below 70%, but decrease when the missing ratio exceeds 70% in the Factor Reduction Scenario, consistent with the results in Figure 7 (a). These findings emphasize the importance of utilizing discrete proxy representations in our framework for maintaining a strong alignment between proxy and original representations under partial observability. This alignment allows the policy to make better decisions and exhibit improved generalization performance in partially observable settings, We also evaluate the performance of models with different numbers of learned discrete representations, as shown in 7 We find that increasing the number of discrete representations can improve model performance, likely because a higher number of discrete codes enhance the model's expressive ability. However, when the number exceeds 40, the performance improvement is marginal, indicating that a trade-off between the number of discrete codes and model performance can be achieved.
Conclusion
We present ORDER, novel probabilistic offline RL. framework that addresses the challenges partial observability in real-world scenarios. By leveraging discrete proxy state representations, ORDER significantly improves robustness and generalization performance across diverse observation functions. Our three-stage training approach effectively harnesses the power of discrete state representations and proxy discrete representation alignment. Extensive experiments demonstrate the effectiveness of ORDER in various partially observable settings. However, some limitations still exist. ORDER currently focuses on masked partial observation functions, leaving other forms like perturbations and noises unaddressed. Moreover, its applicability more complex real-world tasks, such as autonomous driving. remains to be explored. Despite these limitations, we hope ORDER can pave the way for the deployment of RL policy against various partial observabilities in the real world.
Acknowledgements
This research is supported by the National Research Foundation, Singapore under its Industry Align- ment Fund - Pre-positioning (IAF-PP) Funding Initiative. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.
References
[1] Andrea Baisero and Christopher Amato. Unbiased asymmetric reinforcement learning under partial observability. In AAMAS, 2022.
[2] Andrea Baisero, Brett Daley, and Christopher Amato. Asymmetric DQN for partially observable reinforcement learning. In UAI, 2022.
[3] Bram Bakker. Reinforcement learning with long short-term memory. In NeurlPS, pages 1475-1482, 2001.
[4] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 1308.3432, 2013.
[5] Aude Billard and Danica Kragic. Trends and challenges in robot manipulation. Science 364(6446)ccat84144 2019.
10

---

[6] Konstantinos Bousmalis, Alex Irpan, Paul Wohlhart, Yunfei Bai, Matthew Kelcey, Mrinal Kalakrishnan, Laura Downs, Julian Ibarz, Peter Pastor, Kurt Konolige, Sergey Levine, and Vincent Vanhoucke. Using simulation and domain adaptation to improve efficiency of deep robotic grasping. ICRA, pages 4243 2018.
[7] Huayu Chen, Cheng Lu, Chengyang Ying. Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. In ICLR, 2023.
[8] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. arXiv preprint arXiv:2106.01344 2021.
[9] Xiaoyu Chen, Yao Mark Mu, Ping Luo, Shengbo Li, and Jianyu Chen. Flow-based recurrent belief state learning POMDPs. In ICML, pages 3444-3468, 2022.
[10] Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Mag- netic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897):444 419, 2022.
[11] Angelos Filos, Panagiotis Tigkas, Rowan McAllister, Nicholas Rhinehart, Sergey Levine, and Yarin Gal. Can autonomous vehicles identify, recover from, and adapt distribution shifts? In ICML, pages 3145-3153, 2020.
[12] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072119 2020.
[13] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In ICML, pages 2052-2062, 2019.
[14] Karol Gregor, Danilo Jimenez Rezende, Frederic Besse, Yan Wu, Hamza Merzic, and Aäron van den Oord. Shaping belief states with generative environment models for RL. In NeurIPS, pages 13486-134444 2019.
[15] Pengjie Gu, Mengchen Zhao, Chen Chen, Dong Li, Jianye Hao, and Bo An. Learning pseudometric-bee based action representations for offline reinforcement learning. In ICML, pages 7902-7918, 2022.
[16] Pengjie Gu, Mengchen Zhao, Jianye Hao, and Bo An. Online ad hoc teamwork under partial observability. In ICLR, 2022.
[17] Hongyi Guo, Qi Cai, Yufeng Zhang, Zhuoran Yang, and Zhaoran Wang. Provably efficient offline reinforcement learning for partially observable Markov decision processes. In ICML, pages
2022.
[18] Abhishek Gupta, Coline Devin, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Learning invariant feature spaces transfer skills with reinforcement learning, 2017.
[19] David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. NeurIPS, 2018.
[20] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In ICML, pages 3565,5565, 2019.
[21] Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering Atari with discrete world models. In ICLR, 2021.
[22] Dongqi Han, Kenji Doya, and Jun Tani. Variational recurrent models for solving partially observable control tasks. In ICLR, 2020.
[23] Matthew Hausknecht and Peter Stone. Deep recurrent Q-learning for partially observable MDPs. arXiv preprint axXiv:1507.062211 2015.

---

[24] Nicolas Heess, Jonathan J Hunt, Timothy P Lillicrap, and David Silver. Memory-based control with recurrent neural networks. arXiv preprint arXiv:1512.04455, 2015.
[25] Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Irpan, Julian Ibarz, Sergey Levine, Raia Hadsell, and Konstantinos Bousmalis. Sim-to-real via sim-to-sim: Data-efficient robotic grasping via gellomizee-t--canoniin adaptation networks. In CVPR, pages 16619-12229, 2019.
[26] Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In ICML, 2022.
[27] Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2)99-134, 1998.
[28] llya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit Q-learning In ICLR, 2022.
[29] Ily Kostrikov, Jonathan Tompson, Rob Fergus, and Ofrr Nachum. Offline reinforcement learning with Fisher divergence critic regularization. In ICML, pages 5774-5783, 2021.
[30] Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off aff-policy Q-Learning via bootstrapping error reduction. In NeurlPS, pages 1 11761-11771, 2019.
[31] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-Learning for offline reinforcement learning. In NeurlPS, pages 1179-1191, 2020.
[32] Alex X Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with latent variable model. NeurlPS, pages 741-752, 2020.
[33] Jonathan N. Lee, Alekh Agarwal, Christoph Dann, and Tong Zhang. Learning in pomdps is sample-efficient with hindsight observability. In ICML, 2023.
[34] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives open problems. arXiv preprint 2005.01643, 2020.
[35] Bogdan Mazoure, Ilya Kostrikov, Ofir Nachum, and Jonathan Tompson. Improving zero-shot generalization offline reinforcement learning using generalized similarity functions. In NeurlPS, 2022.
[36] Lingheng Meng, Rob Gorbet, and Dana Kulić. Memory-based deep reinforcement learning for pomdps. In IROS, pages 5619-5661 5626, 2021.
[37] Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andy Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, Dharshan Kumaran, and Raia Hadsell. Learning to navigate in complex environments. In ICLR, 2017.
[38] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland. Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 5 518(7540):529-553 533, 2015.
[39] Hai Huu Nguyen, Andrea Baisero, Dian Wang, Christopher Amato, and Robert Platt. Leveraging fully observable policies for learning under partial observability. In CoRL, 2022.
[40] Tianwei Ni. Benjamin Eysenbach, and Ruslan Salakhutdinov. Recurrent model-free RL can be a strong baseline for many POMDPs. In ICML, pages 16691-16723, 2022.
[41] Martin L Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, 2014.
[42] Anton Maximilian Schäfer and Hans Georg Zimmermann. Recurrent neural networks universal approximators. In ICANN, pages 632-640, 2006.
[43] David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mas- tering the game of Go with deep neural networks and tree search. Nature, 529(7587):84444955 2016.
12

---

[44] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation the real world. In IROS, pages 23-30, 2017.
[45] Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning, NeurlPS, 30, 2017.
[46] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of machine learning research, 9(11), 2008.
[47] Yuhui Wang and Xiaoyang Tan. Deep recurrent belief propagation network for POMDPs. In AAAI, pages 10236-10244, 2021.
[48] Andrew Warrington, Jonathan W Lavington, Adam Scibior, Mark Schmidt, and Frank Wood. Robust asymmetric learning in pomdps. In ICMI., pages 1013-11023, 2021.
[49] Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamice model for control from raw images. NeurlPS, 28, 2015.
[50] Daan Wierstra, Alexander Foerster, Jan Peters, and Juergen Schmidhuber. Solving deep memory POMDPs with recurrent policy gradients. In ICANN, pages 697-706, 2007.
[51] Kwangjin Yang. Seng Keat Gan, and Salah Sukkarich. gaussian process-based RRT planner for the exploration of an unknown and cluttered environment with UAV. Advanced Robotics, 27(6):431 443, 2013.
[52] Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. Sim-to-real transfer in deep reinforcement learning for robotics: survey. In 2020 IEEE Symposium Series on Computational Intelligence, SSCI 2020, Canberra, Australia, December 1-4, 2020, pages 737-744. IEEE, 2020.
13


Reviews:
[{"review_id": "cTxAj62o5Z", "reviewer": "Reviewer_iZED", "summary": "In this paper, the authors present a three stage method for offline RL in POMDPs. It is assumed that the agent has access to full observations (state) during training, but only masked versions of this state (partial observations) during inference or deployment. In the first stage of training, a mapping is learnt from the original state space to a discrete state space. In the second stage, an optimal policy is learnt using offline RL which maps this discrete state to an action. Finally, in the third stage, a proxy state representation model is learnt, which maps the history of partial observations to a proxy discrete state, which is aligned with the discrete state corresponding to the underlying system state. As different state factors can be masked during inference, this method presents a solution that generalises well to a family of POMDPs obtained from the underlying MDP using different masking strategies. The authors demonstrate this method on four MuJoCo environments.\n", "questions": "1. [Typo] In Line 164, shouldn't the serial number be \"i\" instead of \"ii\"?\n2. [Typo] In Eq(1) is a term missing after \"where\"?\n3. How can it be proven that the discrete state obtained is indeed a state sufficient for control?\n4. I did not understand the difference and the motivation between the second and the third terms in Eq (3). Can the authors please explain the same?\n5. \\tau is not defined. Also, the subscript corresponding to \\tau is sometimes just \"t-1\" and sometimes \"0:t-1\". Can these be made consistent with a proper definition?\n6. [Typo] In line 230 consider replacing \"has\" with \"have\".\n7. [Typo] Reference missing (??) in Line 66 of Supplementary Material.", "limitations": "The authors have covered the limitations of their method. While they do not mention the societal impact, since the paper deals with a general algorithm, the societal impact is same as in any RL case.\n", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper is novel, very well written and easy to comprehend.\n2. The paper presents generalising an RL solution of an MDP to a family of POMDPs, thereby making RL generalisable and fairly robust to several real-world problems with a type of partial observations (masked state factors).", "weaknesses": "1. There is no analysis/theory presented regarding loss of optimality due to the discrete state representation.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer:\n\nThank you for your thoughtful questions and feedback on our paper. Below, we provide detailed responses to each of your questions:\n\n# Response to Typographical Comments:\n\n   1. **Line 164**: You're right, and we apologize for the oversight. It should indeed be \"i\" instead of \"ii\". This will be rectified.\n    \n   2. **Eq(1)**: Thank you for pointing it out. The term $ j $ was indeed missing after \"where\". We will correct this in the revised manuscript.\n\n   3. **Line 230**: Your observation is accurate, and the correction has already been implemented.\n\n   4. **Line 66 of Supplementary Material**: We apologize for the omission. The intended reference is \"Algorithm 4\". We'll ensure it is properly cited in the revision.\n\n# Response to the Discrete State Query:\n\nYour query regarding the sufficiency of the discrete state for control is insightful. To demonstrate the sufficiency of our discrete representation:\n\n   1. **Performance Metrics**: We utilize standard benchmarks and performance metrics in our experiments. The fact that our method, using the discrete state representation, achieves competitive or superior performance indicates that the representation captures the necessary state information for effective control.\n\n   2. **Robustness in Varied Environments**: In addition to standard benchmarks, we evaluate our approach in a variety of environments and scenarios. The discrete state's ability to generalize across these settings further attests to its sufficiency.\n\n   3. **Comparison with Continuous Representation**: We juxtapose the performance of our discrete representation with its continuous counterpart. The discrete representation's comparable or better performance suggests its capability to serve as an efficient and sufficient control state.\n\nIn the future, one could also explore methods to directly measure the information content or fidelity of the discrete state against the true continuous state. However, for the scope of this paper, our empirical results act as a testament to the discrete state's efficacy for control tasks.\n\n# Response to Query on Eq (3) Terms:\n\nYour query regarding the second and third terms in Eq (3) is pertinent. To provide clarity:\n\n   1. **Purpose of Both Terms**: Both terms originate from the VQ-VAE framework, aimed at ensuring high-quality representations during training.\n\n   2. **Difference in Position of  $sg()$**: The positioning of the \"stop gradient\" function, $sg()$, differentiates the terms. It's pivotal in dictating where the gradient propagates during backpropagation.\n\n   3. **Separate Control Mechanisms**: The distinct placements of $sg()$:\n       - The second term, with $sg()$, focuses on updating the codebook, ensuring it reflects the encodings accurately.\n       - The third term, conversely, guides the encoder learning so that its outputs align closely with the existing codebook entries.\n\n   4. **Intuitive Takeaway**: Think of it as a dance between two partners, the encoder and the codebook. The second term ensures the codebook learns to match the encoder's steps (representations). Simultaneously, the third term ensures the encoder does not stray too far from what the codebook knows, offering a consistent dance.\n\nBy carefully balancing these terms using scale coefficients, we can achieve a harmonious alignment between the encoder and the codebook, ensuring optimal representations.\n\n# Response to Query on $\\tau$ Definition:\n\nThank you for pointing out the inconsistency regarding $\\tau$. We apologize for the confusion.\n\n1. **Definition of $\\tau$**: As highlighted, $tau$ is indeed defined in lines 122-123 of the manuscript. \n\n2. **On Subscript Clarification**: We understand the confusion and regret the inconsistency. To clarify, the notation \"$\\tau_{t-1}$\" is a typographical error. The correct notation is \"$\\tau_{0:t-1}$\", which denotes the entire trajectory from the start up to \"t-1\".\n\nWe will rectify the inconsistency throughout the manuscript to ensure \"$\\tau_{0:t-1}$\" is used consistently and appropriately. Your feedback is instrumental in enhancing the clarity of our work."}, {"review_id": "moMSGYUfgE", "reviewer": "Reviewer_iLBR", "summary": "This work study the masked partial observability in reinforcement learning and propose a novel method ORDER to address this challenge. ORDER leverages the alignment of discrete state representations and significantly improves robustness and generalization performance across diverse masked partial observability. Experiments demonstrate the effectiveness of ORDER in some settings.", "questions": "1. Whether it is reasonable to convert any unobservable state factor into a single mask token $e^{[mask]}$? Is there any convincing reason or it is just a practical implementation to enable the algorithm execution? $e^{[mask]}$ is learned by minimizing (6) along with the tray encoder and prediction heads, can you explain what it means to learn such a mask token?\n2. You mentioned that the state factors can be heterogeneous, but it seems that your experiments only include homogeneous cases where each dimension of the observation vector is treated as an independent state factor. I can accept it as a simplified experiment setting, but I wonder that in general heterogeneous cases where each state factor differs in type and size, how to choose the numbers of discrete codes for each factor? A common discretization degree seems unreasonable now.\n3. Do some kind of heterogeneous state factors like high-dimensional multimedia data suitable for discrete representations? Since in your experiments, each state factor is a single scalar, which is an extreme simplification compared to the heterogeneous factors that you claimed, I really doubt this point.", "limitations": "ORDER currently focuses on masked partial observation, leaving other forms like perturbations and noises unaddressed. Moreover, its applicability to complex heterogeneous high-dimensional real-world tasks remains to be explored.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work leverages the alignment of discrete state representations learned by the fully observable offline data and discrete proxy representations for the partial observable experiences, novelly constructing a proxy policy for the partial observable online interaction, which improves the robustness and generalization for the diverse partial observation.", "weaknesses": "The author claims that ORDER is suitable for heterogeneous state factors, but only provides the experiment results for simple homogeneous state factors settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer:\n\nThank you for your thoughtful questions and feedback on our paper. Below, we provide detailed responses to each of your questions:\n\n# Single Mask Token Justification\n\nThanks for your feedback. Your query regarding the use of a single mask token is apt. The primary intention behind using just one mask token is to provide the network with a cue that certain information is occluded, rather than detailing the specific content of the occlusion. This serves as an efficient indicator. In practice, this approach finds resonance with prevailing techniques in transformer architectures, such as in the MAE [1] model. The essence is to highlight the presence of an occlusion rather than its intricacies, for which a single token suffices.\n\n[1] *Masked Autoencoders Are Scalable Vision Learners.* CVPR 2021.\n\n# Addressing Heterogeneity in State Factors\n\nThanks for your feedback. Your observation regarding the treatment of state factors in our experiments is keen. In the mujoco tasks that we worked on, state factors are indeed heterogeneous in nature. Specifically, the state vector amalgamates two distinct types of data: the robot's sensor positions and its angular velocities. \n\nWhen state factors diverge in type and magnitude, a reasonable approach to select the number of discrete codes would be to gauge the redundancy of information inherent to each state factor. In simpler terms, state factors carrying more intricate or effective information might necessitate a greater number of discrete codes. To facilitate this operation, expertise from human professionals can be sought or alternative information measurement techniques can be employed. This ensures a tailored discretization that aligns with the unique characteristics of each state factor.\n\n# Addressing High-dimensional Multimedia Data\n\nThank you for your thoughtful comment.  We agree that this is an important consideration. In support of our approach, we refer to a relevant study [1] in the domain of model-based reinforcement learning. This work demonstrates the efficacy of discrete representations in enhancing policy performance, particularly when dealing with high-dimensional data such as images in gaming scenarios. Notably, the findings indicate that encoding such data into discrete representations not only preserves performance but also enhances robustness by mitigating the impact of information noise associated with high-dimensional factors.\n\nWe acknowledge the potential oversimplification in our current experimental setup and will address this concern by including a concise discussion in the related work section, highlighting the relevancy of discrete representations for handling high-dimensional multimedia data.\n\n\n[1] *Mastering Atari with Discrete World Models.* ICLR 2021."}, {"review_id": "sFS7Ob1okH", "reviewer": "Reviewer_AAho", "summary": "The work looks to tackle a subset of the partially observable offline reinforcement learning problem setting, where a dataset of offline experience (of full states and masked states) is given during training, and an agent is tested on masked state features during test time. The authors propose the ORDER training framework, where during training time, an agent first learns discrete state representations using the full states, then uses an RNN to learn (in a supervised manner from the discrete representations from the full states) the discrete state representations over the masked states.\n\nWith this framework in place, the authors show that this approach does well in a masked version of the D4RL offline reinforcement learning benchmark, which they create themselves. They show this form of discretization outperforms baselines algorithm performs better on this benchmark as compared to other baselines. They also show an improvement as a higher percentage of the states are masked out.", "questions": "Here is a central question I would like answered with regards to the work: From Appendix B, “Following this, we employ a grid search strategy to finalize our choice of hyper-parameters, the details of which are reported in Table 5.” What was this grid search strategy? How did you arrive at these hyperparameters?\n\nBesides this question, here is a section-by-section review/questions with regard to the work:\n\n****1.****\n\n“While POMDP methods excel in online RL, they rely on continuous environment interactions for policy adaptation, making them unsuitable for offline RL” - This statement confused me a bit. Do you mean that you need on-policy samples for POMDP methods?\n\n“we aim to tackle a more general POMDP problem” - This is false. Your problem set up is not more general, seeing as you have access to states in training. The most general class of POMDP problem settings are settings where you don’t have access to state at all. This setting encompasses “POMDP families” by simply expanding the state space instead of saying that there are multiple observation functions.\n\nThis formulation and claim that the authors are making about tackling a “more general POMDP problem” is also **untrue** in your problem setting. There aren’t different observation functions between train and test time. You simply use a stochastic observation function.\n\n****3.****\n\n“In particular, during the training period, the policy has access to the underlying MDP” - I’m assuming you mean *******samples which include state information*******? If you have the underlying MDP then why not do online RL? Please be more precise.\n\n****4.****\n\n“By converting unseen partial observations into discrete forms akin to those present in the training data”\n\n**Is all these machine learning machinery necessary to discretize states?** We’ve had tile coding and different, very effective, state discretization schemes for a while now. Why do we need a VQ-VAE?\n\n“And we introduce partial observability by randomly setting a portion of mask variables m^i_t to be 0, simulating real-world partially observable scenarios.” - this entire problem set up is a big red flag. **The problem setup is not a realistic representation of partial observability ******at all********. From reading this paper, it sounds like you’ve essentially worked backwards, where you’ve defined a constrained solution method first, then defined your problem setting based on this solution method. It seems like your solution method essentially hinges on the fact that your partial observations are of the form of *************independently************* masking your state feature vectors, which is almost never the case in the real world.\n\nDo I understand this correctly that ************************************************************************************************************************the agent is given the masking vector M?********************************************************************************* If this is the case, then this is another huge assumption. The LSTM here is essentially just predicting missing state features, knowing which are missing.\n\n**4.2.2**\n\nWhat is g here?\n\n****5.****\n\n“Our results clearly indicate that ORDER substantially enhances the generalization performance of policies trained on offline datasets in diverse partially observable conditions.” This is claim is tenuous at best. How is this one partial observability setting considered “diverse observable conditions?”\n\nLooking at your appendix, I’m not sure how you’ve swept hyperparameters and arrived at the hyperparameters given for the baselines. Most importantly, the IQL_R baseline. I’m also unsure as to what IQL_R is exactly. Is it simply IQL with an RNN trained over the masked D4RL datasets?\n\n******5.1******\n\n“An intrinsic characteristic of a policy with strong generalization ability is its ability to maintain performance as the missing ratio increases.” - what does strong generalization performance mean? It seems you mean “generalization in terms of number of masked features”, but more specificity in this language is needed, seeing as “generalization” means a whole deluge of different things in the reinforcement learning context. **This seems to also be an issue throughout this work, seeing the context in which generalization is used.**\n\n“This suggests that it struggles to develop effective policies under diverse and dynamic partial observation settings.” - This is markedly untrue in general - it might be the case where **************************************specifically in your problem setting that you invented************************************** this might be true, but I’m also unconvinced this is the case currently as I’m unsure of how you’ve swept and decided on hyperparameters.\n\n******5.2******\n\nIs using the overlap in t-SNE the best idea for seeing overlap in representations? It’s a dimensionality reduction technique, so the overlap doesn’t necessarily mean much. Why not just use a cosine similarity metric instead? It would be better to have numbers to show the “similarities” in the representations.\n\n****6.****\n\n“that addresses the challenges of partial observability in real-world scenarios.” - **this is just flat-out untrue. Mujoco with random Bernoulli masks is far from real-world scenarios.**", "limitations": "Please see weaknesses and questions.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "Within this specific problem setting, the authors do a good job of showing the benefits of their framework and of discretization. It seems that with this problem set up, discretization and supervised learning over masked states helps quite a bit with performance, and is also quite robust to the percentage of masking based on the plots shown in section 5.", "weaknesses": "There are a few important issues that this paper seems to have, which I’ll describe in broad strokes in this section. For more specific details on these issues, please see the section-by-section review.\n\nThe first big issue is that it seems that the work has mixed their problem setting with their solution method. The two seem innately tied - even the structure of the work mixes the two together. While the authors start the work with describing the general problem setting of partial observability in offline reinforcement learning, the work continues on to describe a problem setting that is quite far off that mark - where the partial observability only stems from simple bernoulli masks, and full state information is given during the training phase. Besides this, additional assumptions are also given to the agent, such as the agent actually knowing *****which***** state features are masked when (which means that **************************************************************************the agent has to know what’s wrong with the state features at every step)**************************************************************************. All of these assumptions on the problem setting are necessary for ORDER to work. While I’m not saying there’s anything fundamentally wrong with the problem setting they describe, **the issue arises with the fact that the proposed solution method, based on the evidence presented in this work, seems to only work in this extremely specific problem setting that they’ve introduced.**\n\nThe language used and structure throughout the work are also quite misleading, especially in the introduction and abstract where it seems the work has overclaimed in the introduction and underdelivered from section 3 onwards. It feels like before reading section 3, you describe your method as general with the ability to tackle POMDPs in an offline setting. But after reading your preliminary section, you seem to reduce the scope suddenly (masked observation functions, different observation functions, access to states during training). I would recommend adding specificity to the problem setting that you describe throughout your work.\n\nBesides this, I find it somewhat suspicious that IQL_R doesn’t seem to work in any of the environments/datasets, and shows the same flat line throughout all environments/datasets. This calls into question implementation or how hyperparameter tuning was done, which was lacking in the appendix.\n\nLastly, I’m also surprised you didn’t reference hindsight state information, seeing as their problem setup is similar to yours. It seems that this setup is more similar to what the authors are trying to tackle.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer:\n\nThank you for your thoughtful questions and feedback on our paper. Below, we provide detailed responses to each of your questions:\n\n# Clarifying Our Problem Setting and Its Real-World Relevance\n\n**Real-world Motivations:** Our specific assumptions, though tailored, are rooted in real scenarios where observation functions are diverse and often unpredictable. Our goal is to bridge this gap with robust solutions.\n\n- **Search and Rescue:** A robot, trained on complete offline data in controlled conditions, may face varied challenges when deployed in different disaster zones. Factors like smoke, murky waters, or sensor noise introduce unpredictable observation functions, emphasizing the need for our approach.\n\n- **Financial Markets:** Algorithms trained on comprehensive historical data can encounter diverse market conditions where only specific subsets of indicators are reliable. The ever-changing nature of markets results in unpredictable observation functions, further underscoring our approach's relevance.\n  \n**Assumption Specificity:** We recognize that our assumptions represent distinct scenarios but stand firm in their practical significance:\n\n- **Full State Information during Training:** Though this assumption is seemingly idealistic, asymmetric RL for POMDPs [1-2] and RL for hindsight observable MDPs [3] have effectively leveraged this for addressing challenges, e.g., self-driving vehicles [1].  \n\n- **Employing Masks for Simulating Diverse Observation Functions:** \n  1. Masks effectively capture some real-world partial observable scenarios, such as robotics where sensor occlusions can be modeled as feature masks.\n  2. Traditional POMDPs, as described in [3-6], utilize a consistent observation function that maps the full state to a partial one, often masking a predetermined set of factors. In contrast, our approach introduces unpredictability by avoiding assumptions about which factors are masked during testing. This results in diverse and unpredictable mapping functions during evaluation.\n  3. To represent this diversity, we employ two specific scenarios (refer to lines 235-241). It's essential to note that **our generated observation functions are not equivalent to a straightforward  stochastic Bernoulli function.** Our practical method of sampling multiple mask observation functions is explained in lines 74-80 of the Appendix. The complex interplay of differing missing ratios, distinct scenarios, and variable masked sections surpasses what can be captured by a singular Bernoulli distribution, highlighting the need for robust generalization.\n\n- **Awareness of Mask Vectors:** While we recognize that our assumption may not hold universally, it's pragmatically valid in many contexts and aligns with existing research conventions.\n  1. Practical domains like robotics and network systems frequently allow the detection of 'missing' or anomalous data, evidenced by discernible sensor disturbances or notable anomaly indicators.\n  2. Our approach resonates with numerous established works in the POMDP realm [4] and studies addressing data missingness [7-8], which operate under similar assumptions regarding the awareness of mask vectors.\n\n\n\n# Addressing Concerns in Method Description: Proposed Revisions\n\nThank you for feedback. We are committed to refining our paper:\n1. We acknowledge discrepancies between our introduction, abstract, and Section 3. We will harmonize these parts to clearly depict our method's capabilities and constraints, ensuring the work's scope is evident from the start.\n2.  We'll clarify the rationale behind our chosen test settings, including our decision to use masked observation functions, the diversity of observation functions, and the accessibility to states during training (as elaborated in the above response).\n3.  For overarching clarity, we will comb through the manuscript ensuring the problem, method, and results' scope are consistently and lucidly presented.\n\n\n\n\n# Addressing IQL_R Performance and Hyperparameter Tuning Concerns\nWe have provided a more detailed explanation, along with supplementary results and hyper-parameter selection tables, in the accompanying pdf file in the global response panel.\n\n# Distinguishing Our Method from Hindsight State Information [3]\nWhile there are similarities, key distinctions also exist. Our method investigates uncertain and diverse mask observation functions in an offline mode, in contrast to the online, singular observation function in hindsight state information. For clarity, we've summarized these differences in the Table 3 in the attached pdf:\n\n# Other Queries:\n- **Online POMDP in Offline RL:** Online RL is optimized for live interactions. Without adjustments, their direct application to offline RL can lead to poor performance [9].\n- **Access to MDP:** None. This will be clarified in the revision.\n- **Use of VQ-VAE:** VQ-VAE stands as SoTA in discrete representation learning, justifying its incorporation.\n- **'g' Defined:** Refers to the nearest lookup, as noted on line 198.\n- **Choosing t-SNE:**  According to [10], t-SNE ensures that similar objects align closely, while dissimilar ones are positioned farther apart, making it suitable for visual similarity comparisons.\n\n\n\n1.  Robust asymmetric learning in pomdps. ICML 2021.\n2.  Leveraging fully observable policies for learning under partial observability. CoRL 2022.\n3.  Learning in POMDPs is Sample-Efficient with Hindsight Observability. ICML 2023.\n4.  Deep recurrent belief propagation network for POMDPs. AAAI 2021.\n5.  Deep variational reinforcement learning for pomdps. ICML 2018.\n6.  Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs. ICML 2022.\n7.  GAIN: Missing Data Imputation using Generative Adversarial Nets. ICML 2018.\n8.  Variational Selective Autoencoder: Learning from Partially-Observed Heterogeneous Data. AISTATS 2021.\n9.  Off-Policy Deep Reinforcement Learning without Exploration. ICML 2019.\n10.  Visualizing Data Using t-SNE. 2008"}, {"review_id": "WND3aRQuP1", "reviewer": "Reviewer_D69B", "summary": "This paper proposes ORDER, an offline RL solution where the agent have access to states in the offline data, but can only witness observations where some groups of dimensions are masked when deployed due to occlusion or perturbation in real-life scenarios. To address this problem, a 3-step solution is proposed. First, a state encoder is trained to convert the states into discrete representations, which is essentially multiple VQ-VAEs grouped by observability. Then, the RL agent is trained with IQL on the offline dataset with state dimensions randomly blocked, using representations produced by the encoder as observations. Finally, a proxy state encoder is trained to estimate the current discrete representation based on the past trajectory and current state in deploy time, and its output is used for the trained RL agent. On several mujoco testbeds, ORDER is proved to work better than the baselines and gives representation with better alignment to ground truth.", "questions": "I have one question below, apart from those in the weakness section:\n\nFrom line 196, we know that one encoder only takes one state factor into account; meanwhile, from line 14-15 in the appendix, we know that each dimension is seen as a state factor. Does this mean that the embedding is generated assuming that (groups of) dimensions are independent to each other? If it is, then there is a problem: the agent perceives the same environment, which should not be independent on different sources (with different mask of visibility).\n\nBelow are my suggestions for the author. I will be happy to increase my score if the authors address those problems (see weakness sections for details on point 1-4):\n\n1. Improve the related work section; \n\n2. Present more ablation results;\n\n3. Append the computational resource consumption, negative societal impact and licenses;\n\n4. Fix the other minor problems;\n\n5. I strongly advice the authors to open source upon the acceptance (or next submission) of the paper.\n", "limitations": "The paper does a good job in discussing the limitations, and I am convinced that despite of those limitations, the work is still valuable to the RL community; however, the paper does not mention any potential negative societal impact of the work. For example, while mujoco is still far from real-life applications, works on automated decision making from lab environment to real life is possible to substitute human work and cause job loss.  ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThis paper is clearly written and easy to follow. Figure 1, 2 and 3 effectively summarizes the main idea of the algorithm, helping the reader to understand. The math symbols are organized and the motivation for each component such as the use of discrete representations are well-explained.\n\n2.\tThe solution proposed by this paper is sound, novel and makes much sense, especially the discrete representation part. From my understanding, there could be two advantages of using discrete representation over continuous: 1) the training of VQ-VAE automatically encourages different values of states to be clustered into one representation, and thus becoming more robust to estimation by the proxy representation, and 2) the RL agent works better on the discrete state space.  \n\n3.\tThe experimental results is neat and clearly shows the superiority of ORDER to other baselines; for example, the visualization result in Fig. 6 effectively shows why discrete representation is used. \n", "weaknesses": "**1.\tThe related work section could be improved.** \n\nA)\tThe reference list at a single point is too long (line 53, 55 and 78), while the occurrence of reference is too few in the section. The referenced related work is not (only) a proof that authors have read many papers, but (also) should summarize the prior works with a taxonomy that helps the reader to understand the main lines of work and the position of this work in the context. For example, in line 78, “both empirical and theoretical studies …” should be “both empirical […] and theoretical […] studies …”; in line 93-97, point i) and ii) should each have a reference list; in line 103-104, a list of example should be made about “specific partial observation” with reference list for each kind.     \n\nB)\tThere should be a brief discussion on sim2real in the robotics community, since the motivation of the paper is to train the agent in a controlled laboratory environment (quoted from line 34) and delopyed in real-world disaster scenarios (quoted from line 37).\n\n**2.\tThe experiment section could be stronger.**\n\nA)\tMore clarification could be made about the second point in the strength. Specifically, is the performance better because of better representations, or better because IQL works better in the discrete state space? \n\nB)\tMore ablation could be added. For example, what if multiple dimensions are in one group of visibility (I assume the hyperparameters for VQ-VAE would change)? What is the effect of the dimension and the number of discrete codes of the codebook of VQ-VAE? What if the dataset is not medium, but rather a mixture of expert and random?\n\n**3.\tThe computational resource consumption, negative societal impact and licenses are not specified.**\n\n**4.\tOther minor problems:**\n \nA)\tLine 285, bseline -> baseline;\n\nB)\tLine 60 in the appendix, GRU should be cited;\n\nC)\tI would suggest the authors to give clearer, more intuitive explanation at the beginning of the paper about why discrete rather than continuous representation is used. Currently, the reason for using discrete representation is only stated at the end of introduction, and is only empirical without intuition.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer:\n\nThank you for your thoughtful questions and feedback on our paper. Below, we provide detailed responses to each of your questions:\n\n# Q1. The embedding is generated assuming that (groups of) dimensions are independent to each other?\n\nWhile our model considers individual state factors as described in line 196, it doesn't inherently imply independence across these dimensions. Here's why:\n\n- **Embedding Concatenation:** Referencing Equation 2 and Equation 4, the embedding concatenation mechanism amalgamates information across all dimensions. This ensures integrated representation and counters the notion of isolated dimensions.\n\n- **Trajectory Encoder Integration:** As highlighted by Equation 5 and elaborated in line 252, our trajectory encoder, denoted as \\(\\xi\\), incorporates historical data from all dimensions when forming current proxy representations. This further solidifies the interdependence among dimensions.\n\nWe hope this provides clarity regarding our model's treatment of state dimensions. The interconnectedness of dimensions, ensured by our design choices, aptly captures the essence of agents perceiving a unified environment, regardless of mask variations.\n\n\n# Improvements to the Related Work Section\n\n**A. Organization of References:**\n\nWe value your constructive feedback on refining our Related Work section. To enhance the readability and utility of our related work section:\n\n1. **Reference Consolidation:** We will reduce long lists of references at a single point, ensuring that they are spread more evenly and are contextually relevant.\n2. **Enhanced Summarization:** We aim to provide a more structured summary of prior works, incorporating a taxonomy that facilitates a clearer understanding of the existing literature landscape.\n\n**B. Discussion on Sim2Real:**\n\nThe term \"sim2real\" embodies the transition from models and algorithms optimized in a simulated environment to their application in real-world settings. This transition is particularly vital in robotics and automated systems where real-world conditions can be unpredictable and varied. In the context of our paper, while we predominantly focus on simulated environments, the end goal is to ensure that our findings are not just theoretically sound but also practically viable in real-world scenarios. Understanding and bridging the \"sim2real\" gap is a crucial step in this direction. We will incorporate a dedicated subsection that discusses prominent contributions from the robotics community on \"sim2real\". This will better position our work in the broader context and underline its relevance.\n\n# Improvements to the Experiment Section\n\nThank you for highlighting the need for clarity on the strengths of our discrete representation approach.\n\nThe improved performance is attributed to two key factors:\n   -  VQ-VAE's training inherently promotes clustering of diverse state values into a unified representation, enhancing the robustness of the proxy representation.\n   - A discrete state space, being more constrained than its continuous counterpart, benefits offline RL algorithms. Such a setup increases the likelihood of encountering similar states during training, thereby mitigating the out-of-distribution (o.o.d.) challenge.\n\nWe'll ensure that the experiment section further elucidates these distinctions. Your feedback is pivotal in sharpening our paper's clarity, and we're grateful for it. \n\n# Additional Ablation Studies \n\nDue to time constraints and space limitations, we couldn't present new ablation results currently. However, we're committed to providing more comprehensive results in the revision:\n   1) Ablations for cases where multiple dimensions fall within a single group.\n   2) Ablation of the dimensionality of the discrete code (note that variations in the number of discrete codes have already been showcased in Figure 6 (d)).\n   3) Expanded results across different datasets, including expert, medium-replay, and medium-expert datasets.\n\n# Computational Resources and Timeframes\n\nAll experiments were executed on Ubuntu 20.04, utilizing an Nvidia Tesla A6000 GPU. On average, training on a dataset required 2732.63 seconds, while inference for one episode (with a maximum of 1000 timesteps) took about 8.17 seconds. We will offer a detailed breakdown of these costs for each task in our revised submission.\n\n# Societal Impact Discussion\n\nWhile our research is grounded in the controlled Mujoco environment, the implications of transferring these findings to real-world settings merit consideration. Automated decision-making systems, if universally adopted, might reduce human roles in certain sectors, potentially leading to job displacements. It's crucial to strike a balance, ensuring technological advancements align with societal welfare, and to approach real-world deployments with an awareness of these broader implications.\n\n# Addressing Minor Concerns\n\nWe'll correct the typo on Line 285 and add a GRU citation on Line 60 in the appendix.\n\n   - **Clarification on Discrete Representation**:  In the revised version, we'll provide a more intuitive explanation early in the introduction: Imagine the difference between sorting colors into broad categories versus matching exact shades. The former, like a discrete state space, makes generalization easier by grouping similar items together. This is beneficial for offline RL algorithms, as it heightens the chance of re-encountering similar states during training, effectively addressing the out-of-distribution (o.o.d.) challenge inherent in continuous spaces.\n\n# Commitment to Code Release \n\nWe will release our implementation after acceptance."}]

Author rebuttal:
**Response to Reviewers:**

Dear Reviewers,

Thank you for your comprehensive feedback on our manuscript. We have addressed each comment individually in the subsequent sections. Additionally, an attached PDF provides further clarifications with relevant tables.

Your insights are crucial for refining our work, and we trust our responses and modifications align with your suggestions.

Looking forward to your continued feedback.

Best regards,

Authors

Meta-review: The overall assessment is that the paper made a solid contribution. The primary debate was about the generality of the problem being considered and how well it mapped to real-world tasks. This is certainly a point that can be discussed more thoroughly in the paper. However, that potential weakness is well balanced by the interesting approach and strong experimental results. 

Please take the critical feedback into account when preparing the camera ready paper.
Decision: Accept (poster)