S2P: State-conditioned Image Synthesis for Data Augmentation in Offline Reinforcement Learning
Authors: Daesol Cho
Dongseok Shim
H. Jin Kim
Venue: NeurIPS 2022 Accept (2022)
Keywords: Offline Reinforcement Learning
Generative Model
Data Augmentation

Abstract: Offline reinforcement learning (Offline RL) suffers from the innate distributional shift as it cannot interact with the physical environment during training. To alleviate such limitation, state-based offline RL leverages a learned dynamics model from the logged experience and augments the predicted state transition to extend the data distribution. For exploiting such benefit also on the image-based RL, we firstly propose a generative model, S2P (State2Pixel), which synthesizes the raw pixel of the agent from its corresponding state. It enables bridging the gap between the state and the image domain in RL algorithms, and virtually exploring unseen image distribution via model-based transition in the state space. Through experiments, we confirm that our S2P-based image synthesis not only improves the image-based offline RL performance but also shows powerful generalization capability on unseen tasks.

S2P: State-conditioned Image Synthesis for Data Augmentation in Offline Reinforcement Learning
Daesol Cho"
Dongseok Shim' Seoul National University
Scoul National University
Automation and Systems Research Institute (ASRI)
Interdisciplinary Program Al
H. Jin Kim
Scoul National University
Artificial Intelligence Institute Scoul National University (AIIS) (dscho1234, tlaehdtjr01, hjinkim] ac
Abstract
Offline reinforcement learning (Offline RL) suffers from the innate distributional shift as cannot interact with the physical enviromment during training. To alleviate such limitation, state-based offline RL leverages learned dynamics model from the logged experience and augments the predicted state transition to extend the data distribution. For exploiting such benefit also the image- RL, firstly propose a generative model, S2P (Pxxte)Pixel), which synthesizes the raw pixel of the agent from its corresponding state. It enables bridging the gap between the state and the image domain in RL algorithms, and virtually exploring unseen image distribution via model-based transition in the state space. Through experiments, we confirm that our S2P-based image synthesis not only improves the image- based offline RL performance but also shows powerful generalization capability on unseen tasks.
Introduction
Deep learning algorithms have shown significant devel- opment thanks to the large pre-collected dataset, such as SQuAD [47] in natural language processing (NLP), and ImageNet [4 in computer vision. On the contrary, rein- forcement learning (RL) requires an online trial-and-error in training process to collect the data by interacting with the environment, which hinders its utilization in many real-world upplications. Due to this intrinsic property of current online RI. algorithms, there exist some approaches that try to deploy large and diverse pre-recorded datasets without online interaction with the environment, which is called offline RL.
Distribution
Vistual
Explonaioo
Escandal Disuibuiia
Image Distribution I
However, recent studies have observed that the current online RL algorithms 10 35) perform poorly in an offline setting. It is primarily attributed the large extrapola- tion error when the Q-function is evaluated on out-of- distribution actions, which called the distributional shift
Figure 1: S2P generates the dynamics- consistent image transition data by virtu- ally exploring in the state space to extend the distribution of the offline datasets.
25 22
That is, due the offline
Equal contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

setting that limits online data collection, the offline RL has struggled to generalize beyond the given offline dataset. Even though some offline RL methods [26] [4 54] achieve reasonable performances in some settings, their training is still limited to behaviors within the given offline dataset distribution, and detours the evaluation on out-t--distiiuttion data rather than directly addressing such an empty space of the offline dataset. Thus, there exists a growing need for the development of algorithms spe- cialized to directly address such out-of-distribution data by extending the offline dataset distribution's support.
To alleviate the fixed dataset distribution problem, recent studies propose some data augmentation strategies. In state- based RL. model-based algorithms [2] [ 27 [6] which learn dynamics model from the pre-recorded dataset and augment the dataset with generated state transitions have emerged as a promising paradigm. As the model- based approach trains dynamics models in a supervised manner, it allows stable training process and generates reliable state transition data for augmentation. Thus, it can be plausible choice that it enables the generalization into the unseen state-action by performing dynamics-consistent planning on unseen state distribution.
When it comes to the image domain, however, there is still no augmentation strategy to miligate the aforementioned distribution shift. Even though some model-based image RL methods [2 that propose to learn latent dynamics using reconstruction error from ELBO objective [8]] [ 23 can be exploited for generating image transition data similar manner to the state-based methods, the quality of the output images from these approaches is not satisfactory because 1) their focus is on learning latent representation suitable for the RL network's inputs rather than generating high-quality and accurate images, and 2) BO-basedd objective cannot generate photo-realistic images compared to other generative models such as GAN p or diffusion [14 41 and it usually produces blurry outputs. Above all, 3) model-based image RL only exploits image input and it makes the generative model fail to capture the accurate dynamics and the details of the agent's 's posture or objects in the image 40 42 These undesired properties discourage offline RL algorithms from adding the reconstruction output of model-based image RL to their training data as an augmentation strategy.
Therefore, we propose S2P (State2Pixel) which utilizes multi-moda input (the state, and the previous image observation of the agent) to synthesize the realistic image from its corresponding state. The key element of S2P is a multi-modal affine transformation (MAT) which effectively exploits both state and image cross-modality information. Unlike previous learned affine transformation [19 21 44 34, which leverages single domain input, MAT fuses the mross-modal representation from the state and the image to produce the scale and the bias modulation parameters. This multi multi-modality of S2P makes it possible generate the dynamics-consistent images from the reliable state transition while preserving high-quality image generation capability.
To sum up, our work makes the following key contributions.
We propose a state-to-pixel generative model (S2P) which generates tynamicc-conssstere image and multimodal affine transformation (MAT) module for aggregating cross-modal inputs.
To the best of the author's knowledge, this work the first to propose image augmentation for offline image RL and overcome innate fixed distribution problem by implicitly leveraging reliable state transition.
We evaluate S2P on the DMControl 52 benchmark environments with the offline setting, and it results in 1.0 - 3.0x higher offline RI performance by augmenting the generated synthetic image transition data.
Even with the state distributions of the unseen tasks, S2P can generalize to unseen image distribution, and we show that the agent can be trained by offline RI. with these generated images only.
2
Related Work
2.1 Image Synthesis
Generative Adversarial Networks (GAN) 99 based deep generative models enjoy huge success in synthesizing high-resolution photo-realistic images via style mapping function and the learned affine transformation 19-2 (4] StyleGAN [9 firstly utilizes a style vector and Adaptive Instance
2

---

Normalization (AdaIN) [5 in the generative networks to disentangle the latent space and control the seale-specific synthesis. Following studies [00 pinpoint that the AdaIN operation which leads to information loss in the feature magnitude makes undesired droplet-like artifacts in the synthesized images and proposes weight demodulation by assuming the variance of input features. SPADE [44] proposes an architecture to synthesize the image using its corresponding semantic masks and spatially Image Affine Combination Module (ACM) which enables the network to manipulate the images using text descriptions given by users. The difference between our proposed MAT and the previous studies is that we leverage cross-modal data, state and image, to estimate the modulation parameters for the learned affine transformation whereas other studies use a single data type such as text or image.
learned affine transformation. ManiGAN [34] suggests
2.2 Offline Reinforcement Learning & Data Augmentation
Offline RL 6 29 33 is the task of learning policies from given static dataset, which is different from online RL that learns useful behaviors through -nd---and in the environment. Prior offline RL algorithms are designed to constrain the policy to the behavior policy used for offline data collection
via direct state or action constraints [8] 37 maximum mean discrepancy (25), KI. divergence
54 159 17, or learning conservative critics 126 24). However, most of these methods are limited to
exploiting the state-action distribution of the given static dataset, rather than exploring and extending the distribution. As the offline setting prohibits online interaction with the environment, we suggest the synthetic data generation method to enable the offline RL agent to virtually explore and extend the distribution by leaving the support the dataset.
Recent works in model-based state RL that involves learning policy with dynamics model 2 28 57 suggest the need augment the data with generated transitions from the model
for extending the data distribution. Also, augmentation strategies in image domain 30 49
31 55 56
emphasize the importance of image augmentation for sample efficiency and robust representation learning. But, these works focus on purely image manipulation techniques on the given image such that use the variational model to train the latent dynamics are studied in similar concept to the state-based ones. However, the generated image from these methods is the byproduct of learning the effective image representation rather than the main purpose of these works, and as these methods only utilize the image input, it leads to missing objects or inaccurate dynamics of the agent in the reconstructed image. Therefore, to bridge the gap between the model-based state transition data augmentation techniques and image augmentation, we propose the method that generates dyyamiccs-onssstern image transition data along timesteps with multi-modal inputs.
as cropping rather than generating image transition. Some image-based methods [45] 11 12
Method
3.1 S2P Generator
Architecture. The goal of S2P (State2Pixel) generator is to synthesize the image i, which perfectly represents all the information of its corresponding state st. Unfortunately, a state-sole condition cannot formulate single deterministic rendered image because, in most cases, state does not provide the agent's 's position from the global coordinate, but rather from an egocentric coordinate. Also, image-based RL algorithms utilize sequential images to capture the agent's 's velocity using the change of the background, e.g. ground checkerboard, between input images. It means that the image of the current step I, is dependent not only on the current state 8t, but also the image of the previous step 4-11 We, therefore, build the generator G' to synthesize the image I, from both the state s; and the previous image I, that the generated image i, can preserve the consiseeyy in the physical environment.
i G(st,
At the first layer of S2P, the input image I and the state s, are projected to the feature space via convolution and MLP encoders respectively. Both features are then fed to the hierarchical generator block which consists of several residual connections [3] and the upsampling layer. After passing through each generator block, the spatial size of the feature map is doubled while the channel dimension is halved. The image features are converted to the RGB images at the last layer of the generator with single convolution layer.
3

---

N W
HxHxx
MAT meddlle
MAT
4-1
(*(()))
State encodar
loss
State- mapping
W;
Adversaniol
Discrimimaoe
loss
Figure 2: overview S2P architecture. State s and the previous image 4-1 are used input to generate current step image It. The spatial size of the features gets larger as it passes through multiple upsampling generators. G and MAT indicate the generator block and the Multimodal Affine Transformation respectively
We observe that the input signals, i.e. state s; and image 4-1, become attenuated as they pass through deeper generating layers and the network produces images with poor quality, So, similar to recent style-based image synthesis algorithms 21 44 34), we adopt learned affine transformation architecture to inject auxiliary signals to the generator. The difference between the previous style- based generative models and S2P is that we propose a multimodal affine transformation (MAT) to produce the learnable modulation parameters, y and B, with the cross-modalit, representation via multimodal feature extractor and ttatetto-latent mapping function. The overall architecture of our proposed S2P is depicted in Figure?) 2
non-linear latent mapping function g S W which is implemented as 8-layer MLP produces a latent code w E W from the given state in the state space S. The latent code w is spatially expanded as the same size of the input feature of MAT module x, and the conditioned image 4,-1 is also linearly interpolated to make its size same as x. The spatially expanded w and the resized 4-1 are channel-wisely concatenated and fed the multimodal feature extractor fuse the state and image cross-modes cross-modality representation. Each estimator then produces the learnalle scale and bias B for effective cross- moosl affine transformation.
Finally, our proposed MAT operation is defined as:
(x(x)) (xt)
y' = (w,
+ B (w, 4-11),
where pe(+) and (-(-) are the channel-wise mean and the standard deviation of the input feature of MAT x² from the 7th block of the generator, and denotes Hadamard element-wise product. The design of MAT is illustrated in Figure
In addition, it is shown that the neural network is biased toward learning a low frequency mapping and has difficulty in representing a high frequency information [46, (9]] To mitigate such undesired tendency, do not naïve state vector input, but employ positional encoding with the high frequency function db + 2L which is defined as:
(x) = (sin (22mm22))) Tw)) cos(2 ππ), ... sin(2 TT), co -x))),
(3)
where r indicates each component of state vector
utilize multi-scale discriminators in [53 increase the receptive field without deeper layers or larger convolution kernels for alleviating the overfitting. Two discriminators with identical architecture are adopted during training, and the synthesized and real images which are resized to several spatial sizes fed to each discriminator.

---

Loss Function. Our S2P generator is trained by linearly combined multiple objectives. First, we leverage pixel-wise L1 loss between the output of the generator 7 and the real image 7,
L = 1 Iellı-
(4)
In addition to the pixel-wise loss, we also utilize the ImageNet [48] pre-trained VGG19 [50] net- work to calculate the perceptual similarity loss pper between two images,
Mulinodui Scare calnecer
Biss exiiaaoo
3
Pper = 110 - (1)
(5)
Expasion
Scale esteratio
where (( denotes the i-th layer of VGG19.
We implement the adversarial objective Lade for both S2P generator and multi-scale discrimina- tors same Hi (53). The difference that we replace the least square loss with the hinge-based based loss [66] and condition state information to the discriminator D so that the generator is induced produce the dynamics- consistent outputs.
X 1t'x 111"
Nocmatiatioon
Charnd-aii
if Wix
Figure 3: Multimodal Affine Transformation (MAT) module.
D = mi[mmi( -1 1 + (81, TJ)) Elmin(0, -1 (s, (.)), LG = -R[---((s 1.)
(6)
Lado La LnD
The total loss function to optimize the S2P generator can be defined as:
L = A1 + X2h + AzCades
(7)
where Å1 A2. and n the hyperparameters balance among the objectives.
3.2 Offline reinforcement learning with synthetic data
We consider the Markov decision process (MDP) M = (Z, 7).
where I denotes the
image space, S the state space corresponding to I, A the action space, T's''s. ss a) the transition dynamics, r(s, a) the reward function, Po the initial distribution, and the discount factor. We denote the discounted image visitation distribution of policy using (T)
?) E PII I|x), where P44 = I|²) is the probability of reaching image observation I at time by using π in M. Similarly, we denote the image-action visitation distribution with d (I, a) (T) (a))t
The objective of RL is to optimize a policy T (a]/) that maximizes the expected discounted return J ((x)
1 ((t.a)-d [s).
In the offline RL setting, the algorithm has access {ll,, 8t. Ce, Tt, St+1)) collected by unknown behavior policy TB. In other words, the dataset policy using the static dataset without online interaction with the environment.
static dataset D
obtained from (I, a) == d (a))) and the goal is to find the best possible
To utilize the dynamics-consister consistent transition data for augmentation in offline RL, take the model- based approach that trains an ensemble of dynamics and reward model To rls,a), which outputs the predicted next state, and reward f(s,a). Once a model has been learned, we can construct the learned MDP M S,, f, Po. 7), which has the same spaces, but uses the learned dynamics and reward function. Naively optimizing the RL objective with the M is known to fail in the offline RL setting, both in theory and practice 22 57), due the distribution shift and model-bias. To overcome these, we take an uncertainty estimation algorithm like bootstrap ensembles 43 38), and obtain u(s, a), estimate of uncertainty in dynamics. Then could utilize the uncertainty penalized reward f(s, a) f(s, a) - Xu(s, a), where À is hyperparameter. We consider the following uncertainty quantification method that the maximum learned variance over the ensemble, u(s,a) a) max; (1, [[s,a]]l= F) [57 22].
As final process, train offline RL with the following hybrid objective J(a) I-y (( [r(s.a) a)), where dy(T,a) sd"a(T + Ad Nd, f E [0, 1] is the
5

---

Table 1: Offline RL results for DMControl. The numbers are the averaged normalized scores proposed in 77. where 100 corresponds to expert and corresponds the random policy. The results with standard deviation are in Appendix ??.
IQL
IQL
CQL
CQL
SLAC-off
SLAC-off
Environment
Dataset
+S2P
+S2P
+S2P
cheetah, walker, walk cup, catch reacher, easy linger, spin cartpole, swingup cheetah,
random
10.28
12.64(16.21)
4.89
11.77(7.52)
16.37
18.14(35.38)
random
40.28
4.03(0.83)
-0.43
10.44(1.99)
18.23
17.38(20.15)
random
74.77
82.28(80.39)
84.87
92.81(91.11)
70.04
85.57(52.81)
random
33.75
70.45(55.33)
52.32
75.01(81 .48)
77.43
85.76(87.84)
random
-0.17
0.46(-0.11)
-0.01
-0.11(0.07)
30.24
27.62(32.65)
random
24,52
38,59(29.1)
27.67
32.93(42.12 12)
35.03
31.01
mixed
41.68
88.53(70.44)
92.63
93.16(93.488
16.63
26.39(24.42)
walker, walk ball cup, catch
mixed
96.07
95.49(97.80)
97.18
97.84(98.00 70)
29.02
92.60(67.00
mixed
41.94
37.79(40.65)
30.82
51722
28.54
40.41(32.88)
reacher, easy finger, spin cartpole, swingup cheetah, run walker, walk ball in cup, eatch reacher, easy finger, spin cartpole, swingup
mixed
66.88
75.61(75.01)
70.37
75.53(77.54)
62.49
63.59
mixed
98.18
94.78(98.65)
98.54
87.17(80.07)
64.41
(8331()3.29)
mixed
14.49
14.04($1.25)
14.76
4.66(36.99)
14.51
16.36(25.41)
expert
79.89
87.18(88.89
94,20
96.28(95.54)
8,92
(4.41(8.42)
expert
94.34
94.97(94.35)
95.43
97.97(98.47)
11.71
70.95(19.66 (19.66)
expert
28.57
28.60(28.48)
28.42
28.62(28.68)
28.56
38.87(28.69
expert
52.13
58.19(57.51)
57.68
32.54(48.46)
26.61
42.85(34.49)
expert 98.42
94.42(99.19)
73.07
97.25(99.51)
24.75
81.05(52.21)
expert
20.43
18.37(18.03)
19.35
18.54(30.22)
14.11
11.18(-3.80)
ratio of the datapoints drawn from the offline dataset D, and 17(-1s) is the state rollout distribution used with the trained dynamics ensemble model Te, and M same as M except that the reward is (I, a) can obtained by rollout obtained state transitions {(Se; ae, Ft: St4 ) T into {(I,, aes T2, I+++))) N by using the trained S2P generator in Section 3.1 For implementation. we collect synthetic image transition data in separate replay buffer modee and train the agent by any offline RL algorithms with the sampled mini-batches from D and Model by the ratio of f and - The overall algorithm and more training details are summarized in Appendix B.
F(8, a) instead of fss a). Samples from d
M and convert the
Experiments
4.1
Environments & Data collection
We evaluate our method large subset of the dataset from the DeepMind Control (DMControl) suite [52]. It includes 6 environments, which were typically used for online image-based RL benchmarks. However, the best of our knowledge, all of these environments have never been properly evaluated in an offline image-basee based RL setting. The datasets in these benchmarks are generated as follows: random: rollout by a random policy that samples actions from uniform distribution. mixed train a policy using based SAC 10 until 500k steps for finger, cheetah and 100k steps for the others, then randomly samples trajectories from the replay buffer. 500k, 100k steps are minimum steps required to reach the expert level performance for each task. expert: train policy using state-based SAC. After convergence, we collect trajectories from the converged policy.
4.2 Offline Reinforcement Learning
To validate whether the S2P can help improve the offline RL performance, we evaluate our algorithm with recent offline RL algorithms like CQL [26] which utilizes conservative training of the critic, and
IQL (24] which trains critic by implicitly querying actions near the distribution of the dataset, and SLAC-off [32] which is ofate-of-the-ree
online image-based RL algorithm. We use this SLAC-off offline RL algorithms like BEAR (25), and behavior cloning, BCC The results for these two algorithms are Appendix To extend the offline RL into the image-based setting, we follow the image encoder architecture from (32 and train a variational model using the offline data. Then, we train the latent space this model.
with the offline setting. We also compare the policy constraint-baut

---

We include the offline RL results on different types of environments and data in Table 1 The results on the originally given offline dataset (50k) are shown in the left column of each algorithm, and the results on the $2P-based augmented dataset are shown in the right column of each algorithm. The S2P augments the same amount of the original offline dataset (50k). As it physically has more data (100k) than the original dataset (50k), for a reference, we also include the results on the 100k offline dataset in the parenthesis in Table Overall, the S2P-based method achieves better performance than the 50k dataset, even exceeding the 100k dataset's score in some tasks.
As the S2P answers how to generate the image transition data, we also have to consider where to generate the image transition data through S2P. Specifically, we use a random policy as 17([(s) is) in the mixed, expert dataset, and policy trained by the state-based offline RL as n7(1ss) the random dataset. These strategies are considered due to the following two assumptions. Firstly, as the random dataset only has random behavior, the dataset may not have any meaningfully rewarded states even with the augmentation by the random policy, especially in the locomotion environment (e.g. cheetah, walker cannot leave the initial states by the random policy). As the S2P's objective is to extend the distribution of the given dataset, the policy trained in an offline manner can help leave the support of the dataset compared to the naive random policy. Secondly, as the non-random datasets may have relatively biased state-action distributions that receive meaningful rewards compared to the random dataset, it is difficult to get out of the support of these datasets by the trained policy as most of the Acte-action induced by the trained policy included in the similar distribution of these datasets. However, the random policy can be effective as it can bring some exploration effects like noise injection or increasing entropy in 35 10].
To verify such assumptions, we analyze the ef- fect of different types of rollout distribution (7(ss) in offline RL performance (Table 2). We denote 50k dataset the results of the given original offline dataset, and +S2P(random 17) as the results of the data augmented by rollout with the random actions, and +S2P(offRL 77) as the results of the data augmented by rollout with the state-based offline RL policy. As expected, the random policy is more effective in the ex- pert dataset. Also, we could find the opposite phenomenon in the random dataset, and trade-off's between these two strategies in the mixed dataset.
50K
+S2P
+S2P
DATASRT
MRTHOD
DATASET
(RANDOM m))
(OFFRL
CREETAB
IQL
10.25
-0.107
12.64
COL
4.89
-0.69
11.77
RANDOM
SLAC
T6.37
11:94
CHEETAR
IQL
41.68
88.53
58.67
COL
92.65
93.16
MIXED
SLAC-OFF
16.63
26.39
26.53
CREETAH
IQL
79.89
87.18
79.20
COL
96.28
93.69
EXPERT
SLAC-OFF
8.92
14.41
7.79
Table Experiments on each different rollout dis-
tribution n in cheetah-rus -un environment.
4.3 Comparison with model-based image RL
50k
+S2P
+ Dreemeer
datuset
method
dataset
cheetah
IQL
41.68
88.53
2.09
Dreamer
CQL
92.63
93.16
58.93
mixed
SLACo=of
16.63
26.39
4,68
walker
IQL
96.07
95.49
1.28
walk
CQL
97.18
97.84
95.88
mixed
SLAC-off
20.02
92.60
$2.65
S2P
cheetah
IQL
79.89
87.18
73.23
CQL
94.20
96.28
53.69
expert
SLAC-off
8.92
14.41
3.65
walker
IQL
94.34
94.97
34.95
GT
walk
COL
95.43
97.97
expert
SLAC-off
11.71
70.95
$2.03
Figure 4: Qualitative comparison of S2P and Dreamer.
Table 3: Quantitative comparison of S2P and Dreamer.
To show why the multi-modal inputs are necessary for augmenting the image transition data in offline image RL, we compare our S2P with the previous model-based image RL algorithm, Dreamer [2]. as it can also reconstruct the images of the agent by training the reconstruction error from ELBO objective only using uni-modal inputs (previous images of the agent). For comparison, we trained Dreamer in an offline manner with the same dataset used for training S2P, and predicted future images with the episode context obtained from 5 consecutive ground truth images. Even though the Dreamer saw more previous steps' images compared to S2P (only single image of the previous
7

---

(a)
t=0
[=
t=2
t=
(b)
t=0
t=1
[=2
t=
GT
Cheetah
jump
π<
Baseline
Walker
run
+ PE
GT
SAT
No Ref
MAT
Figure 5: Qualitative results. We report the generator performance by synthesizing multiple steps with single trajectory. (a) demonstrates the effectiveness of each component in the S2P generator where PE, SAT, and MAT indicate positional encoding, state affine transformation and multimodal affine transformation, respectively. SAT cannot perfectly estimate the correct location of the agent as there misalignment at the ground checkerboard compared to MAT. (b) represents unseen fask adaptation using the trajectories given from the state-level transition model. (c) shows that model can recover the posture the agent without any reference image. Additional qualitative results several environments provided in Appendix A.
step). it still has difficulty in generating accurate posture, which supports the S2P's advantages (Fig 4.. It is because the priority of the model-based image RL algorithms is learning the effective latent representation for RL tasks, and they do not utilize a broader source of supervision from the state inputs.
To prove that inatsinnconiitentt images from the model-based image RL method cannot improve offline RL performance the S2P level, we perform the same experiment in Table but replace the augmented images from S2P with images from Dreamer. We observe that the image augmentation from Dreamer even degrades the original RL performance in several tasks, and the performance improvements with S2P excels the augmentation from Dreamer by large margin in all baselines (Table Thus, we could say that the inaccurate posture and quality the images generated by the model-basee method trained with the uni-moda inputs (images) image transition data in the offline setting. More experiments and details of the augmentation are in the Appendix.
sufficient for augmenting
4.4
Ablation
To observe how each component the S2P contributes to the quality of the synthesized images, we perform ablation study on the model architecture of the S2P and show its qualitative results Figure 5| a). A baseline architecture without any contribution of our proposed method, i.e. positional encoding (PE) and multimodal affine transformation (MAT), shows the worst image quality. We report that a simple application of the high frequency function 1 to the input state 8, (PF) results in noticeable increase in image quality. We also address the necessity of the input image 4-1 to estimate the modulation parameters, S, for the learned affine transformation. We ablate the input 4;-1 in MAT and utilize only the spatially expanded input state which is called State Affine Transformation (SAT). The generator which replaces the MAT module with SAT has difficulty in exploiting dynamics information from the previous image I,-1 1 and the translation error is accumulated as the generator recurrentyy synthesizes the long-hong horizon trajectory. We can observe such dynamics inconsistency especially the locomotion tasks, e.g. cheetah, and walker, which expresses the velocity in images by the change of the checkerboard in the ground. Compared to SAT, our proposed MAT which leverages both the state 8, and the previous image 7, shows better performance in reconstructing not only the posture of the agent but also its consiiics background.

---

4.5 Zero-shot Task Adaptation
To validate whether the S2P can help the offline RL process in settings that require generalization to tasks that are different from the given dataset, construct two environments cheetah-jump and walker-run. In cheetah-jump, which is referred from [57], the agent should solve a task that is different from the original purpose of the behavior policy. Specifically, chuetae-run-mixed dataset to reward the cheetah to jump as high as possible. Then, we generate the image transition data from the states whose positions the S2P-based augmentation 's advantage in tasks that require generalization. By training with the relabeled reward, the agent with the S2P achieves higher return and learns to bounce back and forth to take leap higher (Figure ())) even though the batch data contain little jumping motion.
relabel the rewards in the
bigger than a threshold to validate
To investigate whether the S2P can generate unseen image distributions from unseen state distribution, we collect walker-run state dataset by the same way of other mixed datasets in Section [4. Then, we generate images from these unseen states by recurrently using the S2P generator and we apply offline RL on these generated image transition data. Even though the state distribution is totally different from walker-walk dataset the agent should instead of walk, not only the S2P successfully generalizes to the unseen image distributions (Figure (r))ure[](b)) but also the agent can be trained to run by offline RL only with these synthesized images beyond the expertise of the dataset (Table 4). More details are in Appendix B.
We attribute such satisfactory task generaliza- tion to the model architecture the S2P which leverages both 4-11 and 8 for synthesizing the
METROD
WALKER-KUN
METHOD
CHEETAR-JU JUMP
BATCH MEAN
572.94
BATCH MEAN
2,03
TOL
NIA
TOL.
I. The posture of the agent is deterministic with the sole-state condition and the background of the image such as the ground checkerboard is dependent both on the state and the previous im- age. Therefore, our S2P exploits the state input to generate the posture of the agent and exploits the image input to generate the background. It well shown in Figure (()) where we intention- ally replace all the image input with the zero
IQL+S2 COL
659,25
IQL+S2P
54.4
NIA
CQL
COL+SIP
652.19
CQL+SZP
47.2
Table 4: Average returns of the walker-run and cheetah-jump tasks. We include mean undis- counted return of the episodes in the batch data for comparison.
matrix Øh* x x3 during the inference phase. We observe that S2P still perfectly reconstructs the posture of the agent with its corresponding state only while it fails to recover the background and the ground checkerboard as we expected.
Conclusion
We firstly present the state-to-pixel (S2P) algorithm that synthesizes the raw pixel from its corre- sponding state and the previous image. As the augmentation paradigm of the S2P is generating connises- image transition data, we demonstrate that S2P not only improves the image- based offline RL performance but also shows powerful generalization capability on unseen tasks. Even though S2P shows promising results in offline RL by data augmentation techniques, the assump- tion that datasets consist of pairs of images and states is still a strong assumption. Thus, for future work, plan further extend the idea with more relaxed assumptions such as unpaired datasets or extend other baste applications to the image-based RL algorithms.
Acknowledgement
This research was supported by Institute of Information & communications Technology Planning & Evaluation (ITPP) grant funded by the Korea (ovemmen((MSIT) [NO.2021 10001-0--144, Artificial Intelligence Graduate School Program (Seoul National University)] and by Unmanned Vehicles Core Technology Research and Development Program through the National Research Foundation of Korea(NRF) and Unmanned Vehicle Advanced Research Center(UVARRC) funded by the Ministry of Science and ICT, the Republic of

---

References
[1] Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models, Advances in Neural Information Processing Systems, 31, 2018.
[2] Mare Deisenroth and Carl E Rasmussen. Pileo: A model-based and data-efficient approach policy search. In Proceedings of the 28th International Conference on machine learning (ICML-1I), pages 465 472. Citeseer, 2011.
[3] Fei Deng, Ingook Jang, and Sungjin Ahn. Dreamerpro: Reconstruction free model- reinforcement learning with prototypical representations. arXiv preprint arXiv:2110. 2021.
[4] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hieracchical image database. In 2009 IEEE conference on computer vision and puttern recognition, pages 248-255, leee, 2009.
[5] Vincent Dumoulin. Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. arXiv preprint 1rXiv:1610.07629, 2016.
[6] Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6:503-556, 2005.
[7] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.022) 2020.
[8] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without explo- ration. International Conference on Machine Learning. pages 2052 2062. PMLR, 2019.
[9] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Ghahramani, M. Welling, C. Cortes. N. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, vol- ume 27. Curran Associates, Inc., 2014. wtw.ttts://///// htttttpee:/prooreddiinig proceedings. neurips. cc/ wwwppcppppppaacacctccccccc..eeat 2014//i1ee/
www.ffcccccPPaee.pppppp pdf
[10] Tuomas Haarnoja, Aurick Zhou, Picter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[11] Danijar Hafner, Timothy Lillicrap. Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pages 2555-2565. PMIR, 2019.
[12] Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL htttt::t/pppprrreeeeeee net.
forum? 2 -1d-2110
[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778 2016.
[14] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 6851, 2020.
[15] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE International Conference on Computer Vision, pages 1,501-1510, 2017.
[16] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in Neural Information Processing Systems, 12:11111-, 2019.
[17] Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferenees in dialog. arXiv preprint /rXiv:/907.000 1907,00456, 2019.
[18] Michacl I Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul, An introduction to variational methods for graphical models. Machine learning, 37(2):183-233) 1999.
[19] Tero Karras, Samuli Laine, and Timo Aila. A style- generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14001-4100 2019.
[20] Tero Karras, Samuli Laine, Miik Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 81 10000030 8119, 2020.
10

---

[21] Tero Karras, Miika Aittala. Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances in Neural Information Processing Systems, 34, 2021.
[22] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadselll Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Sys- tems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, De- cember 6-12. 2020, virtual, 2020. URL https: httpp::/prroceednngs wwwurips.c/ppppppeeeoe wwtpwcpppppppcpctccttccccct..ttt hatppp/rz//0//haahh. Abstract.html
[23] Diederik P Kingma and Max Welling, Auto Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114. 2013.
[24] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06166 2021.
[25] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off off-policy g-lcarning via bootstrapping error reduction. Advances Neural Information Processing Systems, 12:11784-1179,
2019.
[26] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for of- fline reinforcement learning. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, Decem- ber 6-12, 2020, virtual, 2020. URL https: ptttope//proccedingg. www.rrcceddinggsnuuuus....... wwweurrpps.ccpppee///00/ /000:///20///haaa hash// 22AAE2225555122221111568888888
[27] Vikash Kumar. Emanuel Todorov, and Sergey Levine. Optimal control with learned local models: Appli- cation to dexterous manipulation. 2016 IEEE International Conference on Robotics and Automation (ICRA). pages 378-383 IEEE, 2016.
[28] Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL http://orrumn-dsSJJnbbbe
https://openrevile
[29] Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning, pages 45-73. Springer, 2012.
[30] Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning, pages 5639-5650. PMLR, 2020.
[31] Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data, Advances in Neural Information Processing Systems, 33, 2020.
[32] Alex Lec, Anusha Nagabandi, Picter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. Advances in Neural Information Processing Systems, 33, 2020.
[33] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005. 004444000..0643, 2020.
[34] Bowen Li, Xiaojuan Qi. Thomas Lukasiewicz, and Philip HS Torr. Manigan: Text-guides guided image manipula- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7880-7889, 2020.
[35] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL URL http:://axxxii http:///wxxvvv..g/./// abe/ 1509 02971
[36] Jac Hyun Lim and Jong Chul Ye, Geometric gan. CoRR, abs/1705.02294, 2017. URL hRt/ttp: http:/ttt:/arrt //rxiv org/abs/1705 02/5002944.00.4
[37] Yao Liu, Adith Swaminathan, Alckh Agarwal, and Emma Brunskill. Provably good batch off-policy reinforcement learning without great exploration. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florine Balcan, and Hsuan-Tiee Lin, editors, Advances in Neural Information Process- ing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6- 12, 2020, virtual, 2020. http:tttt://proeeeeeeeee Dar2366abeeaabc399533888a8 -acaaaatttttcaat----ttttttt.ttt
www.innne..c.m /eapripp.c//aappr//2///aass

---

[38] Kendall Lowrey, Aravind Rajeswaran, Sham M. Kakade, Emanuel Todorov, and Igor Mordatch. Plan online. learn offline: Efficient learning and exploration via based control. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
URL [ttttttts///perreeee www.::/wwe.eee.eereeeeoee.
[39] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing seenes as neural radiance fields for view synthesis. In European conference on computer vision, pages 405 421. Springer, 2020.
[40] Tung D Nguyen, Rui Shu, Tuan Pham, Hung Bui, and Stefano Ermon. Temporal predictive coding for model-based planning in latent space. In International Conference on Machine Learning, pages $130-8139.
PMLR, 2021.
[41] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-8171. PMLR, 2021.
[42] Masashi Okada and Tadahiro Taniguchi. Dreaming: Model-based reinforcement learning by latent imagination without reconstruction. In 2021 IEEE International Conference on Robotics and Automation (ICRA). pages 4209-4215. IEEE, 2021.
[43] Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. Advances in Neural Information Processing Systems, 31, 2018.
[44] Taesung Park, Ming Liu, Ting- Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially- adaptive normalization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2337-2346, 2019.
[45] Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offline reinforcement learning from images with latent space models. In Learning for Dynamics and Control, pages 1 1154-1668 PMLR, 2021.
[46] Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning, pages 5301-5310. PMLR, 2019.
[47] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.052: 05250, 2016.
[48] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211225 2015. doi: 10. 1007/s 263-015-0816
[49] Max Schwarzer. Ankesh Anand. Rishab Goel, R. Devon Hjelm, Aaron C. Courville, and Philip Bachman. Data-efficient reinforcement learning with self- pref-redietive representations. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. net
URL httl:/tttp//pppeeeeeet
[50] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv: 1409,1556, 2014.
[51] Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physiccs0004057, 2000.
[52] Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillierap, Nicolas Heess, and Yuval Tassa, dm__conrol Software and tasks for continuous control. Software Impacts, 100022, 2020.
[53] Ting-Chun Wang, Ming- Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEFE conference on compuder vision and pattern recognition, pages 8798-8807, 2018.
[54] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 1.001.113666 11361, 2019.
[55] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint wwXiv:2107.09644 2021.
[56] Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL htt.tttp://ooooreeeee.eett forum? ? Pid:dddd----a--- -
[57] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y. Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: basel- offline policy optimization. In Hugo Larochelle, Mare Aurelio
12

---

Ranzato, Raia Hadsell, Maria-Floring Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL / www.:nneerriiiioeooooo.o cttppipepppcppppee//2000 hash/
Abstract htmml
[58] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. arXiv preprint urXiv:2102.08363 2021.
[59] Wenxuan Zhou, Sujay Bajracharya, and David Held. PLAS:: latent action space for offline reinforcement learning. In Jens Kober, Fabio Ramos, and Claire J. Tomlin, editors, 4th Conference on Robot Learning, CoRL 2020, 16-18 November 2020, Virtual Event / Cambridge, MA, USA, volume 155 of Proceedings of Machine Learning Research, pages 1719-1735. PMLR, 2020. URL https://proceeddii mlr press/v15b/zhouse hhtml
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Section 4
(b) Did you describe the limitations of your work? [Yes] See Section 5
(c) Did you discuss any potential negative societal impacts of your work? [Yes] This work inherits the potential negative societal impacts of reinforcement learning. We do not anticipate any additional negative impacts that are unique this work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] We included code the supplemental materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section Jadd Appendix
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] See Section
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 4
(b) Did you mention the license of the assets? [Yes] See Appendix.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes]
We include the code & model in the supplemental materials.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "kKl_f3LmLZC", "reviewer": "Reviewer_bQSP", "summary": "This paper proposes S2P: a method for state-conditioned image synthesis for offline image-based RL. The authors motivate their method by the fact that offline RL suffers from distribution shifts in the low data regime, and augmenting datasets using a learned dynamics model has previously been shown to be effective for mitigating this distribution shift. The proposed method is evaluated on 6 continuous control tasks from DMControl by training a number of offline RL methods on the original dataset as well as an augmented dataset and comparing the difference in downstream performance.", "questions": "I believe that my greatest concerns are already made explicit in the \"weaknesses\" section above. Specifically, I'd like the authors to\n\n- Explicitly state key assumptions made + limitations of the method\n- Justify why simpler approaches are not sufficient given the assumption of (I,s) pairs in the dataset\n- Clarify whether Dreamer leverages the available state information in the comparison, and if not, what the justification for that is", "limitations": "The authors do *not* discuss potential negative societal impact of their work and do *not* explicitly discuss limitations of their method despite claiming so in the checklist. See my previous comments on limitations. I do not anticipate ethical issues.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Simple and intuitive extension of a state-based RL method to image-based RL.\n- The paper is generally easy to follow and the contributions are clearly presented.\n- Experimental evaluation demonstrates that synthetic transitions are valuable across a variety of algorithms, tasks, and data distributions. The comparison to conventional image augmentations in appendix is appreciated.\n\nWeaknesses:\n- The assumption that datasets consist of pairs of images and ground-truth states is obviously a very strong assumption that will not hold in a lot applications. This is not a problem in itself, but the authors don't seem to ever state it explicitly in the paper. I'd like to see this made absolutely unambiguous to readers.\n- Given the above assumption of (I,s) pairs, there are a number of conceptually simpler methods that may or may not suffice. For example, one could simply learn an I -> s predictor using supervised learning and then apply one of the existing state-based offline RL algorithms. A slightly more sophisticated method could learn the I -> s predictor, augment the dataset using a state-based dynamics model, and then learn a state-based policy on the augmented dataset, and several other such approaches come to mind. In light of this, I'd like the authors to justify their more complex approach, ideally backed by data although that may not be feasible during the rebuttal itself.\n- It is unclear if it is indeed the case, but it appears that Dreamer does not leverage state information at all in the comparison, although it is assumed available. A more fair comparison would be to also condition the RSSM of Dreamer on state. Judging by the results in appendix, it appears that data quantity is not a big factor in the poor performance of Dreamer, so such an experiment would help clarify whether the poor results indeed are due to the objective or lack of access to privileged state information.\n\n**Post-rebuttal:** The author response addresses my main concerns and I am willing to raise my rating with the expectation that authors include reviewer suggestions in a future revision.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "e0Qa6hg0hDa", "reviewer": "Reviewer_i7hg", "summary": "This paper suggests image synthesis method to improve data efficiency in offline RL.  In offline RL, we can only use stored data, but it is too small for improving performance. Thus, many researches try to learn model dynamics, and explore new states. For image based RL, they need observations as pixels. Thus, authors generate image data from new states with recurrent generative model from input state and previous observed image. ", "questions": "1. Standard deviation of scores in tables. \n2. In table1, they mix dataset with random and expert dataset. Then, in some tasks, mixed dataset improve performance rather than expert dataset. It means that expert dataset cannot show diverse state and random policy would be Thus, S2P can be not needed, and we can improve performance with simple augmentations or exploration with random policy or small perturbed policy.  \n3. How much do you need data to train S2P model? \n4. How can you evaluate the quality of synthesized images?  ", "limitations": "It is hard to check the quality of synthesized images without a proper metric for evaluation. \nNow, I cannot assure that this generative model is needed instead of other data augmentation methods without comparison. \nIf S2P does not improve performance significantly, it is not used because of the load of training for generative model. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\nTo generate data in offline RL, they try to use multi-modal generative model for enough information. \nS2P largely improves the performance in DMControl tasks. \n\nWeakness\nNot enough baseline, there are no comparison with other augmentation. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GwUqkcQI2yW", "reviewer": "Reviewer_WCLw", "summary": "This paper focuses on an offline reinforcement learning problem. A generative model S2P is proposed to generate image transition data by virtually exploring the state space while keeping the dynamics consistency. In detail, a multi-modal affine transformation module is proposed to utilize both state and previous image data to generate modulation parameters. The proposed method is evaluated on the DMControl benchmark, and thorough empirical results are given from aspects including offline policy learning, model-based method, i.e. Dreamer comparison, ablation study, and generalization performance.", "questions": "1. Why is high-quality and accurate synthesis image generation important than data generation in latent space? Given the generated synthesis image, the offline agent should also to learn how to encode the image, which may cause additional learning difficulty.\n2. What is the offline learning performance if only using state-based affine transformation instead of multimodal affine transformation, i.e. SAT vs. MAT?\n3. What is the performance comparison with other offline reinforcement learning method, e.g. COMBO [1], ROMI [2] in which a learned dynamics model are used for data generation?\n4. Why use the previous step image instead of the previous step state? By concatenating the current step and previous step state, the features change can also bring velocity which is the mean reason to introduce the image as in line  117:126.\n5. Why VGG encoder pre-trained on Imagenet can be used for RL tasks, especially DMControl image observation feature encoding in Eq. 5 since the two domains differ a lot?\n6. Is The baselines’ encoder different from S2P as mentioned in line 230:231? How do keep comparison fair?\n7. Why introducing more generated image data cannot bring distinct performance improvement as shown in Table 1?\n\n[1] Yu T, Kumar A, Rafailov R, et al. Combo: Conservative offline model-based policy optimization[J]. Advances in neural information processing systems, 2021, 34: 28954-28967.\n\n[2] Wang J, Li W, Jiang H, et al. Offline reinforcement learning with reverse model-based imagination[J]. Advances in Neural Information Processing Systems, 2021, 34: 29420-29432.\n\n\nUpdate after rebuttal\n\nAll questions have been addressed.", "limitations": "This submission is not related to a distinct negative social impact.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Originality**\n\nAlthough the model-based generation is commonly-used in reinforcement learning (RL) methods, the image-based offline RL problem is novel, and the authors proposed the first image-based offline data generation method. But the core module in S2P, i.e. the multimodal affine transformation (MAT) is not novel, which has been investigated in previous works [1], [2].\n\n**Quality**\n\nThe authors conduct detailed experiments to validate the proposed method S2P, and the results sound. It shows significant performance improvement in 6 image-based DMControl tasks. The authors also explore the influence of different data generation policies and give thorough visualization to help understand the image generation capability and the importance of each module in S2P. The overall empirical results are convincing. But there are still a few problems with the importance of multimodal input and other baselines, which are listed in the following Questions section.\n\n**Clarity**\n\nThe paper is well-written and easy to understand. The motivation is clear. Sufficient related literature is included. The experiment configuration details are given in the appendix which helps the further implementation.\n\n**Significance**\n\nThe underlying offline RL problem is important both in academic and industrial communities. The authors try to tackle an interesting image-based data generation problem. Although the core module MAT has been investigated in previous computer-vision community, the proposed S2P is still essential since the overall idea is simple and the performance improvement is significant. This encourages the RL community to explore more complex offline RL problems, i.e. with image input.\n\n[1] Li B, Qi X, Lukasiewicz T, et al. Manigan: Text-guided image manipulation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 7880-7889.\n\n[2] Park T, Liu M Y, Wang T C, et al. Semantic image synthesis with spatially-adaptive normalization[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019: 2337-2346.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Dn48CgkZG7U", "reviewer": "Reviewer_h9Ka", "summary": "This work extends the distribution of the data generating policy in offline RL by first learning a generative model (S2P) which synthesizes raw pixels given state information and a previous frame. This generative model is dynamics-consistent and employs multiple modalities which is a new contribution compared to the single modality models of previous works. They show that their data augmentation strategy improves SOTA algorithms like CQL, IQL and an offline version of SLAC. They conduct several ablations showing the value of different components they introduce such as position encoding, multimodal affine transformations, state only affine transformations etc. They also show that even on unseen tasks with unseen image distribution, just having the state distributions allows S2P to generate reliable images to train an offline RL algorithm. \n", "questions": "* I wanted to confirm if the baselines without and with S2P are trained with exactly the same number of transitions. If so, please clarify in the paper. Also, if the total frames are fixed, it will be good to know how the performance varies as a fraction of the mixing ratios. \n* There are a few very minor corrections in the paper that can be fixed in the final version if the authors do another pass over the paper  – eg. (“only single the previous image” in Sec 4.3). Section 3.2 claims that “naively optimizing a learned MDP is prone to failure”, but misses appropriate citations. \n* There is an interesting ablation where you zero out the image but keep the state intact in Figure 5c. Was the other ablation done ? i.e. to mask the state and only use the image to generate the next image. I understand that you present the Dreamer baseline for this case, but it would be nice to have the ablation on the same model to be more precise about the gains.\n* You have shown interesting ablations with SAT, MAT and positional encoding in Figure 5. Since you introduce three components of the loss, it will also be interesting to see what value each component adds in terms of generalization. \n", "limitations": "The authors have not addressed any specific limitations of the work. Please refer to the earlier replies for potential areas of improvement.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper proposes a state-to-pixel generative model that can be used to generate dynamics-consistent transition augmentations to any offline RL algorithms. The paper is clearly written, describes the contributions and how it differs from other similar works in the area. The paper also conducts extensive experiments with three different algorithms on the DMControl datasets. The results are solid and the ablations show the values of each of the algorithmic pieces that were added. I believe that the work is still significant and demonstrates that augmenting data with dynamics-consistency is significantly useful for any offline RL learning algorithm. Thus it is a useful contribution to the offline RL research.\n\nThe paper has conducted experiments limited to only DMControl with reliable state transitions. It would make the model claims stronger if one were to see if the same data augmentation strategy could be applied to other image observation environments and partially observable environments. This model relies on the fact that the state representations are reliable and learns to generate new transitions via “exploring” in the state space and generating corresponding image observations. Not all problems provide reliable state transitions and representations and this work has not fully explored cases where state representations/dynamics are intrinsically less reliable and noisy.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept