SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The video background in the environment incorporates grayscale videos from Kinetics-400, where pixels with high blue channel values are replaced. Camera jittering is introduced through a smooth random walk shift using Gaussian-perturbing acceleration, velocity decay, and pulling force. Sensor noise is added by perturbing a specific sensor based on the intensity of a patch in the background video. The perturbation involves adding the average patch value minus 0.5. Different sensors are perturbed for different environments. These sensor values undergo non-linear transformations, primarily piece-wise linear, to compute rewards. While the additive reward noise model may not capture sensor behavior perfectly, it is generally sufficient as long as the values remain within moderate ranges and stay within one linear region. (Note: the variants of Robodesk and DMC are not the contributions of this paper. We kindly refer readers to the paper of Denoised MDP [9] for a more detailed introduction.)
Computing Hardware We used a machine with the following CPU specifications: Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz; 32 CPUs, eight physical cores per CPU, a total of 256 logical CPU units. The machine has two GeForce RTX 2080 Ti GPUs with 11GB GPU memory.
Reproducibility We've included the code for the framework and all experiments in the supplement. We plan to release our code under the MIT License after the paper review period.
E.1 Synthetic Dataset
Hyperparameter Selection and Network Structure We adopt a similar experimental setup to TDRL [19], while extending it by decomposing the dynamics into four causally related latent processes proposed in this paper (refer to Equation 43). For all experiments, we assign  $\beta_{1} = \beta_{2} = \beta_{3} = \beta_{4} = 0.003$  as the weights for the KL divergence terms. In this particular experiment, we set  $\lambda_{1}$  and  $\lambda_{2}$  to 0 because the utilization of the ELBO loss alone has effectively maximized  $J_{\mathrm{RS}}^{t}$  and  $J_{\mathrm{AS}}^{t}$ , as illustrated in Figure 10. Here,  $J_{\mathrm{RS}}^{t}$  represents  $I_{\alpha_1} - I_{\alpha_2}$ , and  $J_{\mathrm{AS}}^{t}$  represents  $I_{\alpha_3} - I_{\alpha_4}$ . The network structure employed in this experiment is presented in Table 2.
Training Details The models are implemented in PyTorch 1.13.1. The VAE network is trained using AdamW optimizer for 100 epochs. A learning rate of 0.001 and a mini-batch size of 64 are used. We have used three random seeds in each experiment and reported the mean performance with standard deviation averaged across random seeds.
E.1.1 The identifiability socres for baselines
(a) DenoisedMDP
(b) DenoisedMDP
(c) BetaVAE
(d) BetaVAE
(e) FactorVAE
(f) FactorVAE
(g) PCL
(h) PCL
(i) SlowVAE
(j) SlowVAE
(k) TDRL
(1) TDRL
Figure 9: The identifiability socres for baselines in the experiments on synthetic data set. It can be observed that all baselines do not enjoy the property of block-wise identifiability.
Table 2: Architecture details. BS: batch size, T: length of time series, o_dim: observation dimension, s_dim: latent dimension,  ${s}_{t}^{ar}$  _dim: latent dimension for  ${s}_{t}^{ar},{s}_{t}^{\bar{a} r}$  _dim: latent dimension for  ${s}_{t}^{\bar{a} r},{s}_{t}^{\bar{a}\bar{r}}$  _dim: latent dimension for  ${s}_{t}^{\bar{a}\bar{r}},{s}_{t}^{\bar{a}\bar{r}}$  _dim: latent dimension for  ${s}_{t}^{\bar{a}\bar{r}}$  (s_dim =  ${s}_{t}^{\bar{a} r}$  _dim +  ${s}_{t}^{\bar{a}\bar{r}}$  _dim +  ${s}_{t}^{\bar{a}\bar{r}}$  _dim +  ${s}_{t}^{\bar{a}\bar{r}}$  _dim), LeakyReLU: Leaky Rectified Linear Unit.
Configuration	Description	Output
1. MLP-Obs-Encoder	Observation Encoder for Synthetic Data
Input: o1:T	Observed time series	BS × T × o_dim
Dense	128 neurons, LeakyReLU	BS × T × 128
Dense	128 neurons, LeakyReLU	BS × T × 128
Dense	128 neurons, LeakyReLU	BS × T × 128
Dense	Temporal embeddings	BS × T × s_dim
2. MLP-Obs-Decoder	Observation Decoder for Synthetic Data
Input:帽子1:T	Sampled latent variables	BS × T × s_dim
Dense	128 neurons, LeakyReLU	BS × T × 128
Dense	128 neurons, LeakyReLU	BS × T × 128
Dense	o_dim neurons, reconstructed帽子1:T	BS × T × o_dim
3. MLP-Reward-Decoder	Reward Decoder for Synthetic Data
Input:帽子1:T	Sampled latent variables	BS × T × s_dim
Dense	1 neurons, LeakyReLU	BS × T × 1
4. Disentangled Prior for star	Nonlinear Transition Prior Network
Input	Sampled latents and actions sr1:T, a1:T	BS × T × (sr1:-dim + a_dim)
Dense	star -dim neurons, prior output	BS × T × star -dim
5. Disentangled Prior for star	Nonlinear Transition Prior Network
Input	Sampled latent variable sequence sr1:T	BS × T × sr1:-dim
Dense	star -dim neurons, prior output	BS × T × star -dim
6. Disentangled Prior for star	Nonlinear Transition Prior Network
Input	Sampled latents and actions s1:T, a1:T	BS × T × (s_dim + a_dim)
Dense	star -dim neurons, prior output	BS × T × star -dim
7. Disentangled Prior for star	Nonlinear Transition Prior Network
Input	Sampled latent variable sequence s1:T	BS × T × s_dim
Dense	star -dim neurons, prior output	BS × T × star -dim
E.1.2 Extra Results.
During the training process, we record the estimation value of four mutual information (MI) terms. The corresponding results are presented in Figure 10. Despite not being explicitly incorporated into the objective function, the terms  $I_{\alpha_1} - I_{\alpha_2}$  and  $I_{\alpha_3} - I_{\alpha_4}$  exhibit significant maximization. Furthermore, the estimation values of  $I_{\alpha_2}$  and  $I_{\alpha_4}$  are found to be close to 0. These findings indicate that the state variable  $s_t^{\bar{r}}$  contains little information about the reward, and the predictability of  $s_t^{\bar{a}}$  by the action is also low.
E.2 Modified Cartpole
In the modified Cartpole environment, we configure the values as follows:  $\beta_{1} = \beta_{2} = \beta_{3} = \beta_{4} = 0.1$  and  $\lambda_{1} = \lambda_{2} = 0.1$ . Recurrent State Space Model (RSSM) uses a deterministic part and a stochastic part to represent latent variables. The deterministic state size for four dynamics are set to be (15, 15, 15, 15), and the stochastic state size are set to be (2, 2, 1, 4). The architecture of the encoder and decoder for observation is shown in Table 3 and Table 4 (64 × 64 resolution). Reward model uses 3-layer MLPs with hidden size to be 100 and four mutual information neural estimators are 4-layer MLPs with hidden size to be 128. Unlike the synthetic dataset, where there are clear categories
Figure 10: Estimation of the value of four mutual information terms and their differences in experiments on synthetic data.
for latent state variables, real-world situations pose a challenge due to the potential ambiguity in categorizing true variables. In our modified Cartpole environment, we defined the ground-truth  $s_t^{ar}$  as the cart's position,  $s_t^{\bar{a}r}$  as the pole's angle,  $s_t^{\bar{a}\bar{r}}$  as the light's greenness, and  $s_t^{\bar{a}\bar{r}}$  as the distractor Cartpole's state (including cart position and pole angle). The disentanglement scores for individual  $s_t^{ar}$  and  $s_t^{\bar{a}r}$ , as well as the combined  $s_t^r$ , are shown in Figure 11. We can obviously see that the true latent variables can be clearly identified.
Operator	Input Shape	Kernel Size	Stride	Padding
Input	[3,96,96]	—	—	—
Conv. +ReLU	[32,47,47]	4	2	0
Conv. +ReLU	[64,22,22]	4	2	0
Conv. +ReLU	[128,10,10]	4	2	0
Conv. +ReLU	[256,4,4]	4	2	0
Conv. +ReLU *	[256,2,2]	3	1	0
Reshape + FC	[1024]	—	—	—
Table 3: The encoder architecture designed for observation resolution of  $(96\times 96)$ . Its output is then fed into other networks for posterior inference. The default activation function used in the network is RELU. For observations with a resolution of  $(64\times 64)$ , the last convolutional layer(*) is removed.
(b) Cartpole
(c) Cartpole
Figure 11: The identifiability socres for IFactor in the experiments on cartpole.
(d) Cartpole
Operator	Input Shape	Kernel Size	Stride	Padding
Input	[input_size]	—	—	—
FC + ReLU + Reshape	[1024, 1, 1]	—	—	—
Conv. Transpose + ReLU *	[128, 3, 3]	3	1	0
Conv. Transpose + ReLU	[128, 9, 9]	5	2	0
Conv. Transpose + ReLU	[64, 21, 21]	5	2	0
Conv. Transpose + ReLU	[32, 46, 46]	6	2	0
Conv. Transpose + ReLU	[3, 96, 96]	6	2	0
Table 4: The decoder architecture designed for  $(96\times 96)$ -resolution observation. For  $(64\times 64)$ -resolution observation, the first transpose convolutional layer(*) is removed.
E.3 Variant of Robodesk
In the variant of Robodesk, we conduct experiments with the following hyperparameter settings:  $\beta_{1} = \beta_{2} = 2$ ,  $\beta_{3} = \beta_{4} = 0.25$ , and  $\lambda_{1} = \lambda_{2} = 0.1$ . For the four dynamics, we set the deterministic state sizes to (120, 40, 40, 40), and the stochastic state sizes to (30, 10, 10, 10). Denoised MDP utilizes two latent processes with deterministic state sizes [120, 120] and stochastic state sizes [20, 10]. For the mutual information neural estimators, we employ 4-layer MLPs with a hidden size of 128. To ensure a fair comparison, we align the remaining hyperparameters and network structure with those in the Denoised MDP. We reproduce the results of the Denoised MDP using their released code, maintaining consistency with their paper by employing the default hyperparameters. In order to evaluate the impact of the Mutual Information (MI) constraints, we conduct an ablation study. The results are shown in Figure 12. The constraints  $\mathcal{I}_{RS}^{t}$  and  $\mathcal{I}_{AS}^{t}$  are observed to stabilize the training process of IFactor. The results of IFactor are areaveraged over 5 runs, while the results of Denoised MDP and IFactor without MI are averaged over three runs.
Figure 12: Comparison between IFactor and Denoised MDP in the variant of Robodesk environment.
Policy learning based on the learned representations by IFactor We retrain policies using the Soft Actor-Critic algorithm [40] with various combinations of the four learned latent categories as input. We wrap the original environment with visual output using our representation model to obtain compact features. In this process, both deterministic states and stochastic states are utilized to form the feature. For instance, when referring to  $s_t^r$ , we use both the deterministic states and stochastic states of  $s_t^r$ . The implementation of SAC algorithm is based on Stableb-Baselines3[59], with a learning rate of 0.0002. Both the policy network and Q network consist of 4-layer MLPs with a hidden size of 256. We use the default hyperparameter settings in Stable-Baselines3 for other parameters.
E.4 Variants of Deep Mind Control Suite
In the noiseless DMC environments, we set  $\beta_{1} = \beta_{2} = \beta_{3} = \beta_{4} = 1$ . For the DMC environments with video background, we set  $\beta_{1} = \beta_{2} = 1$  and  $\beta_{3} = \beta_{4} = 0.25$ . In the DMC environments with video background and noisy sensor, we set  $\beta_{1} = \beta_{2} = 2$  and  $\beta_{3} = \beta_{4} = 0.25$ . Lastly, for the DMC environments with video background and jittering camera, we set  $\beta_{1} = \beta_{2} = 1$  and  $\beta_{3} = \beta_{4} = 0.25$ . Regarding the Reacher environment with video background and jittering camera, we set  $\lambda_{1} = \lambda_{2} = 0.1$  for our experiments. For the other environments, we set  $\lambda_{1} = \lambda_{2} = 0$ . The deterministic state sizes for the four dynamics are set to (120, 120, 60, 60), while the stochastic state sizes are set to (20, 20, 10, 10). The four mutual information neural estimators utilize a 4-layer MLPs with a hidden size of 128. We align the other hyperparameters and network structure with those used in the Denoised MDP for a fair comparison.
Environment Steps	Action Repeat	Train Every	Collection Intervals	Batch Size	Sequence Length	Horizon
Modified Cartpole	200,000	1	5	5	20	30	8
Robodesk	1,000,000	2	1000	100	50	50	15
DMC	1,000,000	2	1000	100	25	50	12
Table 5: Some hyperparameters of our method in the environment of Modified Cartpole, Robodesk and DMC. Environment Steps represents the number of interactions between the agent and the environment. Action Repeat determines how many times an agent repeats an action in a step. Train Every specifies the environment step between adjacent training iterations. Collection Intervals defines the number of times the model is trained in each training iteration (including world models, policy networks and value networks). Batch Size refers to the number of trajectories in each mini-batch. Sequence Length denotes the length of the chuck used in training the world models. Horizon determines the length of dreaming when training the policy using the world model. Hyperparameters are aligned with those used in the Denoised MDP for fair comparison.
E.4.1 Policy optimization results on variants of DMC
We present detailed evaluation results in Table 6, showcasing both the mean values and the corresponding standard deviations for the final policy performance across each task. Results are averaged across three seeds. Denoised MDP performs well across all four variants with distinct noise types.
E.4.2 Mutual information
Figure 7 demonstrates the notable improvement in policy performance in the Reacher environment with video background and jittering camera due to the inclusion of the constraints  $J_{RS}^{t}$  and  $J_{AS}^{t}$ . To further investigate how they affects the model learning, we record the estimation values of four Mutual Information terms throughout the training process, as depicted in Figure 13. The results indicate that both  $I_{\alpha_1} - I_{\alpha_2}$  and  $I_{\alpha_3} - I_{\alpha_4}$  are maximized for both IFactor and IFactor without MI. However, IFactor exhibits a significantly higher rate of maximizing  $I_{\alpha_3} - I_{\alpha_4}$  compared to IFactor without MI. This increased maximization leads to greater predictability of  $s_t^a$  by the action, ultimately contributing to the observed performance gain.
E.5 Visualization for DMC
In this experiment, we investigate five types of representations, which can be derived from the combination of four original disentangled representation categories. Specifically,  $s_t^a$  is the controllable and reward relevant representation.  $s_t^r = (s_t^{ar}, s_t^{\bar{a}r})$  is the reward-relevant representation.  $s_t^{\bar{a}\bar{r}}$  is the controllable but reward-irrelevant representation.  $s_t^{\bar{a}\bar{r}}$  is the uncontrollable and reward-irrelevant representation (noise).  $s_t^{\bar{r}} = (s_t^{\bar{a}\bar{r}}, s_t^{\bar{a}\bar{r}})$  is the reward-irrelevant representation. Only representations of  $s_t^r$  are used for policy optimization. We retrain 5 extra observation decoders to reconstruct the original image, which can precisely characterize what kind of information each type of representation contains, surpassing the limitations of the original decoder that is used in latent traversal. The visualization results are shown in Figure 14. It can be observed that  $s_t^{ar}$  captures the movement of the agent partially but not well enough;  $s_t^r$  captures the movement of the agent precisely but  $s_t^{\bar{r}}$  fails (Reacher and Cheetah) or captures extra information of the background (Walker). This finding
Cheetah	IFactor (Ours)	Dreamer Pro	Denoised MDP	Dreamer	TIA	DBC	CURL	PI-SAC
Noiseless	874±39	803±75	771±60	714±348	766±29	182±50	171±34	369±28
Video Background	573±193	366±96	431±111	171±45	388±297	248±71	70±48	112±31
Video Background + Noisy Sensor	456±88	134±41	400±190	166±42	227±13	141±13	199±7	151±14
Video Background + Camera Jittering	418±22	150±104	294±100	160±32	202±93	141±35	169±22	156±20
Walker	IFactor (Ours)	Dreamer Pro	Denoised MDP	Dreamer	TIA	DBC	CURL	PI-SAC
Noiseless	966±5	941±16	947±13	955±6	955±5	614±111	417±296	203±92
Video Background	917±52	909±48	790±113	247±135	685±337	199±67	608±100	200±18
Video Background + Noisy Sensor	701±174	242±65	661±120	279±145	425±281	95±54	338±92	222±21
Video Background + Camera Jittering	524±194	368±301	291±104	106±22	230±332	62±17	448±70	116±6
Reacher	IFactor (Ours)	Dreamer Pro	Denoised MDP	Dreamer	TIA	DBC	CURL	PI-SAC
Noiseless	924±37	924±61	686±216	876±57	587±256	95±58	663±221	166±235
Video Background	963±10	555±92	544±121	253±127	123±21	102±58	751±189	76±35
Video Background + Noisy Sensor	839±51	675±137	561±182	202±82	264±280	97±39	607±260	85±5
Video Background + Camera Jittering	736±53	675±82	213±106	109±19	89±26	87±51	632±96	84±13
Table 6: Policy optimization evaluation on variants of DMC with various distractors. IFactor consistently performs well across all four variants with distinct noise types. Bold numbers show the best model-learning result for specific policy learning choices. Results are averaged over 3 runs.
suggests that  $s_t^r$  contains sufficient information within the original noisy observation for effective control, while effectively excluding other sources of noise.
Figure 13: Estimation of the value of four mutual information terms and their differences in the Reacher Easy environment with video background and jittering camera.
Walker Walk
Video Background + Noisy Sensor
Figure 14: Visualization of the DMC variants and the factorization learned by IFactor.
F Comparison between IFactor and Related Work
F.1 Comparison between IFactor and Denoised MDP
While both IFactor and Denoised MDP share the common aspect of factorizing latent variables based on controllability and reward relevance, it is crucial to recognize the numerous fundamental distinctions between them.
(a) IFactor (ours).
(b) Denoised MDP.
Figure 15: Graphical illustration of our world model and Denoised MDP.
First and foremost, Denoised MDP adopts a rather stringent assumption by solely considering three types of latent variables and assuming independent latent processes for  $x_{t}$  (controllable and reward-relevant) and  $y_{t}$  (uncontrollable but reward-relevant). However, this strict assumption may not hold in many scenarios where uncontrollable yet reward-relevant factors exhibit dependencies on controllable and reward-relevant factors. Take, for instance, the case of car driving: the agent lacks control over surrounding vehicles, yet their behaviors are indirectly influenced by the agent's actions. In contrast, our approach encompasses four types of causally related latent variables while only assuming conditional independence when conditioning on the state in the previous time step. This assumption holds true naturally within the (PO)MDP framework.
Secondly, Denoised MDP is limited to only factoring out additive rewards  $r_{y_t}$  from  $r_{x_t}$ , disregarding the possibility of non-additive effects in many uncontrollable yet reward-relevant factors. In contrast, our method embraces the inclusion of non-additive effects of  $s_t^{\bar{a}r}$  on the reward, which is more general.
Thirdly, Denoised MDP uses only controllable and reward-relevant latent variables  $x_{t}$  for policy optimization, which we show in the theoretical analysis that it is generally insufficient. In contrast, our method utilizes both controllable and uncontrollable reward-relevant factors for policy training.
Finally, Denoised MDP makes the assumption of an instantaneous causal effect from  $x_{t}$  to  $z_{t}$  and from  $y_{t}$  to  $z_{t}$ , which is inherently unidentifiable without further intervention. It is worth noting that imposing interventions on the latent states is unrealistic in most control tasks, as agents can only choose actions at specific states and cannot directly intervene on the state itself. In contrast, our method assumes that there exists no instantaneous causal effect for latent variables. In conjunction with several weak assumptions, we provide a proof of block-wise identifiability for our four categories of latent variables. This property serves two important purposes: (1) it ensures the removal of reward-irrelevant factors and the utilization of minimal and sufficient reward-relevant variables for policy optimization, and (2) it provides a potential means for humans to comprehend the learned representations within the reinforcement learning (RL) framework. Through latent traversal of the four types of latent variables, humans can gain insights into the specific kind of information that each category of representation contains within the image.
From the perspective of model structure, it is worth highlighting that the architecture of both the transition model (prior) and the representation model (posterior) in IFactor differs from that of Denoised MDP. The structure of prior and posterior of IFactor is shown as follows:
$$
\left\{ \begin{array}{l l} p _ {\gamma_ {1}} \left(s _ {t} ^ {a r} \mid s _ {t - 1} ^ {r}, a _ {t - 1}\right) & \text {P o s t e r i o r :} \\ p _ {\gamma_ {2}} \left(s _ {t} ^ {\bar {a} r} \mid s _ {t - 1} ^ {r}\right) & q _ {\phi_ {1}} \left(s _ {t} ^ {a r} \mid o _ {t}, s _ {t - 1} ^ {r}, a _ {t - 1}\right) \\ p _ {\gamma_ {3}} \left(s _ {t} ^ {\bar {a} r} \mid \mathbf {s} _ {\mathbf {t} - 1}, a _ {t - 1}\right) & q _ {\phi_ {3}} \left(s _ {t} ^ {\bar {a} r} \mid o _ {t}, \mathbf {s} _ {\mathbf {t} - 1}, a _ {t - 1}\right) \\ p _ {\gamma_ {4}} \left(s _ {t} ^ {\bar {a} r} \mid \mathbf {s} _ {\mathbf {t} - 1}\right) & q _ {\phi_ {4}} \left(s _ {t} ^ {\bar {a} r} \mid o _ {t}, \mathbf {s} _ {\mathbf {t} - 1}\right) \end{array} \right. \tag {44}
$$
While Denoised MDP has the following prior and posterior:
Prior:
$$
\left\{ \begin{array}{l} p _ {\gamma_ {1}} \left(x _ {t} \mid x _ {t - 1}, a _ {t - 1}\right) \\ p _ {\gamma_ {2}} \left(y _ {t} \mid y _ {t - 1}\right) \\ p _ {\gamma_ {3}} \left(z _ {t} \mid x _ {t}, y _ {t}, z _ {t - 1}\right) \end{array} \quad \left\{ \begin{array}{l} p _ {\phi_ {1}} \left(x _ {t} \mid x _ {t - 1}, y _ {t - 1}, z _ {t - 1}, o _ {t}, a _ {t - 1}\right) \\ p _ {\phi_ {2}} \left(y _ {t} \mid x _ {t - 1}, y _ {t - 1}, z _ {t - 1}, o _ {t}, a _ {t - 1}\right) \\ p _ {\phi_ {3}} \left(z _ {t} \mid x _ {t}, y _ {t}, o _ {t}, a _ {t - 1}\right) \end{array} \right. \right. \tag {45}
$$
A notable distinction can be observed between Denoised MDP and IFactor in terms of the assumptions made for the prior and posterior structures. Denoised MDP assumes independent priors for  $x_{t}$  and  $y_{t}$ , whereas IFactor only incorporates conditional independence, utilizing  $s_{t-1}^{r}$  as input for the transition of both  $s_{t}^{ar}$  and  $s_{t}^{\bar{a}r}$ . Moreover, the posterior of  $y_{t}$  receives  $a_{t-1}$  as input, potentially implying controllability. Similarly, the posterior of  $x_{t}$  incorporates  $z_{t-1}$  as input, which may introduce noise from  $z_{t-1}$  into  $x_{t}$ . These implementation details can deviate from the original concept. In contrast, our implementation ensures consistency between the prior and posterior, facilitating a clean disentanglement in our factored model.
From the perspective of the objective function, IFactor incorporates two supplementary mutual information constraints, namely  $\mathcal{I}_{\mathrm{RS}}^t$  and  $\mathcal{I}_{\mathrm{AS}}^t$ , to promote disentanglement and improve policy performance.
F.2 Comparison between IFactor, InfoPower and IsoDream
IFactor learns different categories of state representations according to their relation with action and reward, which is different from InfoPower [48] and IsoDream [12], and moreover, IFactor emphasizes block-wise identifiability for the four categories of representations while InfoPower and Iso-Dream do not. Specifically, InfoPower learns 3 types of latent variables, including  $s_t^{ar}$ ,  $s_t^{\bar{a}r}$  and  $s_t^{\bar{a}\bar{r}}$ . IsoDream uses three branches for latent dynamics, distinguishing controllable, noncontrollable, and static parts.
Other differences with InfoPower:
- Reconstruction Basis: InfoPower is reconstruction-free, while IFactor is reconstruction-based.
- Objective Functions: InfoPower prioritizes mutual information and empowerment, while IFactor utilizes reconstruction, KL constraints, and mutual information constraints for disentanglement. InfoPower formulates policy using task reward value estimates and the empowerment objective, while IFactor learns policy by maximizing the estimated Q value and dynamics backpropagating.
Other differences with IsoDream: