Year: 2022

Abstract:
Reinforcement learning (RL) experiments have notoriously high variance, and minor details can have disproportionately large effects on measured outcomes. This is problematic for creating reproducible research and also serves as an obstacle when applying RL to sensitive real-world applications. In this paper, we investigate causes for this perceived instability. To allow for an in-depth analysis, we focus on a specifically popular setup with high variance – continuous control from pixels with an actor-critic agent. In this setting, we demonstrate that poor outlier runs which completely fail to learn are an important source of variance, but that weight initialization and initial exploration are not at fault. We show that one cause for these outliers is unstable network parametrization which leads to saturating nonlinearities. We investigate several fixes to this issue and find that simply normalizing penultimate features is surprisingly effective. For sparse tasks, we also find that partially disabling clipped double Q-learning decreases variance. By combining fixes we significantly decrease variances, lowering the average standard deviation across 21 tasks by a factor > 3 for a state-of-the-art agent. This demonstrates that the perceived variance is not necessarily inherent to RL. Instead, it may be addressed via simple modifications and we argue that developing low-variance agents is an important goal for the RL community.

heading: 1 INTRODUCTION
text: With the advent of deep learning, reinforcement learning (RL) with function approximators has become a promising strategy for learning complex behavior (Silver et al., 2017; Mnih et al., 2015). While RL can work remarkably well, deep RL experiments have been observed to exhibit high variance (Chan et al., 2020; Clary et al., 2018; Lynnerup et al., 2020). Specifically, the rewards obtained by RL agents commonly vary significantly depending on the random seeds used for training, even if no other modifications are made. Furthermore, implementation level details (Engstrom et al., 2019) and even hardware scheduling (Henderson et al., 2018) often have an outsized effect on performance.
High variance is problematic for creating reproducible RL research, and achieving statistically significant results over large RL benchmarks is often prohibitively time-consuming (Colas et al., 2018; Agarwal et al., 2021). Furthermore, high variance provides an obstacle for real-world applications of RL, where safety and predictability can be critical (Garcıa & Fernández, 2015; Srinivasan et al., 2020). Especially relevant RL applications include medical testing (Bastani et al., 2021), autonomous vehicles (Bellemare et al., 2020) and communications infrastructure (Liu et al., 2020).
In this paper, we investigate causes for high variance in RL. To limit the paper’s scope and computational footprint we focus on continuous control from pixels with a state-of-the-art (SOTA) actorcritic algorithm (Lillicrap et al., 2021; Yarats et al., 2021b). This is a popular setup that exhibits high variance, and where actor-critic algorithms often achieve SOTA results (Laskin et al., 2020b;a; Kostrikov et al., 2020). In this setting, we first show that an important cause of variance is outlier runs that fail early and never recover. It is natural to hypothesize that the typical sources of randomness, network initialization, and initial exploration, are to blame. However, we demonstrate that these do in fact have little impact on the final performance. We similarly provide experiments that suggest that neither poor feature representations nor excessive sensitivity of the environment is at fault. We instead demonstrate that unstable network parametrization can drive the variance and lead to excessively large activations that saturate action distributions. We propose several methods to
avoid such issues and find that they decrease variance. The most effective method is simply setting penultimate features to unit norm – a strategy we call penultimate normalization – which avoids exploding activations by design. We also find that partially disabling double clipped Q-learning (Fujimoto et al., 2018), a strategy we call asymmetrically double clipped Q-learning, reduces variance for many sparse tasks. By combining multiple approaches we increase average rewards while decreasing variance as measured of 21 tasks. The average standard deviation decreases by a factor > 3 and for some tasks by more than an order of magnitude. Our experiments suggest that high variance is not necessarily inherent to RL. Instead, one can directly optimize for lower variance without hurting the average reward. We argue that developing such low-variance agents is an important goal for the RL community – both for the sake of reproducible research and real-world deployment.

heading: 2 BACKGROUND
text: We consider continuous control modeled as a Markov decision process (MDP) with a continuous action space A and state space S. In practice, the action space A is often bounded to account for physical limitations, e.g., how much force robotic joints can exert. A popular choice is simply A = [−1, 1]dim(A). Within this setting, Deep Deterministic Policy Gradients (DDPG) (Lillicrap et al., 2021) is a performant algorithm that forms the backbone of many state-of-the-art agents (Laskin et al., 2020b;a; Kostrikov et al., 2020; Yarats et al., 2021b). DDPG uses a neural network (the critic) to predict the Q-valueQφ(at, st) for each state-action pair. The parameters φ of the critic are trained by minimizing the soft Bellman residual:
E ( Qφ(at, st)− [ rt + γE[Q̂(at+1, st+1)] ])2 (1)
Here, rt is the reward and E[Q̂(at+1, st+1)] is the Q-value estimated by the target critic – a network whose weights can e.g. be the exponentially averaged weights of the critic. One can use multi-step returns for the critic target (Sutton, 1988). Another common trick is double clipped q-learning, i.e. using two Q-networks Q1, Q2 and taking Q = min(Q1, Q2) to decrease Q-value overestimation (Fujimoto et al., 2018). The policy is obtained via an actor-network with parameters θ, which maps each state st to a distribution πθ(st) over actions. The action is obtained by calculating a pre θ (s), the vector output of the actor network. Thereafter a tanh non-linearity is applied elementwise to ensure a ∈ [−1, 1]n for n = dim(A). The policy is then obtained as:
aθ(s) = tanh(a pre θ (s)) + (2)
is some additive noise which encourages exploration, e.g. a Gaussian variable with time-dependent variance σ2t . Various noise types have been employed (Achiam, 2018; Lillicrap et al., 2021; Yarats et al., 2021b). Since the Q-values correspond to the expected discounted rewards, the actions aθ should maximize Qφ(aθ, s). Thus, to obtain a gradient update for the actor one uses the gradient ∇θQφ(aθ, s), effectively calculating gradients through the critic network. In control from pixels, the state s is typically an image (possibly data augmented) processed by a convolutional network.

heading: 3 BENCHMARKING VARIANCE
text: Experimental Setup. To limit the paper’s scope and computational footprint, we focus on continuous control from pixels with an actor-critic algorithm. This setup is relatively close to real-world robotics, where reliable performance is important. We consider the standard continuous control benchmark deepmind control (dm-control) (Tassa et al., 2020). Within dm-control, a long line of state-of-the-art models relies on the actor-critic setup (Laskin et al., 2020b;a; Kostrikov et al., 2020). We consider the recent data-augmentation based DDPG-type agent DRQv2 (Yarats et al., 2021b) which provides state-of-the-art performance and runs fast. We use the default hyperparameters that Yarats et al. (2021b) uses on the medium benchmark (listed in Appendix A) throughout the paper. We will refer to this agent as the baseline agent. We are interested in using a large number of random seeds to obtain accurate statistics and thus need to limit the number of environments we consider. To select environments we simply rank the dm-control tasks based upon their relative variance as measured by Yarats et al. (2021b). We select the five tasks with the highest relative variance – finger turn hard, walker walk, hopper hop, acrobot swingup, and reacher hard (see Table 4 in Appendix A for details). In Section 4.2 we consider a larger set of tasks. We will refer to these as environments or tasks interchangeably. For each run, we train the agent for one million frames, or equivalently 1,000
episodes, and evaluate over ten episodes. We use 10 or 40 seeds for each method we consider, these are not shared between experiments, but we reuse the runs for the baseline agent across figures.

heading: 3.1 VERIFYING VARIANCE
text: We first aim to verify that the environments indeed exhibit high variance. To this end, we run the baseline agent with 40 random seeds on each task. Figure 1 shows the result of this experiment for three tasks, see Appendix A for the two other tasks. We show both individual runs as well as mean and standard deviations. For some tasks, the variance is dominated by outlier runs that fail to learn early in the training and never recover. E.g. for the walker task, we see several runs which have not improved during the first 500 episodes, and most of these do not improve in the following 500 episodes either. Except for these outliers, the variance of the walker task is relatively small. We also note that these outliers separate themselves relatively early on, suggesting that there are some issues in early training that cause variance. If it was possible to eliminate poor outlier runs, we would expect the variance to go down. Motivated by these observations, and the walker task in particular, we will investigate various potential causes of outlier runs and high variance.

heading: 3.2 DOES INITIALIZATION MATTER?
text: There are multiple sources of randomness specific to the early training iterations. The agent networks are randomly initialized and the agent is given some initial experience, collected by random actions, to start training on. Could these two initial factors be the cause of the high variance? To study this hypothesis we measure the correlation between two runs that share the same network initialization and initial experience, but otherwise have independent randomness (from e.g. the environment, action sampling, and mini-batch sampling). Across all tasks, we repeat this process for ten combinations of initializations and initial experience, from each such combination we start two runsA,B which do not share other sources of randomness. For each time-step we measure the Pearson correlation between score(A) and score(B). This correlation, averaged across all timesteps, is shown in Figure 2 (left). We see that the correlation is very small, suggesting that the initialization and initial experience does not have a large effect on the outcome. Specifically, the hypothesis that there are good and bad initializations that leads to good or bad performance is inconsistent with the results of Figure 2 (left). Correlation across time is noisy but centers around zero, see Appendix A.

heading: 3.3 ARE THE ENVIRONMENTS TOO SENSITIVE?
text: Another cause of variance is the environments. Perhaps the environments require very specific policies to yield rewards, and unless the agent stumbles upon a good policy early on, learning fails. To study the sensitivity of the environments we consider gradually exchanging a good learned policy with a random policy. Specifically, at every timestep t, with probability p, we exchange the policy’s action with a random action. We use the final policy of a performant run with the baseline agent as the learned policy. The effect of this, for ten final policies learned for the walker environment, is shown in Figure 2 (right). Performance degrades gracefully as we gradually move to a random policy. This suggests that the environment is relatively stable with regard to actions. Results for other environments are similar and are shown in Appendix A. Furthermore, in Appendix E we verify that the variance arising from evaluating the agent on a finite number of episodes is small compared to the inter-run variance. Thus, environment sensitivity and evaluation methodology do not appear to explain the excessive variance.

heading: 3.4 IS FEATURE LEARNING TO BLAME?
text: In the early stages of training, the image encoder will extract poor features. Without good representations, it is hard to learn a good policy, and without a policy generating interesting behavior it is hard to learn good representations. If such a vicious cycle was causing runs to fail, jump-starting the encoder to extract good features should help to decrease the variance. To test this hypothesis, we consider initializing the weights of the convolutional encoder to some good pretrained values before training as normal. For each task, we pick a run with above-average final performance and use its convolutional weights at episode 500. We refer to this strategy as pretrain. We also consider the same strategy but decrease the learning rate of the pre-trained weights by a factor of 10x. We refer to this strategy as pretrain + small lr. In addition to this fix, we also consider improving the features with self-supervised learning during early training (Jing & Tian, 2020). We use the loss function proposed in Grill et al. (2020) which is useful for smaller batch sizes (we use 256) and has already successfully been applied in RL (Schwarzer et al., 2021). We use standard methods in self-supervised learning: a target network with exponentially moving weights and a head network. See Appendix A for details. We consider adding the self-supervised loss to the first 10,000 and 20,000 steps and refer to these modifications as self-supervised 10k and self-supervised 20k. The results for this experiment are shown in Table 1. For these fixes, the large variance persists even though the performance improves marginally with self-supervised learning. We thus hypothesize that representation learning is not the main bottleneck. Indeed, experiments in later sections of the paper show that one can significantly decrease variance without improving representation learning.

heading: 3.5 STABILITY OF NETWORK PARAMETRIZATION
text: We have considered the effects of initialization, feature learning, and environment sensitivity and found that none of these factors satisfactorily explain the observed variance. Another factor to consider is the stability of the network parametrization. Unstable parametrization could make the learning process chaotic and thus cause high variance. To study this hypothesis we investigate the failure modes of individual runs. In Figure 3 we consider some runs that fail as well as some successful runs for the walker task. Plots for all runs are available in Appendix A. We plot three quantities during training: rewards, actor gradient norm ‖∇`‖, and average absolute value of actions |a|. All agents initially have poor rewards, but once the rewards start to improve, they improve steadily. During this initial period of no or little learning, the gradients of the actor-network (row two in Figure 3) are small. Recall that the actor is optimized via gradients: ∇θQφ(s,aθ(s)) =
∑ i ∂Qφ ∂ai ∂ai ∂θ . If this gradient is close to zero, we might suspect that ∂ai ∂θ ≈ 0. This could easily happen if the tanh non-linearity in equation 2 is saturated. In row three of Figure 3 we show the average absolute value of the actions, which are all bound to lie in [−1, 1]. We indeed see that for the runs that fail the average absolute action stays around 1 throughout training, which implies that the gradients for the actor are small since | tanh(x)| ≈ 1 implies ∂∂x tanh(x) ≈ 0. The cause of saturating tanh must be excessively large activations, i.e. apreθ (s) in equation 2 becoming too large, which saturates the tanh non-linearity. In fact, all runs seem to initially suffer from saturating tanh, but most runs escape from this poor policy. Somewhat similarly, Wang et al. (2020) has observed that saturating nonlinearities can impede exploration when removing the entropy bonus from SAC agents. However, for the DDPG agent saturation does not harm exploration since the exploration noise is added outside the non-linearity as per eq. (2). Instead, the issue here appears to be vanishing gradients which impedes learning. The Q-values could of course also suffer from exploding activations, but since there are no saturating nonlinearities for the Q-values, such issues are harder to pinpoint. These experiments suggest that avoiding excessively large activations could reduce the number of poor outlier runs, and thus decrease variance.

heading: 4 IMPROVING STABILITY
text: Motivated by our qualitative observations we now consider a few ideas for improving the stability of learning, hoping that these will decrease the variance. We consider several methods:
Normalizing activations. As per eq. (2), the actor selects actions using apreθ (s), where a pre θ (s) is a deep neural network. Figure 3 shows that learning can fail due to excessively large activations. To avoid this we propose to parametrize apreθ (s) as a linear function of features with a fixed magnitude. Specifically, let Λθ(s) be the penultimate features of the actor-network. We propose to simply normalize Λnθ (s) = Λθ(s)/‖Λθ(s)‖ and let apre be a linear function of Λnθ , i.e. apreθ (s) = LΛnθ (s) for a linear function L. Note that no mean subtraction is performed. We refer to this straightforward modification as penultimate normalization or pnorm for short. When applied to the actor, we simply call the strategy actor pnorm. Our motivation has been to stabilize the actor output, but the critic network could also benefit from the stability offered by normalizing penultimate features. We refer to the strategy of normalizing both the actor and the critic as both pnorm. We also consider layer norm (Ba et al., 2016) applied similarly to both the actor and critic. We also consider the normalization scheme of Wang et al. (2020), which directly constrains the output of the actor. We refer to this method as output norm. At last, following Gogianu et al. (2021), we consider adding spectral normalization (Miyato et al., 2018) to the penultimate layer.
Penalizing saturating actions. Another strategy for avoiding actions with large magnitude is to add a term to the loss function which directly penalizes large actions. One can imagine many such loss functions, we here consider the simple and natural choice of squared loss `(θ, s) = λ‖apreθ (s)‖2. We can simply add the square loss times some hyperparameter λ to the loss function. By tuning λ, one should be able to remove the saturating actions illustrated in Figure 3. We heuristically find that λ = 0.000001 works well in practice (see Appendix A) and use this value. We refer to this modification as penalty. Since the Adam optimizer is invariant under re-scaling the loss, the specific value of λ has a small effect as long as ∇θQφ(s,aθ(s)) ≈ 0, which happen as per Figure 3. Learning rate warmup. One common strategy to improve stability is to start the learning rate at zero and increase it linearly to its top value during early training (Goyal et al., 2017). This can improve numerical stability (Ma & Yarats, 2021). We consider using this strategy and linearly increasing the learning rate from zero over the first 10,000 steps. We refer to this strategy as lr warmup.
Small weights final layer. Following Andrychowicz et al. (2020), we consider using smaller weights for the final layers of the actor and the critic. Specifically, we downscale these weights by a factor of 100 at initialization time. We refer to this strategy as scale down.
Gradient clipping. Another stabilizing strategy is gradient clipping (Zhang et al., 2020). To avoid occasional exploding gradients, one simply clips the norm of gradients that are above some threshold. This strategy is popular in NLP (Gehring et al., 2017; Peters et al., 2018) and is sometimes used in RL (Raffin et al., 2019). We consider clipping the gradient when the norm, calculated independently for the actor, critic, and convolutional encoder, is larger than 1 or 10. We refer to these strategies as grad clip 1 and grad clip 10.
Results. In Table 2 we evaluate the effects of these ideas. Penultimate normalization has the best performance and lowest variability on average, whereas layer norm can make performance worse. Since normalizing the critic improves performance, the critic might also suffer from instabilities. This instability can be observed by simply plotting the average Q-values during training for various agents, see Figure 4. We study this question in detail in Appendix C. The penalty method improves the performance while many other methods decrease average reward, and lowering variance by using a poor policy is not hard to achieve and not particularly desirable. The output normalization method of Wang et al. (2020) performs worse than actor normalization, and importantly fails completely on the acrobot task. We hypothesize that directly constraining the output space is harmful on the acrobot task. These experiments show that it is possible to significantly decrease the variance of the baseline agent by applying stabilization methods. Additionally, decreasing the variance often increases the average reward – likely as bad outlier runs are eliminated.

heading: 4.1 COMBINING FIXES
text: We have found that many of the proposed methods can decrease variance and improve average reward. For optimal performance, we now consider combining three orthogonal methods which improve performance individually – actor and critic penultimate normalization, pre-tanh penalty, and self-supervised learning (we apply it to the first 10,000 steps). In Table 3 we compare the performance between the baseline and an agent using these three modifications (which we refer to as combined) over 40 seeds. We see that the variances go down while the average score goes up. In fact, for the walker task, the variance decreases by orders of magnitude. Eliminating poor
outlier runs seems to benefit the average performance in addition to decreasing variance. While our methods have mostly been motivated by outlier runs that fail catastrophically, they likely help more normal runs too. Learning curves for the combined agent are given in Appendix A. In Appendix D, we verify that our fixes enable the agent to learn successfully with a larger learning rate.

heading: 4.2 ADDITIONAL TASKS
text: We now apply the combined agent of Table 3 to additional tasks. Specifically, we consider all 21 tasks identified as easy or medium by Yarats et al. (2021b). Note that the combined agent has been constructed using only 5 of these tasks, so this experiment measures generalization. For computational reasons, the additional 16 tasks only use 10 seeds each. We use the baseline hyperparameters across all tasks without tuning (Yarats et al. (2021b) tunes parameters on a per-task basis). Results are shown in Figure 5. Our methods consistently improve the average reward while the average standard deviation decreases by a factor > 2. Furthermore, a few additional tasks (finger spin, hopper stand, walker run, walker stand) have their variance decreased by an order of magnitude or more. The tasks cheetah run and quadruped walk perform worse than the baseline, although this could potentially be alleviated by tuning hyperparameters. Just increasing the learning rate on these tasks improve performance, see Figure 13 in Appendix A. These results demonstrate that our modifications significantly decrease variance and that directly optimizing variance can be feasible and fruitful. Performance profiles (Agarwal et al., 2021) are available in Figure 16 of Appendix A.

heading: 4.3 SPARSE TASKS
text: Consider the five tasks in Figure 5 where the combined agent has the highest standard deviation. Four out of these have sparse rewards – finger turn hard, reacher hard, pendulum swingup, and cartpole swingup sparse. Once stability has been addressed, sparsity is a major driver of residual variance. Motivated by this, we consider three fixes that could help for sparse tasks. Firstly, not training until at least one non-zero reward has been observed. Secondly, extending the self-supervised training throughout the whole training process. Thirdly, exchanging the clipped double Q-learning Q = min(Q1, Q2) (Fujimoto et al., 2018) with Q = avg(Q1, Q2), to decrease pessimistic bias. Through ablations (see Appendix B), we find that doing this for the critic loss (eq. (1)) but retaining the clipping Q = min(Q1, Q2) for the actor loss performs the best. We dub this strategy asymmetrically clipped double Q-learning. In Appendix B we present an ablation study over these three modifications, finding that using more self-supervised learning and asymmetrically clipped double Q-learning improves performance. We add these two modifications to our combined agent, which we now call combined++. In Figure 5 we show the performance of this agent (seeds are independent from experiments in Appendix B). Performance improves, especially for sparse tasks, and compared to the baseline agent the average standard deviation decreases by a factor > 3.

heading: 5 EPILOGUE
text: Related Work. There are multiple papers documenting the high variance of RL (Henderson et al., 2018), focusing on implementation details (Engstrom et al., 2019), continuous control (Mania et al., 2018), Atari games (Clary et al., 2018) and real-world robots (Lynnerup et al., 2020). Given this high variance, there is ample work on how to accurately measure performance in RL (Colas et al., 2019; Chan et al., 2020; Jordan et al., 2020; Agarwal et al., 2021; Islam et al., 2017). Our strategy of simply reducing the variance of RL algorithms is complementary to developing robust metrics. Within Q-learning, there is a large amount of work aiming to improve and stabilize Q-estimates (Fujimoto et al., 2018; Nikishin et al., 2018; Anschel et al., 2017; Jia et al., 2020; Chen et al., 2018). These apply to the discrete DQN setting, distinct from continuous control. There is also work on stabilizing RL for specific architectures (Parisotto et al., 2020) and task types (Kumar et al., 2019; Mao et al., 2019). Normalization in RL have been investigated by Salimans & Kingma (2016); Bjorck et al. (2021b); Gogianu et al. (2021), and Wang et al. (2020). Wang et al. (2020) directly normalizes the output of the actor, which e.g. for a one-dimensional task shrinks the action space to [tanh(−1), tanh(1)] (modulo noise). As per Table 2, this method underperforms our methods. Limitations. We have only considered a single SOTA agent (DRQv2 of (Yarats et al., 2021b)) on one popular suite of tasks. Furthermore, we focus on an agent with tanh nonlinearities for the policy. While this is an important agent class, other important agent types (Hafner et al., 2021) and environments (Bellemare et al., 2013) remain. We show that it is possible to decrease variance with simple modifications, although which methods are effective likely depends upon the base algorithm. While many of our fixes can potentially be added to other agents, it remains to be experimentally tested if doing so improves performance. We leave this question for future work. While our fixes typically are only a few lines of code, they nonetheless increase the agent complexity.
Conclusion. We have investigated variance in a popular continuous control setup and proposed several methods to decrease it. We demonstrate across 21 tasks that these dramatically decrease variance while improving average rewards for a state-of-the-art agent. Our experiments show that directly optimizing for lower variance can be both feasible and fruitful.
Ethics Statement. There are many positive RL applications – examples include safe autonomous cars, accelerated scientific discovery, and medical testing. Studying causes of high variance, and proposing fixes to this issue, can benefit real-world adoption of RL techniques. Additionally, reducing the variance of RL experiments would aid the research community as reproducibility can be problematic in the field. However, there are also many potentially harmful military applications. Our study is academic in nature and only considers simulated toy environments, but insights from our work could nonetheless be useful in harmful applications. We do not believe that our work differs from other RL work significantly concerning ethics.
Reproducability Statement. In Appendix A we detail the infrastructure and software used. The perhaps most important component of generating reproducible findings in RL is to use many seeds. To this end, we have focused on five environments but evaluated these thoroughly. The main experiments of the paper (Table 3) use 40 seeds whereas all others use 10 seeds.
Acknowledgement. This research is supported in part by the grants from the National Science Foundation (III-1618134, III-1526012, IIS1149882, IIS-1724282, and TRIPODS- 1740822), the Office of Naval Research DOD (N00014- 17-1-2175), Bill and Melinda Gates Foundation. We are thankful for generous support by Zillow and SAP America Inc. This material is based upon work supported by the National Science Foundation under Grant Number CCF-1522054. We are also grateful for support through grant AFOSR-MURI (FA9550-18-1-0136). Any opinions, findings, conclusions, or recommendations expressed here are those of the authors and do not necessarily reflect the views of the sponsors. We thank Rich Bernstein, Wenting Zhao, and Ziwei Liu for help with the manuscript. At last, we thank our anonymous reviewers for fruitful interactions.

heading: A APPENDIX
text: Tuning λ. For tuning the hyperparameters λ we search over the set {1e−1, 1e−2, 1e−3, 1e−4, 1e−5, 1e−6}. For each value, we run the hopper task for a short amount of time with five different seeds. We found that too large λ penalized actions too much, and that even very small values like 1e−6 sufficed to give improvement. Details on self-supervised Learning. We extract features from the convolutional network and the first feedforward layer of the critic, which projects the feature map to 50-dimensions. Feature vectors are normalized to unit norm before being fed into the self-supervised loss (Bachman et al., 2019). We use exponentially moving weights for the target network (He et al., 2020), using the same decay parameters as for the target critic. We use a head for the online network which is simply a two-layer MLP with hidden dimension 512 and final dimension 50 (Chen et al., 2020). The two augmentations we consider are simply two adjacent frames with augmentations that the DDPG agent of Yarats et al. (2021b) considers naturally. We use the square loss as proposed in Grill et al. (2020), which has been shown to be useful in smaller batch sizes. The self-supervised loss is then simply added to the normal loss function for the critic.
Infrastructure. We run our experiments on Nvidia Tesla V100 GPUs and Intel Xeon CPUs. The GPUs use CUDA 11.1 and CUDNN 8.0.0.5. We use PyTorch 1.9.0 and python 3.8.10. Our experiments are based upon the open-source DRQv2 implementation of Yarats et al. (2021b).
kwk <latexit sha1_base64="x4JnZHO29Z5jQM8bwLo2kGuAwYc=">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDbbSbt0swm7G6Wk/RFePCji1d/jzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU1HGqGDZYLGLVDqhGwSU2DDcC24lCGgUCW8Hodua3HlFpHssHM07Qj+hA8pAzaqzU6k7IE+lOeuWKW3XnIKvEy0kFctR75a9uP2ZphNIwQbXueG5i/Iwqw5nAaambakwoG9EBdiyVNELtZ/Nzp+TMKn0SxsqWNGSu/p7IaKT1OApsZ0TNUC97M/E/r5Oa8NrPuExSg5ItFoWpICYms99JnytkRowtoUxxeythQ6ooMzahkg3BW355lTQvqp5b9e4vK7WbPI4inMApnIMHV1CDO6hDAxiM4Ble4c1JnBfn3flYtBacfOYY/sD5/AG5PI8n</latexit><latexit sha1_base64="x4JnZHO29Z5jQM8bwLo2kGuAwYc=">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDbbSbt0swm7G6Wk/RFePCji1d/jzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU1HGqGDZYLGLVDqhGwSU2DDcC24lCGgUCW8Hodua3HlFpHssHM07Qj+hA8pAzaqzU6k7IE+lOeuWKW3XnIKvEy0kFctR75a9uP2ZphNIwQbXueG5i/Iwqw5nAaambakwoG9EBdiyVNELtZ/Nzp+TMKn0SxsqWNGSu/p7IaKT1OApsZ0TNUC97M/E/r5Oa8NrPuExSg5ItFoWpICYms99JnytkRowtoUxxeythQ6ooMzahkg3BW355lTQvqp5b9e4vK7WbPI4inMApnIMHV1CDO6hDAxiM4Ble4c1JnBfn3flYtBacfOYY/sD5/AG5PI8n</latexit><latexit sha1_base64="x4JnZHO29Z5jQM8bwLo2kGuAwYc=">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDbbSbt0swm7G6Wk/RFePCji1d/jzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU1HGqGDZYLGLVDqhGwSU2DDcC24lCGgUCW8Hodua3HlFpHssHM07Qj+hA8pAzaqzU6k7IE+lOeuWKW3XnIKvEy0kFctR75a9uP2ZphNIwQbXueG5i/Iwqw5nAaambakwoG9EBdiyVNELtZ/Nzp+TMKn0SxsqWNGSu/p7IaKT1OApsZ0TNUC97M/E/r5Oa8NrPuExSg5ItFoWpICYms99JnytkRowtoUxxeythQ6ooMzahkg3BW355lTQvqp5b9e4vK7WbPI4inMApnIMHV1CDO6hDAxiM4Ble4c1JnBfn3flYtBacfOYY/sD5/AG5PI8n</latexit><latexit sha1_base64="x4JnZHO29Z5jQM8bwLo2kGuAwYc=">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDbbSbt0swm7G6Wk/RFePCji1d/jzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU1HGqGDZYLGLVDqhGwSU2DDcC24lCGgUCW8Hodua3HlFpHssHM07Qj+hA8pAzaqzU6k7IE+lOeuWKW3XnIKvEy0kFctR75a9uP2ZphNIwQbXueG5i/Iwqw5nAaambakwoG9EBdiyVNELtZ/Nzp+TMKn0SxsqWNGSu/p7IaKT1OApsZ0TNUC97M/E/r5Oa8NrPuExSg5ItFoWpICYms99JnytkRowtoUxxeythQ6ooMzahkg3BW355lTQvqp5b9e4vK7WbPI4inMApnIMHV1CDO6hDAxiM4Ble4c1JnBfn3flYtBacfOYY/sD5/AG5PI8n</latexit>
re w
ar d
<latexit sha1_base64="Z8+uiuObr0q1nshzGW/IrRYnZXg=">AAAB9HicbVBNS8NAEN34WetX1aOXxSJ4KokIeix68VjBfkAbymYzbZduNnF3Ui2hv8OLB0W8+mO8+W/ctjlo64OBx3szzMwLEikMuu63s7K6tr6xWdgqbu/s7u2XDg4bJk41hzqPZaxbATMghYI6CpTQSjSwKJDQDIY3U785Am1ErO5xnIAfsb4SPcEZWsnvIDxhpuGR6XDSLZXdijsDXSZeTsokR61b+uqEMU8jUMglM6btuQn6GdMouIRJsZMaSBgfsj60LVUsAuNns6Mn9NQqIe3F2pZCOlN/T2QsMmYcBbYzYjgwi95U/M9rp9i78jOhkhRB8fmiXiopxnSaAA2FBo5ybAnjWthbKR8wzTjanIo2BG/x5WXSOK94bsW7uyhXr/M4CuSYnJAz4pFLUiW3pEbqhJMH8kxeyZszcl6cd+dj3rri5DNH5A+czx+Q+pKa</latexit><latexit sha1_base64="Z8+uiuObr0q1nshzGW/IrRYnZXg=">AAAB9HicbVBNS8NAEN34WetX1aOXxSJ4KokIeix68VjBfkAbymYzbZduNnF3Ui2hv8OLB0W8+mO8+W/ctjlo64OBx3szzMwLEikMuu63s7K6tr6xWdgqbu/s7u2XDg4bJk41hzqPZaxbATMghYI6CpTQSjSwKJDQDIY3U785Am1ErO5xnIAfsb4SPcEZWsnvIDxhpuGR6XDSLZXdijsDXSZeTsokR61b+uqEMU8jUMglM6btuQn6GdMouIRJsZMaSBgfsj60LVUsAuNns6Mn9NQqIe3F2pZCOlN/T2QsMmYcBbYzYjgwi95U/M9rp9i78jOhkhRB8fmiXiopxnSaAA2FBo5ybAnjWthbKR8wzTjanIo2BG/x5WXSOK94bsW7uyhXr/M4CuSYnJAz4pFLUiW3pEbqhJMH8kxeyZszcl6cd+dj3rri5DNH5A+czx+Q+pKa</latexit><latexit sha1_base64="Z8+uiuObr0q1nshzGW/IrRYnZXg=">AAAB9HicbVBNS8NAEN34WetX1aOXxSJ4KokIeix68VjBfkAbymYzbZduNnF3Ui2hv8OLB0W8+mO8+W/ctjlo64OBx3szzMwLEikMuu63s7K6tr6xWdgqbu/s7u2XDg4bJk41hzqPZaxbATMghYI6CpTQSjSwKJDQDIY3U785Am1ErO5xnIAfsb4SPcEZWsnvIDxhpuGR6XDSLZXdijsDXSZeTsokR61b+uqEMU8jUMglM6btuQn6GdMouIRJsZMaSBgfsj60LVUsAuNns6Mn9NQqIe3F2pZCOlN/T2QsMmYcBbYzYjgwi95U/M9rp9i78jOhkhRB8fmiXiopxnSaAA2FBo5ybAnjWthbKR8wzTjanIo2BG/x5WXSOK94bsW7uyhXr/M4CuSYnJAz4pFLUiW3pEbqhJMH8kxeyZszcl6cd+dj3rri5DNH5A+czx+Q+pKa</latexit><latexit sha1_base64="Z8+uiuObr0q1nshzGW/IrRYnZXg=">AAAB9HicbVBNS8NAEN34WetX1aOXxSJ4KokIeix68VjBfkAbymYzbZduNnF3Ui2hv8OLB0W8+mO8+W/ctjlo64OBx3szzMwLEikMuu63s7K6tr6xWdgqbu/s7u2XDg4bJk41hzqPZaxbATMghYI6CpTQSjSwKJDQDIY3U785Am1ErO5xnIAfsb4SPcEZWsnvIDxhpuGR6XDSLZXdijsDXSZeTsokR61b+uqEMU8jUMglM6btuQn6GdMouIRJsZMaSBgfsj60LVUsAuNns6Mn9NQqIe3F2pZCOlN/T2QsMmYcBbYzYjgwi95U/M9rp9i78jOhkhRB8fmiXiopxnSaAA2FBo5ybAnjWthbKR8wzTjanIo2BG/x5WXSOK94bsW7uyhXr/M4CuSYnJAz4pFLUiW3pEbqhJMH8kxeyZszcl6cd+dj3rri5DNH5A+czx+Q+pKa</latexit>
kr`k <latexit sha1_base64="2/B+anDEmlxWmP1lDld58msxalw=">AAAB+nicbVBNS8NAEN3Ur1q/Uj16WSyCp5KIoMeiF48VbCs0oUy203bpZhN2N0pp+1O8eFDEq7/Em//GbZuDtj4YeLw3w8y8KBVcG8/7dgpr6xubW8Xt0s7u3v6BWz5s6iRTDBssEYl6iECj4BIbhhuBD6lCiCOBrWh4M/Nbj6g0T+S9GaUYxtCXvMcZGCt13HIwoYGESAANUAgaTDpuxat6c9BV4uekQnLUO+5X0E1YFqM0TIDWbd9LTTgGZTgTOC0FmcYU2BD62LZUQow6HM9Pn9JTq3RpL1G2pKFz9ffEGGKtR3FkO2MwA73szcT/vHZmelfhmMs0MyjZYlEvE9QkdJYD7XKFzIiRJcAUt7dSNgAFzNi0SjYEf/nlVdI8r/pe1b+7qNSu8ziK5JickDPik0tSI7ekThqEkSfyTF7JmzNxXpx352PRWnDymSPyB87nD0S4k1g=</latexit><latexit sha1_base64="2/B+anDEmlxWmP1lDld58msxalw=">AAAB+nicbVBNS8NAEN3Ur1q/Uj16WSyCp5KIoMeiF48VbCs0oUy203bpZhN2N0pp+1O8eFDEq7/Em//GbZuDtj4YeLw3w8y8KBVcG8/7dgpr6xubW8Xt0s7u3v6BWz5s6iRTDBssEYl6iECj4BIbhhuBD6lCiCOBrWh4M/Nbj6g0T+S9GaUYxtCXvMcZGCt13HIwoYGESAANUAgaTDpuxat6c9BV4uekQnLUO+5X0E1YFqM0TIDWbd9LTTgGZTgTOC0FmcYU2BD62LZUQow6HM9Pn9JTq3RpL1G2pKFz9ffEGGKtR3FkO2MwA73szcT/vHZmelfhmMs0MyjZYlEvE9QkdJYD7XKFzIiRJcAUt7dSNgAFzNi0SjYEf/nlVdI8r/pe1b+7qNSu8ziK5JickDPik0tSI7ekThqEkSfyTF7JmzNxXpx352PRWnDymSPyB87nD0S4k1g=</latexit><latexit sha1_base64="2/B+anDEmlxWmP1lDld58msxalw=">AAAB+nicbVBNS8NAEN3Ur1q/Uj16WSyCp5KIoMeiF48VbCs0oUy203bpZhN2N0pp+1O8eFDEq7/Em//GbZuDtj4YeLw3w8y8KBVcG8/7dgpr6xubW8Xt0s7u3v6BWz5s6iRTDBssEYl6iECj4BIbhhuBD6lCiCOBrWh4M/Nbj6g0T+S9GaUYxtCXvMcZGCt13HIwoYGESAANUAgaTDpuxat6c9BV4uekQnLUO+5X0E1YFqM0TIDWbd9LTTgGZTgTOC0FmcYU2BD62LZUQow6HM9Pn9JTq3RpL1G2pKFz9ffEGGKtR3FkO2MwA73szcT/vHZmelfhmMs0MyjZYlEvE9QkdJYD7XKFzIiRJcAUt7dSNgAFzNi0SjYEf/nlVdI8r/pe1b+7qNSu8ziK5JickDPik0tSI7ekThqEkSfyTF7JmzNxXpx352PRWnDymSPyB87nD0S4k1g=</latexit><latexit sha1_base64="2/B+anDEmlxWmP1lDld58msxalw=">AAAB+nicbVBNS8NAEN3Ur1q/Uj16WSyCp5KIoMeiF48VbCs0oUy203bpZhN2N0pp+1O8eFDEq7/Em//GbZuDtj4YeLw3w8y8KBVcG8/7dgpr6xubW8Xt0s7u3v6BWz5s6iRTDBssEYl6iECj4BIbhhuBD6lCiCOBrWh4M/Nbj6g0T+S9GaUYxtCXvMcZGCt13HIwoYGESAANUAgaTDpuxat6c9BV4uekQnLUO+5X0E1YFqM0TIDWbd9LTTgGZTgTOC0FmcYU2BD62LZUQow6HM9Pn9JTq3RpL1G2pKFz9ffEGGKtR3FkO2MwA73szcT/vHZmelfhmMs0MyjZYlEvE9QkdJYD7XKFzIiRJcAUt7dSNgAFzNi0SjYEf/nlVdI8r/pe1b+7qNSu8ziK5JickDPik0tSI7ekThqEkSfyTF7JmzNxXpx352PRWnDymSPyB87nD0S4k1g=</latexit>
|a| <latexit sha1_base64="dvyirmkA3j4FKzXX7P9m/lLtYgc=">AAAB7HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68VjBtIU2lMl20y7dbMLuRihpf4MXD4p49Qd589+4bXPQ1gcDj/dmmJkXpoJr47rfztr6xubWdmmnvLu3f3BYOTpu6iRTlPk0EYlqh6iZ4JL5hhvB2qliGIeCtcLR3cxvPTGleSIfzThlQYwDySNO0VjJnxAkk16l6tbcOcgq8QpShQKNXuWr209oFjNpqECtO56bmiBHZTgVbFruZpqlSEc4YB1LJcZMB/n82Ck5t0qfRImyJQ2Zq78ncoy1Hseh7YzRDPWyNxP/8zqZiW6CnMs0M0zSxaIoE8QkZPY56XPFqBFjS5Aqbm8ldIgKqbH5lG0I3vLLq6R5WfPcmvdwVa3fFnGU4BTO4AI8uIY63EMDfKDA4Rle4c2Rzovz7nwsWtecYuYE/sD5/AEzOI5F</latexit><latexit sha1_base64="dvyirmkA3j4FKzXX7P9m/lLtYgc=">AAAB7HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68VjBtIU2lMl20y7dbMLuRihpf4MXD4p49Qd589+4bXPQ1gcDj/dmmJkXpoJr47rfztr6xubWdmmnvLu3f3BYOTpu6iRTlPk0EYlqh6iZ4JL5hhvB2qliGIeCtcLR3cxvPTGleSIfzThlQYwDySNO0VjJnxAkk16l6tbcOcgq8QpShQKNXuWr209oFjNpqECtO56bmiBHZTgVbFruZpqlSEc4YB1LJcZMB/n82Ck5t0qfRImyJQ2Zq78ncoy1Hseh7YzRDPWyNxP/8zqZiW6CnMs0M0zSxaIoE8QkZPY56XPFqBFjS5Aqbm8ldIgKqbH5lG0I3vLLq6R5WfPcmvdwVa3fFnGU4BTO4AI8uIY63EMDfKDA4Rle4c2Rzovz7nwsWtecYuYE/sD5/AEzOI5F</latexit><latexit sha1_base64="dvyirmkA3j4FKzXX7P9m/lLtYgc=">AAAB7HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68VjBtIU2lMl20y7dbMLuRihpf4MXD4p49Qd589+4bXPQ1gcDj/dmmJkXpoJr47rfztr6xubWdmmnvLu3f3BYOTpu6iRTlPk0EYlqh6iZ4JL5hhvB2qliGIeCtcLR3cxvPTGleSIfzThlQYwDySNO0VjJnxAkk16l6tbcOcgq8QpShQKNXuWr209oFjNpqECtO56bmiBHZTgVbFruZpqlSEc4YB1LJcZMB/n82Ck5t0qfRImyJQ2Zq78ncoy1Hseh7YzRDPWyNxP/8zqZiW6CnMs0M0zSxaIoE8QkZPY56XPFqBFjS5Aqbm8ldIgKqbH5lG0I3vLLq6R5WfPcmvdwVa3fFnGU4BTO4AI8uIY63EMDfKDA4Rle4c2Rzovz7nwsWtecYuYE/sD5/AEzOI5F</latexit><latexit sha1_base64="dvyirmkA3j4FKzXX7P9m/lLtYgc=">AAAB7HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68VjBtIU2lMl20y7dbMLuRihpf4MXD4p49Qd589+4bXPQ1gcDj/dmmJkXpoJr47rfztr6xubWdmmnvLu3f3BYOTpu6iRTlPk0EYlqh6iZ4JL5hhvB2qliGIeCtcLR3cxvPTGleSIfzThlQYwDySNO0VjJnxAkk16l6tbcOcgq8QpShQKNXuWr209oFjNpqECtO56bmiBHZTgVbFruZpqlSEc4YB1LJcZMB/n82Ck5t0qfRImyJQ2Zq78ncoy1Hseh7YzRDPWyNxP/8zqZiW6CnMs0M0zSxaIoE8QkZPY56XPFqBFjS5Aqbm8ldIgKqbH5lG0I3vLLq6R5WfPcmvdwVa3fFnGU4BTO4AI8uIY63EMDfKDA4Rle4c2Rzovz7nwsWtecYuYE/sD5/AEzOI5F</latexit>
run 1 <latexit sha1_base64="30We44UabOXpSdbemgkcJ55F7H0=">AAAB83icbVBNS8NAEN3Ur1q/qh69LBbBU0lE0GPRi8cK9gOaUDbbSbt0swm7E7GE/g0vHhTx6p/x5r9x2+agrQ8GHu/NMDMvTKUw6LrfTmltfWNzq7xd2dnd2z+oHh61TZJpDi2eyER3Q2ZACgUtFCihm2pgcSihE45vZ37nEbQRiXrASQpBzIZKRIIztJLvIzxhrjNFvWm/WnPr7hx0lXgFqZECzX71yx8kPItBIZfMmJ7nphjkTKPgEqYVPzOQMj5mQ+hZqlgMJsjnN0/pmVUGNEq0LYV0rv6eyFlszCQObWfMcGSWvZn4n9fLMLoOcqHSDEHxxaIokxQTOguADoQGjnJiCeNa2FspHzHNONqYKjYEb/nlVdK+qHtu3bu/rDVuijjK5IScknPikSvSIHekSVqEk5Q8k1fy5mTOi/PufCxaS04xc0z+wPn8ARQ2kbE=</latexit><latexit sha1_base64="30We44UabOXpSdbemgkcJ55F7H0=">AAAB83icbVBNS8NAEN3Ur1q/qh69LBbBU0lE0GPRi8cK9gOaUDbbSbt0swm7E7GE/g0vHhTx6p/x5r9x2+agrQ8GHu/NMDMvTKUw6LrfTmltfWNzq7xd2dnd2z+oHh61TZJpDi2eyER3Q2ZACgUtFCihm2pgcSihE45vZ37nEbQRiXrASQpBzIZKRIIztJLvIzxhrjNFvWm/WnPr7hx0lXgFqZECzX71yx8kPItBIZfMmJ7nphjkTKPgEqYVPzOQMj5mQ+hZqlgMJsjnN0/pmVUGNEq0LYV0rv6eyFlszCQObWfMcGSWvZn4n9fLMLoOcqHSDEHxxaIokxQTOguADoQGjnJiCeNa2FspHzHNONqYKjYEb/nlVdK+qHtu3bu/rDVuijjK5IScknPikSvSIHekSVqEk5Q8k1fy5mTOi/PufCxaS04xc0z+wPn8ARQ2kbE=</latexit><latexit sha1_base64="30We44UabOXpSdbemgkcJ55F7H0=">AAAB83icbVBNS8NAEN3Ur1q/qh69LBbBU0lE0GPRi8cK9gOaUDbbSbt0swm7E7GE/g0vHhTx6p/x5r9x2+agrQ8GHu/NMDMvTKUw6LrfTmltfWNzq7xd2dnd2z+oHh61TZJpDi2eyER3Q2ZACgUtFCihm2pgcSihE45vZ37nEbQRiXrASQpBzIZKRIIztJLvIzxhrjNFvWm/WnPr7hx0lXgFqZECzX71yx8kPItBIZfMmJ7nphjkTKPgEqYVPzOQMj5mQ+hZqlgMJsjnN0/pmVUGNEq0LYV0rv6eyFlszCQObWfMcGSWvZn4n9fLMLoOcqHSDEHxxaIokxQTOguADoQGjnJiCeNa2FspHzHNONqYKjYEb/nlVdK+qHtu3bu/rDVuijjK5IScknPikSvSIHekSVqEk5Q8k1fy5mTOi/PufCxaS04xc0z+wPn8ARQ2kbE=</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="p7IDirXBjsqnTAkOTrHHbBPUQw4=">AAAB6HicbZBLSwMxFIXv1FetVatbN8EiuCozbnQpuHFZwT6gU0omvdOGZjJDckcsQ/+GGxeK+Ivc+W9MHwttPRD4OCfh3pwoU9KS7397pa3tnd298n7loHp4dFw7qbZtmhuBLZGq1HQjblFJjS2SpLCbGeRJpLATTe7meecJjZWpfqRphv2Ej7SMpeDkrDAkfKbC5JoFs0Gt7jf8hdgmBCuow0rNQe0rHKYiT1CTUNzaXuBn1C+4ISkUziphbjHjYsJH2HOoeYK2Xyx2nrEL5wxZnBp3NLGF+/tFwRNrp0nkbiacxnY9m5v/Zb2c4pt+IXWWE2qxHBTnilHK5gWwoTQoSE0dcGGk25WJMTdckKup4koI1r+8Ce2rRuA3ggcfynAG53AJAVzDLdxDE1ogIIMXeIN3L/devY9lXSVv1dsp/JH3+QPNtJBT</latexit><latexit sha1_base64="p7IDirXBjsqnTAkOTrHHbBPUQw4=">AAAB6HicbZBLSwMxFIXv1FetVatbN8EiuCozbnQpuHFZwT6gU0omvdOGZjJDckcsQ/+GGxeK+Ivc+W9MHwttPRD4OCfh3pwoU9KS7397pa3tnd298n7loHp4dFw7qbZtmhuBLZGq1HQjblFJjS2SpLCbGeRJpLATTe7meecJjZWpfqRphv2Ej7SMpeDkrDAkfKbC5JoFs0Gt7jf8hdgmBCuow0rNQe0rHKYiT1CTUNzaXuBn1C+4ISkUziphbjHjYsJH2HOoeYK2Xyx2nrEL5wxZnBp3NLGF+/tFwRNrp0nkbiacxnY9m5v/Zb2c4pt+IXWWE2qxHBTnilHK5gWwoTQoSE0dcGGk25WJMTdckKup4koI1r+8Ce2rRuA3ggcfynAG53AJAVzDLdxDE1ogIIMXeIN3L/devY9lXSVv1dsp/JH3+QPNtJBT</latexit><latexit sha1_base64="cRcmr7R7mFV4HulQsRaxPiMqd9A=">AAAB83icbVA9TwJBEJ3DL8Qv1NJmIzGxInc2WhJtLDERMIEL2Vv2YMPe3mV31kgu/A0bC42x9c/Y+W9c4AoFXzLJy3szmZkXZVIY9P1vr7S2vrG5Vd6u7Ozu7R9UD4/aJrWa8RZLZaofImq4FIq3UKDkD5nmNIkk70Tjm5nfeeTaiFTd4yTjYUKHSsSCUXRSr4f8CXNtFQmm/WrNr/tzkFUSFKQGBZr96ldvkDKbcIVMUmO6gZ9hmFONgkk+rfSs4RllYzrkXUcVTbgJ8/nNU3LmlAGJU+1KIZmrvydymhgzSSLXmVAcmWVvJv7ndS3GV2EuVGaRK7ZYFFtJMCWzAMhAaM5QThyhTAt3K2EjqilDF1PFhRAsv7xK2hf1wK8Hd36tcV3EUYYTOIVzCOASGnALTWgBgwye4RXePOu9eO/ex6K15BUzx/AH3ucPEvaRrQ==</latexit><latexit sha1_base64="30We44UabOXpSdbemgkcJ55F7H0=">AAAB83icbVBNS8NAEN3Ur1q/qh69LBbBU0lE0GPRi8cK9gOaUDbbSbt0swm7E7GE/g0vHhTx6p/x5r9x2+agrQ8GHu/NMDMvTKUw6LrfTmltfWNzq7xd2dnd2z+oHh61TZJpDi2eyER3Q2ZACgUtFCihm2pgcSihE45vZ37nEbQRiXrASQpBzIZKRIIztJLvIzxhrjNFvWm/WnPr7hx0lXgFqZECzX71yx8kPItBIZfMmJ7nphjkTKPgEqYVPzOQMj5mQ+hZqlgMJsjnN0/pmVUGNEq0LYV0rv6eyFlszCQObWfMcGSWvZn4n9fLMLoOcqHSDEHxxaIokxQTOguADoQGjnJiCeNa2FspHzHNONqYKjYEb/nlVdK+qHtu3bu/rDVuijjK5IScknPikSvSIHekSVqEk5Q8k1fy5mTOi/PufCxaS04xc0z+wPn8ARQ2kbE=</latexit><latexit sha1_base64="30We44UabOXpSdbemgkcJ55F7H0=">AAAB83icbVBNS8NAEN3Ur1q/qh69LBbBU0lE0GPRi8cK9gOaUDbbSbt0swm7E7GE/g0vHhTx6p/x5r9x2+agrQ8GHu/NMDMvTKUw6LrfTmltfWNzq7xd2dnd2z+oHh61TZJpDi2eyER3Q2ZACgUtFCihm2pgcSihE45vZ37nEbQRiXrASQpBzIZKRIIztJLvIzxhrjNFvWm/WnPr7hx0lXgFqZECzX71yx8kPItBIZfMmJ7nphjkTKPgEqYVPzOQMj5mQ+hZqlgMJsjnN0/pmVUGNEq0LYV0rv6eyFlszCQObWfMcGSWvZn4n9fLMLoOcqHSDEHxxaIokxQTOguADoQGjnJiCeNa2FspHzHNONqYKjYEb/nlVdK+qHtu3bu/rDVuijjK5IScknPikSvSIHekSVqEk5Q8k1fy5mTOi/PufCxaS04xc0z+wPn8ARQ2kbE=</latexit><latexit sha1_base64="30We44UabOXpSdbemgkcJ55F7H0=">AAAB83icbVBNS8NAEN3Ur1q/qh69LBbBU0lE0GPRi8cK9gOaUDbbSbt0swm7E7GE/g0vHhTx6p/x5r9x2+agrQ8GHu/NMDMvTKUw6LrfTmltfWNzq7xd2dnd2z+oHh61TZJpDi2eyER3Q2ZACgUtFCihm2pgcSihE45vZ37nEbQRiXrASQpBzIZKRIIztJLvIzxhrjNFvWm/WnPr7hx0lXgFqZECzX71yx8kPItBIZfMmJ7nphjkTKPgEqYVPzOQMj5mQ+hZqlgMJsjnN0/pmVUGNEq0LYV0rv6eyFlszCQObWfMcGSWvZn4n9fLMLoOcqHSDEHxxaIokxQTOguADoQGjnJiCeNa2FspHzHNONqYKjYEb/nlVdK+qHtu3bu/rDVuijjK5IScknPikSvSIHekSVqEk5Q8k1fy5mTOi/PufCxaS04xc0z+wPn8ARQ2kbE=</latexit><latexit sha1_base64="30We44UabOXpSdbemgkcJ55F7H0=">AAAB83icbVBNS8NAEN3Ur1q/qh69LBbBU0lE0GPRi8cK9gOaUDbbSbt0swm7E7GE/g0vHhTx6p/x5r9x2+agrQ8GHu/NMDMvTKUw6LrfTmltfWNzq7xd2dnd2z+oHh61TZJpDi2eyER3Q2ZACgUtFCihm2pgcSihE45vZ37nEbQRiXrASQpBzIZKRIIztJLvIzxhrjNFvWm/WnPr7hx0lXgFqZECzX71yx8kPItBIZfMmJ7nphjkTKPgEqYVPzOQMj5mQ+hZqlgMJsjnN0/pmVUGNEq0LYV0rv6eyFlszCQObWfMcGSWvZn4n9fLMLoOcqHSDEHxxaIokxQTOguADoQGjnJiCeNa2FspHzHNONqYKjYEb/nlVdK+qHtu3bu/rDVuijjK5IScknPikSvSIHekSVqEk5Q8k1fy5mTOi/PufCxaS04xc0z+wPn8ARQ2kbE=</latexit><latexit sha1_base64="30We44UabOXpSdbemgkcJ55F7H0=">AAAB83icbVBNS8NAEN3Ur1q/qh69LBbBU0lE0GPRi8cK9gOaUDbbSbt0swm7E7GE/g0vHhTx6p/x5r9x2+agrQ8GHu/NMDMvTKUw6LrfTmltfWNzq7xd2dnd2z+oHh61TZJpDi2eyER3Q2ZACgUtFCihm2pgcSihE45vZ37nEbQRiXrASQpBzIZKRIIztJLvIzxhrjNFvWm/WnPr7hx0lXgFqZECzX71yx8kPItBIZfMmJ7nphjkTKPgEqYVPzOQMj5mQ+hZqlgMJsjnN0/pmVUGNEq0LYV0rv6eyFlszCQObWfMcGSWvZn4n9fLMLoOcqHSDEHxxaIokxQTOguADoQGjnJiCeNa2FspHzHNONqYKjYEb/nlVdK+qHtu3bu/rDVuijjK5IScknPikSvSIHekSVqEk5Q8k1fy5mTOi/PufCxaS04xc0z+wPn8ARQ2kbE=</latexit><latexit sha1_base64="30We44UabOXpSdbemgkcJ55F7H0=">AAAB83icbVBNS8NAEN3Ur1q/qh69LBbBU0lE0GPRi8cK9gOaUDbbSbt0swm7E7GE/g0vHhTx6p/x5r9x2+agrQ8GHu/NMDMvTKUw6LrfTmltfWNzq7xd2dnd2z+oHh61TZJpDi2eyER3Q2ZACgUtFCihm2pgcSihE45vZ37nEbQRiXrASQpBzIZKRIIztJLvIzxhrjNFvWm/WnPr7hx0lXgFqZECzX71yx8kPItBIZfMmJ7nphjkTKPgEqYVPzOQMj5mQ+hZqlgMJsjnN0/pmVUGNEq0LYV0rv6eyFlszCQObWfMcGSWvZn4n9fLMLoOcqHSDEHxxaIokxQTOguADoQGjnJiCeNa2FspHzHNONqYKjYEb/nlVdK+qHtu3bu/rDVuijjK5IScknPikSvSIHekSVqEk5Q8k1fy5mTOi/PufCxaS04xc0z+wPn8ARQ2kbE=</latexit>
run 2 <latexit sha1_base64="/8e8xoWKKMoIUxa6M49bnDNbI9o=">AAAB83icbVBNS8NAEN34WetX1aOXxSJ4KkkR9Fj04rGC/YAmlM120i7dbMLuRCyhf8OLB0W8+me8+W/ctjlo64OBx3szzMwLUykMuu63s7a+sbm1Xdop7+7tHxxWjo7bJsk0hxZPZKK7ITMghYIWCpTQTTWwOJTQCce3M7/zCNqIRD3gJIUgZkMlIsEZWsn3EZ4w15mi9Wm/UnVr7hx0lXgFqZICzX7lyx8kPItBIZfMmJ7nphjkTKPgEqZlPzOQMj5mQ+hZqlgMJsjnN0/puVUGNEq0LYV0rv6eyFlszCQObWfMcGSWvZn4n9fLMLoOcqHSDEHxxaIokxQTOguADoQGjnJiCeNa2FspHzHNONqYyjYEb/nlVdKu1zy35t1fVhs3RRwlckrOyAXxyBVpkDvSJC3CSUqeySt5czLnxXl3Phata04xc0L+wPn8ARW7kbI=</latexit><latexit sha1_base64="/8e8xoWKKMoIUxa6M49bnDNbI9o=">AAAB83icbVBNS8NAEN34WetX1aOXxSJ4KkkR9Fj04rGC/YAmlM120i7dbMLuRCyhf8OLB0W8+me8+W/ctjlo64OBx3szzMwLUykMuu63s7a+sbm1Xdop7+7tHxxWjo7bJsk0hxZPZKK7ITMghYIWCpTQTTWwOJTQCce3M7/zCNqIRD3gJIUgZkMlIsEZWsn3EZ4w15mi9Wm/UnVr7hx0lXgFqZICzX7lyx8kPItBIZfMmJ7nphjkTKPgEqZlPzOQMj5mQ+hZqlgMJsjnN0/puVUGNEq0LYV0rv6eyFlszCQObWfMcGSWvZn4n9fLMLoOcqHSDEHxxaIokxQTOguADoQGjnJiCeNa2FspHzHNONqYyjYEb/nlVdKu1zy35t1fVhs3RRwlckrOyAXxyBVpkDvSJC3CSUqeySt5czLnxXl3Phata04xc0L+wPn8ARW7kbI=</latexit><latexit sha1_base64="/8e8xoWKKMoIUxa6M49bnDNbI9o=">AAAB83icbVBNS8NAEN34WetX1aOXxSJ4KkkR9Fj04rGC/YAmlM120i7dbMLuRCyhf8OLB0W8+me8+W/ctjlo64OBx3szzMwLUykMuu63s7a+sbm1Xdop7+7tHxxWjo7bJsk0hxZPZKK7ITMghYIWCpTQTTWwOJTQCce3M7/zCNqIRD3gJIUgZkMlIsEZWsn3EZ4w15mi9Wm/UnVr7hx0lXgFqZICzX7lyx8kPItBIZfMmJ7nphjkTKPgEqZlPzOQMj5mQ+hZqlgMJsjnN0/puVUGNEq0LYV0rv6eyFlszCQObWfMcGSWvZn4n9fLMLoOcqHSDEHxxaIokxQTOguADoQGjnJiCeNa2FspHzHNONqYyjYEb/nlVdKu1zy35t1fVhs3RRwlckrOyAXxyBVpkDvSJC3CSUqeySt5czLnxXl3Phata04xc0L+wPn8ARW7kbI=</latexit><latexit sha1_base64="/8e8xoWKKMoIUxa6M49bnDNbI9o=">AAAB83icbVBNS8NAEN34WetX1aOXxSJ4KkkR9Fj04rGC/YAmlM120i7dbMLuRCyhf8OLB0W8+me8+W/ctjlo64OBx3szzMwLUykMuu63s7a+sbm1Xdop7+7tHxxWjo7bJsk0hxZPZKK7ITMghYIWCpTQTTWwOJTQCce3M7/zCNqIRD3gJIUgZkMlIsEZWsn3EZ4w15mi9Wm/UnVr7hx0lXgFqZICzX7lyx8kPItBIZfMmJ7nphjkTKPgEqZlPzOQMj5mQ+hZqlgMJsjnN0/puVUGNEq0LYV0rv6eyFlszCQObWfMcGSWvZn4n9fLMLoOcqHSDEHxxaIokxQTOguADoQGjnJiCeNa2FspHzHNONqYyjYEb/nlVdKu1zy35t1fVhs3RRwlckrOyAXxyBVpkDvSJC3CSUqeySt5czLnxXl3Phata04xc0L+wPn8ARW7kbI=</latexit>
run 4 <latexit sha1_base64="knQT6fZNf3khW5vGun9JeS0BMIs=">AAAB83icbVBNS8NAEN3Ur1q/qh69LBbBU0lE0GPRi8cK9gOaUDbbSbt0swm7E7GE/g0vHhTx6p/x5r9x2+agrQ8GHu/NMDMvTKUw6LrfTmltfWNzq7xd2dnd2z+oHh61TZJpDi2eyER3Q2ZACgUtFCihm2pgcSihE45vZ37nEbQRiXrASQpBzIZKRIIztJLvIzxhrjNFL6f9as2tu3PQVeIVpEYKNPvVL3+Q8CwGhVwyY3qem2KQM42CS5hW/MxAyviYDaFnqWIxmCCf3zylZ1YZ0CjRthTSufp7ImexMZM4tJ0xw5FZ9mbif14vw+g6yIVKMwTFF4uiTFJM6CwAOhAaOMqJJYxrYW+lfMQ042hjqtgQvOWXV0n7ou65de/+sta4KeIokxNySs6JR65Ig9yRJmkRTlLyTF7Jm5M5L86787FoLTnFzDH5A+fzBxjFkbQ=</latexit><latexit sha1_base64="knQT6fZNf3khW5vGun9JeS0BMIs=">AAAB83icbVBNS8NAEN3Ur1q/qh69LBbBU0lE0GPRi8cK9gOaUDbbSbt0swm7E7GE/g0vHhTx6p/x5r9x2+agrQ8GHu/NMDMvTKUw6LrfTmltfWNzq7xd2dnd2z+oHh61TZJpDi2eyER3Q2ZACgUtFCihm2pgcSihE45vZ37nEbQRiXrASQpBzIZKRIIztJLvIzxhrjNFL6f9as2tu3PQVeIVpEYKNPvVL3+Q8CwGhVwyY3qem2KQM42CS5hW/MxAyviYDaFnqWIxmCCf3zylZ1YZ0CjRthTSufp7ImexMZM4tJ0xw5FZ9mbif14vw+g6yIVKMwTFF4uiTFJM6CwAOhAaOMqJJYxrYW+lfMQ042hjqtgQvOWXV0n7ou65de/+sta4KeIokxNySs6JR65Ig9yRJmkRTlLyTF7Jm5M5L86787FoLTnFzDH5A+fzBxjFkbQ=</latexit><latexit sha1_base64="knQT6fZNf3khW5vGun9JeS0BMIs=">AAAB83icbVBNS8NAEN3Ur1q/qh69LBbBU0lE0GPRi8cK9gOaUDbbSbt0swm7E7GE/g0vHhTx6p/x5r9x2+agrQ8GHu/NMDMvTKUw6LrfTmltfWNzq7xd2dnd2z+oHh61TZJpDi2eyER3Q2ZACgUtFCihm2pgcSihE45vZ37nEbQRiXrASQpBzIZKRIIztJLvIzxhrjNFL6f9as2tu3PQVeIVpEYKNPvVL3+Q8CwGhVwyY3qem2KQM42CS5hW/MxAyviYDaFnqWIxmCCf3zylZ1YZ0CjRthTSufp7ImexMZM4tJ0xw5FZ9mbif14vw+g6yIVKMwTFF4uiTFJM6CwAOhAaOMqJJYxrYW+lfMQ042hjqtgQvOWXV0n7ou65de/+sta4KeIokxNySs6JR65Ig9yRJmkRTlLyTF7Jm5M5L86787FoLTnFzDH5A+fzBxjFkbQ=</latexit><latexit sha1_base64="knQT6fZNf3khW5vGun9JeS0BMIs=">AAAB83icbVBNS8NAEN3Ur1q/qh69LBbBU0lE0GPRi8cK9gOaUDbbSbt0swm7E7GE/g0vHhTx6p/x5r9x2+agrQ8GHu/NMDMvTKUw6LrfTmltfWNzq7xd2dnd2z+oHh61TZJpDi2eyER3Q2ZACgUtFCihm2pgcSihE45vZ37nEbQRiXrASQpBzIZKRIIztJLvIzxhrjNFL6f9as2tu3PQVeIVpEYKNPvVL3+Q8CwGhVwyY3qem2KQM42CS5hW/MxAyviYDaFnqWIxmCCf3zylZ1YZ0CjRthTSufp7ImexMZM4tJ0xw5FZ9mbif14vw+g6yIVKMwTFF4uiTFJM6CwAOhAaOMqJJYxrYW+lfMQ042hjqtgQvOWXV0n7ou65de/+sta4KeIokxNySs6JR65Ig9yRJmkRTlLyTF7Jm5M5L86787FoLTnFzDH5A+fzBxjFkbQ=</latexit> run 3 <latexit sha1_base64="qR259uL/I8nt87cuvQbEbaJv0KQ=">AAAB83icbVBNS8NAEJ3Ur1q/qh69LBbBU0lU0GPRi8cKthaaUDbbTbt0swm7E7GE/g0vHhTx6p/x5r9x2+agrQ8GHu/NMDMvTKUw6LrfTmlldW19o7xZ2dre2d2r7h+0TZJpxlsskYnuhNRwKRRvoUDJO6nmNA4lfwhHN1P/4ZFrIxJ1j+OUBzEdKBEJRtFKvo/8CXOdKXI+6VVrbt2dgSwTryA1KNDsVb/8fsKymCtkkhrT9dwUg5xqFEzyScXPDE8pG9EB71qqaMxNkM9unpATq/RJlGhbCslM/T2R09iYcRzazpji0Cx6U/E/r5thdBXkQqUZcsXmi6JMEkzINADSF5ozlGNLKNPC3krYkGrK0MZUsSF4iy8vk/ZZ3XPr3t1FrXFdxFGGIziGU/DgEhpwC01oAYMUnuEV3pzMeXHenY95a8kpZg7hD5zPHxdAkbM=</latexit><latexit sha1_base64="qR259uL/I8nt87cuvQbEbaJv0KQ=">AAAB83icbVBNS8NAEJ3Ur1q/qh69LBbBU0lU0GPRi8cKthaaUDbbTbt0swm7E7GE/g0vHhTx6p/x5r9x2+agrQ8GHu/NMDMvTKUw6LrfTmlldW19o7xZ2dre2d2r7h+0TZJpxlsskYnuhNRwKRRvoUDJO6nmNA4lfwhHN1P/4ZFrIxJ1j+OUBzEdKBEJRtFKvo/8CXOdKXI+6VVrbt2dgSwTryA1KNDsVb/8fsKymCtkkhrT9dwUg5xqFEzyScXPDE8pG9EB71qqaMxNkM9unpATq/RJlGhbCslM/T2R09iYcRzazpji0Cx6U/E/r5thdBXkQqUZcsXmi6JMEkzINADSF5ozlGNLKNPC3krYkGrK0MZUsSF4iy8vk/ZZ3XPr3t1FrXFdxFGGIziGU/DgEhpwC01oAYMUnuEV3pzMeXHenY95a8kpZg7hD5zPHxdAkbM=</latexit><latexit sha1_base64="qR259uL/I8nt87cuvQbEbaJv0KQ=">AAAB83icbVBNS8NAEJ3Ur1q/qh69LBbBU0lU0GPRi8cKthaaUDbbTbt0swm7E7GE/g0vHhTx6p/x5r9x2+agrQ8GHu/NMDMvTKUw6LrfTmlldW19o7xZ2dre2d2r7h+0TZJpxlsskYnuhNRwKRRvoUDJO6nmNA4lfwhHN1P/4ZFrIxJ1j+OUBzEdKBEJRtFKvo/8CXOdKXI+6VVrbt2dgSwTryA1KNDsVb/8fsKymCtkkhrT9dwUg5xqFEzyScXPDE8pG9EB71qqaMxNkM9unpATq/RJlGhbCslM/T2R09iYcRzazpji0Cx6U/E/r5thdBXkQqUZcsXmi6JMEkzINADSF5ozlGNLKNPC3krYkGrK0MZUsSF4iy8vk/ZZ3XPr3t1FrXFdxFGGIziGU/DgEhpwC01oAYMUnuEV3pzMeXHenY95a8kpZg7hD5zPHxdAkbM=</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="TK7ci21hazYsI2n8wtmNnUauRV0=">AAAB6HicbZDNSgMxFIXv1L9aq1a3boJFcFVmdKFLwY3LCrYV2qFk0jttaCYzJHfEMvQ13LhQxCdy59uY/iy09UDg45yEe3OiTElLvv/tlTY2t7Z3yruVver+wWHtqNq2aW4EtkSqUvMYcYtKamyRJIWPmUGeRAo70fh2lnee0FiZ6geaZBgmfKhlLAUnZ/V6hM9UmFyzy2m/Vvcb/lxsHYIl1GGpZr/21RukIk9Qk1Dc2m7gZxQW3JAUCqeVXm4x42LMh9h1qHmCNizmO0/ZmXMGLE6NO5rY3P39ouCJtZMkcjcTTiO7ms3M/7JuTvF1WEid5YRaLAbFuWKUslkBbCANClITB1wY6XZlYsQNF+RqqrgSgtUvr0P7ohH4jeDehzKcwCmcQwBXcAN30IQWCMjgBd7g3cu9V+9jUVfJW/Z2DH/kff4A0KiQVQ==</latexit><latexit sha1_base64="TK7ci21hazYsI2n8wtmNnUauRV0=">AAAB6HicbZDNSgMxFIXv1L9aq1a3boJFcFVmdKFLwY3LCrYV2qFk0jttaCYzJHfEMvQ13LhQxCdy59uY/iy09UDg45yEe3OiTElLvv/tlTY2t7Z3yruVver+wWHtqNq2aW4EtkSqUvMYcYtKamyRJIWPmUGeRAo70fh2lnee0FiZ6geaZBgmfKhlLAUnZ/V6hM9UmFyzy2m/Vvcb/lxsHYIl1GGpZr/21RukIk9Qk1Dc2m7gZxQW3JAUCqeVXm4x42LMh9h1qHmCNizmO0/ZmXMGLE6NO5rY3P39ouCJtZMkcjcTTiO7ms3M/7JuTvF1WEid5YRaLAbFuWKUslkBbCANClITB1wY6XZlYsQNF+RqqrgSgtUvr0P7ohH4jeDehzKcwCmcQwBXcAN30IQWCMjgBd7g3cu9V+9jUVfJW/Z2DH/kff4A0KiQVQ==</latexit><latexit sha1_base64="Td1HvCil+CJpCYxnyj+OwEYqyB0=">AAAB83icbVBNS8NAEJ3Ur1q/qh69LBbBU0n0oMeiF48V7Ac0oWy2m3bpZhN2J2IJ/RtePCji1T/jzX/jts1BWx8MPN6bYWZemEph0HW/ndLa+sbmVnm7srO7t39QPTxqmyTTjLdYIhPdDanhUijeQoGSd1PNaRxK3gnHtzO/88i1EYl6wEnKg5gOlYgEo2gl30f+hLnOFLmc9qs1t+7OQVaJV5AaFGj2q1/+IGFZzBUySY3peW6KQU41Cib5tOJnhqeUjemQ9yxVNOYmyOc3T8mZVQYkSrQthWSu/p7IaWzMJA5tZ0xxZJa9mfif18swug5yodIMuWKLRVEmCSZkFgAZCM0ZyokllGlhbyVsRDVlaGOq2BC85ZdXSfui7rl1796tNW6KOMpwAqdwDh5cQQPuoAktYJDCM7zCm5M5L86787FoLTnFzDH8gfP5AxYAka8=</latexit><latexit sha1_base64="qR259uL/I8nt87cuvQbEbaJv0KQ=">AAAB83icbVBNS8NAEJ3Ur1q/qh69LBbBU0lU0GPRi8cKthaaUDbbTbt0swm7E7GE/g0vHhTx6p/x5r9x2+agrQ8GHu/NMDMvTKUw6LrfTmlldW19o7xZ2dre2d2r7h+0TZJpxlsskYnuhNRwKRRvoUDJO6nmNA4lfwhHN1P/4ZFrIxJ1j+OUBzEdKBEJRtFKvo/8CXOdKXI+6VVrbt2dgSwTryA1KNDsVb/8fsKymCtkkhrT9dwUg5xqFEzyScXPDE8pG9EB71qqaMxNkM9unpATq/RJlGhbCslM/T2R09iYcRzazpji0Cx6U/E/r5thdBXkQqUZcsXmi6JMEkzINADSF5ozlGNLKNPC3krYkGrK0MZUsSF4iy8vk/ZZ3XPr3t1FrXFdxFGGIziGU/DgEhpwC01oAYMUnuEV3pzMeXHenY95a8kpZg7hD5zPHxdAkbM=</latexit><latexit sha1_base64="qR259uL/I8nt87cuvQbEbaJv0KQ=">AAAB83icbVBNS8NAEJ3Ur1q/qh69LBbBU0lU0GPRi8cKthaaUDbbTbt0swm7E7GE/g0vHhTx6p/x5r9x2+agrQ8GHu/NMDMvTKUw6LrfTmlldW19o7xZ2dre2d2r7h+0TZJpxlsskYnuhNRwKRRvoUDJO6nmNA4lfwhHN1P/4ZFrIxJ1j+OUBzEdKBEJRtFKvo/8CXOdKXI+6VVrbt2dgSwTryA1KNDsVb/8fsKymCtkkhrT9dwUg5xqFEzyScXPDE8pG9EB71qqaMxNkM9unpATq/RJlGhbCslM/T2R09iYcRzazpji0Cx6U/E/r5thdBXkQqUZcsXmi6JMEkzINADSF5ozlGNLKNPC3krYkGrK0MZUsSF4iy8vk/ZZ3XPr3t1FrXFdxFGGIziGU/DgEhpwC01oAYMUnuEV3pzMeXHenY95a8kpZg7hD5zPHxdAkbM=</latexit><latexit sha1_base64="qR259uL/I8nt87cuvQbEbaJv0KQ=">AAAB83icbVBNS8NAEJ3Ur1q/qh69LBbBU0lU0GPRi8cKthaaUDbbTbt0swm7E7GE/g0vHhTx6p/x5r9x2+agrQ8GHu/NMDMvTKUw6LrfTmlldW19o7xZ2dre2d2r7h+0TZJpxlsskYnuhNRwKRRvoUDJO6nmNA4lfwhHN1P/4ZFrIxJ1j+OUBzEdKBEJRtFKvo/8CXOdKXI+6VVrbt2dgSwTryA1KNDsVb/8fsKymCtkkhrT9dwUg5xqFEzyScXPDE8pG9EB71qqaMxNkM9unpATq/RJlGhbCslM/T2R09iYcRzazpji0Cx6U/E/r5thdBXkQqUZcsXmi6JMEkzINADSF5ozlGNLKNPC3krYkGrK0MZUsSF4iy8vk/ZZ3XPr3t1FrXFdxFGGIziGU/DgEhpwC01oAYMUnuEV3pzMeXHenY95a8kpZg7hD5zPHxdAkbM=</latexit><latexit sha1_base64="qR259uL/I8nt87cuvQbEbaJv0KQ=">AAAB83icbVBNS8NAEJ3Ur1q/qh69LBbBU0lU0GPRi8cKthaaUDbbTbt0swm7E7GE/g0vHhTx6p/x5r9x2+agrQ8GHu/NMDMvTKUw6LrfTmlldW19o7xZ2dre2d2r7h+0TZJpxlsskYnuhNRwKRRvoUDJO6nmNA4lfwhHN1P/4ZFrIxJ1j+OUBzEdKBEJRtFKvo/8CXOdKXI+6VVrbt2dgSwTryA1KNDsVb/8fsKymCtkkhrT9dwUg5xqFEzyScXPDE8pG9EB71qqaMxNkM9unpATq/RJlGhbCslM/T2R09iYcRzazpji0Cx6U/E/r5thdBXkQqUZcsXmi6JMEkzINADSF5ozlGNLKNPC3krYkGrK0MZUsSF4iy8vk/ZZ3XPr3t1FrXFdxFGGIziGU/DgEhpwC01oAYMUnuEV3pzMeXHenY95a8kpZg7hD5zPHxdAkbM=</latexit><latexit sha1_base64="qR259uL/I8nt87cuvQbEbaJv0KQ=">AAAB83icbVBNS8NAEJ3Ur1q/qh69LBbBU0lU0GPRi8cKthaaUDbbTbt0swm7E7GE/g0vHhTx6p/x5r9x2+agrQ8GHu/NMDMvTKUw6LrfTmlldW19o7xZ2dre2d2r7h+0TZJpxlsskYnuhNRwKRRvoUDJO6nmNA4lfwhHN1P/4ZFrIxJ1j+OUBzEdKBEJRtFKvo/8CXOdKXI+6VVrbt2dgSwTryA1KNDsVb/8fsKymCtkkhrT9dwUg5xqFEzyScXPDE8pG9EB71qqaMxNkM9unpATq/RJlGhbCslM/T2R09iYcRzazpji0Cx6U/E/r5thdBXkQqUZcsXmi6JMEkzINADSF5ozlGNLKNPC3krYkGrK0MZUsSF4iy8vk/ZZ3XPr3t1FrXFdxFGGIziGU/DgEhpwC01oAYMUnuEV3pzMeXHenY95a8kpZg7hD5zPHxdAkbM=</latexit><latexit sha1_base64="qR259uL/I8nt87cuvQbEbaJv0KQ=">AAAB83icbVBNS8NAEJ3Ur1q/qh69LBbBU0lU0GPRi8cKthaaUDbbTbt0swm7E7GE/g0vHhTx6p/x5r9x2+agrQ8GHu/NMDMvTKUw6LrfTmlldW19o7xZ2dre2d2r7h+0TZJpxlsskYnuhNRwKRRvoUDJO6nmNA4lfwhHN1P/4ZFrIxJ1j+OUBzEdKBEJRtFKvo/8CXOdKXI+6VVrbt2dgSwTryA1KNDsVb/8fsKymCtkkhrT9dwUg5xqFEzyScXPDE8pG9EB71qqaMxNkM9unpATq/RJlGhbCslM/T2R09iYcRzazpji0Cx6U/E/r5thdBXkQqUZcsXmi6JMEkzINADSF5ozlGNLKNPC3krYkGrK0MZUsSF4iy8vk/ZZ3XPr3t1FrXFdxFGGIziGU/DgEhpwC01oAYMUnuEV3pzMeXHenY95a8kpZg7hD5zPHxdAkbM=</latexit>
Figure 10: Four quantities shown during training for the combined agent. We show reward, norm of actor gradient ‖∇`‖, norm of actor weight ‖w‖ and average absolute action |a|. The actions never saturate and learning starts quickly. Compare with Figure 3, which shows the same quantities for the baseline algorithm.
run 1 <latexit sha1_base64="K2LHkVIbGBbhAACjOdf7sVnMn+4=">AAAB9XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaWDbbTbt0swm7E7WE/g8vHhTx6n/x5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqxhsslrFuB9RwKRRvoEDJ24nmNAokbwWj66nfeuDaiFjd4TjhfkQHSoSCUbTSfRf5E+oo06ki3qRXrrhVdwayTLycVCBHvVf+6vZjlkZcIZPUmI7nJuhnVKNgkk9K3dTwhLIRHfCOpYpG3PjZ7OoJObFKn4SxtqWQzNTfExmNjBlHge2MKA7NojcV//M6KYaXfiZUkiJXbL4oTCXBmEwjIH2hOUM5toQyLeythA2ppgxtUCUbgrf48jJpnlU9t+rdnldqV3kcRTiCYzgFDy6gBjdQhwYw0PAMr/DmPDovzrvzMW8tOPnMIfyB8/kDuwSSpA==</latexit><latexit sha1_base64="K2LHkVIbGBbhAACjOdf7sVnMn+4=">AAAB9XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaWDbbTbt0swm7E7WE/g8vHhTx6n/x5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqxhsslrFuB9RwKRRvoEDJ24nmNAokbwWj66nfeuDaiFjd4TjhfkQHSoSCUbTSfRf5E+oo06ki3qRXrrhVdwayTLycVCBHvVf+6vZjlkZcIZPUmI7nJuhnVKNgkk9K3dTwhLIRHfCOpYpG3PjZ7OoJObFKn4SxtqWQzNTfExmNjBlHge2MKA7NojcV//M6KYaXfiZUkiJXbL4oTCXBmEwjIH2hOUM5toQyLeythA2ppgxtUCUbgrf48jJpnlU9t+rdnldqV3kcRTiCYzgFDy6gBjdQhwYw0PAMr/DmPDovzrvzMW8tOPnMIfyB8/kDuwSSpA==</latexit><latexit sha1_base64="K2LHkVIbGBbhAACjOdf7sVnMn+4=">AAAB9XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaWDbbTbt0swm7E7WE/g8vHhTx6n/x5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqxhsslrFuB9RwKRRvoEDJ24nmNAokbwWj66nfeuDaiFjd4TjhfkQHSoSCUbTSfRf5E+oo06ki3qRXrrhVdwayTLycVCBHvVf+6vZjlkZcIZPUmI7nJuhnVKNgkk9K3dTwhLIRHfCOpYpG3PjZ7OoJObFKn4SxtqWQzNTfExmNjBlHge2MKA7NojcV//M6KYaXfiZUkiJXbL4oTCXBmEwjIH2hOUM5toQyLeythA2ppgxtUCUbgrf48jJpnlU9t+rdnldqV3kcRTiCYzgFDy6gBjdQhwYw0PAMr/DmPDovzrvzMW8tOPnMIfyB8/kDuwSSpA==</latexit><latexit sha1_base64="K2LHkVIbGBbhAACjOdf7sVnMn+4=">AAAB9XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaWDbbTbt0swm7E7WE/g8vHhTx6n/x5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqxhsslrFuB9RwKRRvoEDJ24nmNAokbwWj66nfeuDaiFjd4TjhfkQHSoSCUbTSfRf5E+oo06ki3qRXrrhVdwayTLycVCBHvVf+6vZjlkZcIZPUmI7nJuhnVKNgkk9K3dTwhLIRHfCOpYpG3PjZ7OoJObFKn4SxtqWQzNTfExmNjBlHge2MKA7NojcV//M6KYaXfiZUkiJXbL4oTCXBmEwjIH2hOUM5toQyLeythA2ppgxtUCUbgrf48jJpnlU9t+rdnldqV3kcRTiCYzgFDy6gBjdQhwYw0PAMr/DmPDovzrvzMW8tOPnMIfyB8/kDuwSSpA==</latexit>
run 2 <latexit sha1_base64="5Dgsnw1SpIG9RSwzoiyubX6nkEQ=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KkkR9Fj04rGC/YA2ls120i7dbMLuRC2h/8OLB0W8+l+8+W/ctjlo64OBx3szzMwLEikMuu63s7K6tr6xWdgqbu/s7u2XDg6bJk41hwaPZazbATMghYIGCpTQTjSwKJDQCkbXU7/1ANqIWN3hOAE/YgMlQsEZWum+i/CEOsp0qmh10iuV3Yo7A10mXk7KJEe9V/rq9mOeRqCQS2ZMx3MT9DOmUXAJk2I3NZAwPmID6FiqWATGz2ZXT+ipVfo0jLUthXSm/p7IWGTMOApsZ8RwaBa9qfif10kxvPQzoZIUQfH5ojCVFGM6jYD2hQaOcmwJ41rYWykfMs042qCKNgRv8eVl0qxWPLfi3Z6Xa1d5HAVyTE7IGfHIBamRG1InDcKJJs/klbw5j86L8+58zFtXnHzmiPyB8/kDvImSpQ==</latexit><latexit sha1_base64="5Dgsnw1SpIG9RSwzoiyubX6nkEQ=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KkkR9Fj04rGC/YA2ls120i7dbMLuRC2h/8OLB0W8+l+8+W/ctjlo64OBx3szzMwLEikMuu63s7K6tr6xWdgqbu/s7u2XDg6bJk41hwaPZazbATMghYIGCpTQTjSwKJDQCkbXU7/1ANqIWN3hOAE/YgMlQsEZWum+i/CEOsp0qmh10iuV3Yo7A10mXk7KJEe9V/rq9mOeRqCQS2ZMx3MT9DOmUXAJk2I3NZAwPmID6FiqWATGz2ZXT+ipVfo0jLUthXSm/p7IWGTMOApsZ8RwaBa9qfif10kxvPQzoZIUQfH5ojCVFGM6jYD2hQaOcmwJ41rYWykfMs042qCKNgRv8eVl0qxWPLfi3Z6Xa1d5HAVyTE7IGfHIBamRG1InDcKJJs/klbw5j86L8+58zFtXnHzmiPyB8/kDvImSpQ==</latexit><latexit sha1_base64="5Dgsnw1SpIG9RSwzoiyubX6nkEQ=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KkkR9Fj04rGC/YA2ls120i7dbMLuRC2h/8OLB0W8+l+8+W/ctjlo64OBx3szzMwLEikMuu63s7K6tr6xWdgqbu/s7u2XDg6bJk41hwaPZazbATMghYIGCpTQTjSwKJDQCkbXU7/1ANqIWN3hOAE/YgMlQsEZWum+i/CEOsp0qmh10iuV3Yo7A10mXk7KJEe9V/rq9mOeRqCQS2ZMx3MT9DOmUXAJk2I3NZAwPmID6FiqWATGz2ZXT+ipVfo0jLUthXSm/p7IWGTMOApsZ8RwaBa9qfif10kxvPQzoZIUQfH5ojCVFGM6jYD2hQaOcmwJ41rYWykfMs042qCKNgRv8eVl0qxWPLfi3Z6Xa1d5HAVyTE7IGfHIBamRG1InDcKJJs/klbw5j86L8+58zFtXnHzmiPyB8/kDvImSpQ==</latexit><latexit sha1_base64="5Dgsnw1SpIG9RSwzoiyubX6nkEQ=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KkkR9Fj04rGC/YA2ls120i7dbMLuRC2h/8OLB0W8+l+8+W/ctjlo64OBx3szzMwLEikMuu63s7K6tr6xWdgqbu/s7u2XDg6bJk41hwaPZazbATMghYIGCpTQTjSwKJDQCkbXU7/1ANqIWN3hOAE/YgMlQsEZWum+i/CEOsp0qmh10iuV3Yo7A10mXk7KJEe9V/rq9mOeRqCQS2ZMx3MT9DOmUXAJk2I3NZAwPmID6FiqWATGz2ZXT+ipVfo0jLUthXSm/p7IWGTMOApsZ8RwaBa9qfif10kxvPQzoZIUQfH5ojCVFGM6jYD2hQaOcmwJ41rYWykfMs042qCKNgRv8eVl0qxWPLfi3Z6Xa1d5HAVyTE7IGfHIBamRG1InDcKJJs/klbw5j86L8+58zFtXnHzmiPyB8/kDvImSpQ==</latexit>
run 3 <latexit sha1_base64="ozX/0t3JPmydgwQzsf6ljZW+R6o=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KokKeix68VjBfkAby2Y7aZduNmF3opbQ/+HFgyJe/S/e/Ddu2xy09cHA470ZZuYFiRQGXffbWVpeWV1bL2wUN7e2d3ZLe/sNE6eaQ53HMtatgBmQQkEdBUpoJRpYFEhoBsPrid98AG1ErO5wlIAfsb4SoeAMrXTfQXhCHWU6VfRs3C2V3Yo7BV0kXk7KJEetW/rq9GKeRqCQS2ZM23MT9DOmUXAJ42InNZAwPmR9aFuqWATGz6ZXj+mxVXo0jLUthXSq/p7IWGTMKApsZ8RwYOa9ifif104xvPQzoZIUQfHZojCVFGM6iYD2hAaOcmQJ41rYWykfMM042qCKNgRv/uVF0jiteG7Fuz0vV6/yOArkkByRE+KRC1IlN6RG6oQTTZ7JK3lzHp0X5935mLUuOfnMAfkD5/MHvg6Spg==</latexit><latexit sha1_base64="ozX/0t3JPmydgwQzsf6ljZW+R6o=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KokKeix68VjBfkAby2Y7aZduNmF3opbQ/+HFgyJe/S/e/Ddu2xy09cHA470ZZuYFiRQGXffbWVpeWV1bL2wUN7e2d3ZLe/sNE6eaQ53HMtatgBmQQkEdBUpoJRpYFEhoBsPrid98AG1ErO5wlIAfsb4SoeAMrXTfQXhCHWU6VfRs3C2V3Yo7BV0kXk7KJEetW/rq9GKeRqCQS2ZM23MT9DOmUXAJ42InNZAwPmR9aFuqWATGz6ZXj+mxVXo0jLUthXSq/p7IWGTMKApsZ8RwYOa9ifif104xvPQzoZIUQfHZojCVFGM6iYD2hAaOcmQJ41rYWykfMM042qCKNgRv/uVF0jiteG7Fuz0vV6/yOArkkByRE+KRC1IlN6RG6oQTTZ7JK3lzHp0X5935mLUuOfnMAfkD5/MHvg6Spg==</latexit><latexit sha1_base64="ozX/0t3JPmydgwQzsf6ljZW+R6o=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KokKeix68VjBfkAby2Y7aZduNmF3opbQ/+HFgyJe/S/e/Ddu2xy09cHA470ZZuYFiRQGXffbWVpeWV1bL2wUN7e2d3ZLe/sNE6eaQ53HMtatgBmQQkEdBUpoJRpYFEhoBsPrid98AG1ErO5wlIAfsb4SoeAMrXTfQXhCHWU6VfRs3C2V3Yo7BV0kXk7KJEetW/rq9GKeRqCQS2ZM23MT9DOmUXAJ42InNZAwPmR9aFuqWATGz6ZXj+mxVXo0jLUthXSq/p7IWGTMKApsZ8RwYOa9ifif104xvPQzoZIUQfHZojCVFGM6iYD2hAaOcmQJ41rYWykfMM042qCKNgRv/uVF0jiteG7Fuz0vV6/yOArkkByRE+KRC1IlN6RG6oQTTZ7JK3lzHp0X5935mLUuOfnMAfkD5/MHvg6Spg==</latexit><latexit sha1_base64="ozX/0t3JPmydgwQzsf6ljZW+R6o=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KokKeix68VjBfkAby2Y7aZduNmF3opbQ/+HFgyJe/S/e/Ddu2xy09cHA470ZZuYFiRQGXffbWVpeWV1bL2wUN7e2d3ZLe/sNE6eaQ53HMtatgBmQQkEdBUpoJRpYFEhoBsPrid98AG1ErO5wlIAfsb4SoeAMrXTfQXhCHWU6VfRs3C2V3Yo7BV0kXk7KJEetW/rq9GKeRqCQS2ZM23MT9DOmUXAJ42InNZAwPmR9aFuqWATGz6ZXj+mxVXo0jLUthXSq/p7IWGTMKApsZ8RwYOa9ifif104xvPQzoZIUQfHZojCVFGM6iYD2hAaOcmQJ41rYWykfMM042qCKNgRv/uVF0jiteG7Fuz0vV6/yOArkkByRE+KRC1IlN6RG6oQTTZ7JK3lzHp0X5935mLUuOfnMAfkD5/MHvg6Spg==</latexit>
run 4 <latexit sha1_base64="ue3WBNuo2mSgITFm5zybLczT74E=">AAAB9XicbVBNS8NAEN3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaWDbbSbt0swm7E7WE/g8vHhTx6n/x5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqDg0ey1i3A2ZACgUNFCihnWhgUSChFYyup37rAbQRsbrDcQJ+xAZKhIIztNJ9F+EJdZTpVNHzSa9ccavuDHSZeDmpkBz1Xvmr2495GoFCLpkxHc9N0M+YRsElTErd1EDC+IgNoGOpYhEYP5tdPaEnVunTMNa2FNKZ+nsiY5Ex4yiwnRHDoVn0puJ/XifF8NLPhEpSBMXni8JUUozpNALaFxo4yrEljGthb6V8yDTjaIMq2RC8xZeXSfOs6rlV7/a8UrvK4yiSI3JMTolHLkiN3JA6aRBONHkmr+TNeXRenHfnY95acPKZQ/IHzucPv5OSpw==</latexit><latexit sha1_base64="ue3WBNuo2mSgITFm5zybLczT74E=">AAAB9XicbVBNS8NAEN3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaWDbbSbt0swm7E7WE/g8vHhTx6n/x5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqDg0ey1i3A2ZACgUNFCihnWhgUSChFYyup37rAbQRsbrDcQJ+xAZKhIIztNJ9F+EJdZTpVNHzSa9ccavuDHSZeDmpkBz1Xvmr2495GoFCLpkxHc9N0M+YRsElTErd1EDC+IgNoGOpYhEYP5tdPaEnVunTMNa2FNKZ+nsiY5Ex4yiwnRHDoVn0puJ/XifF8NLPhEpSBMXni8JUUozpNALaFxo4yrEljGthb6V8yDTjaIMq2RC8xZeXSfOs6rlV7/a8UrvK4yiSI3JMTolHLkiN3JA6aRBONHkmr+TNeXRenHfnY95acPKZQ/IHzucPv5OSpw==</latexit><latexit sha1_base64="ue3WBNuo2mSgITFm5zybLczT74E=">AAAB9XicbVBNS8NAEN3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaWDbbSbt0swm7E7WE/g8vHhTx6n/x5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqDg0ey1i3A2ZACgUNFCihnWhgUSChFYyup37rAbQRsbrDcQJ+xAZKhIIztNJ9F+EJdZTpVNHzSa9ccavuDHSZeDmpkBz1Xvmr2495GoFCLpkxHc9N0M+YRsElTErd1EDC+IgNoGOpYhEYP5tdPaEnVunTMNa2FNKZ+nsiY5Ex4yiwnRHDoVn0puJ/XifF8NLPhEpSBMXni8JUUozpNALaFxo4yrEljGthb6V8yDTjaIMq2RC8xZeXSfOs6rlV7/a8UrvK4yiSI3JMTolHLkiN3JA6aRBONHkmr+TNeXRenHfnY95acPKZQ/IHzucPv5OSpw==</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="/tomWnPbbdkqRAl0U8LNJQYXIBo=">AAAB6nicbZBLSwMxFIXv+Ky1anXrJlgEV2VGBF0KblxWsA9oa8mkd9rQTGZI7qhl6P9w40IRf5A7/43pY6GtBwIf5yTcmxOmSlry/W9vbX1jc2u7sFPcLe3tH5QPSw2bZEZgXSQqMa2QW1RSY50kKWylBnkcKmyGo5tp3nxEY2Wi72mcYjfmAy0jKTg566FD+Ewmzk2m2cWkV674VX8mtgrBAiqwUK1X/ur0E5HFqEkobm078FPq5tyQFAonxU5mMeVixAfYdqh5jLabz7aesFPn9FmUGHc0sZn7+0XOY2vHcehuxpyGdjmbmv9l7Yyiq24udZoRajEfFGWKUcKmFbC+NChIjR1wYaTblYkhN1yQK6roSgiWv7wKjfNq4FeDOx8KcAwncAYBXMI13EIN6iDAwAu8wbv35L16H/O61rxFb0fwR97nD27qkUc=</latexit><latexit sha1_base64="/tomWnPbbdkqRAl0U8LNJQYXIBo=">AAAB6nicbZBLSwMxFIXv+Ky1anXrJlgEV2VGBF0KblxWsA9oa8mkd9rQTGZI7qhl6P9w40IRf5A7/43pY6GtBwIf5yTcmxOmSlry/W9vbX1jc2u7sFPcLe3tH5QPSw2bZEZgXSQqMa2QW1RSY50kKWylBnkcKmyGo5tp3nxEY2Wi72mcYjfmAy0jKTg566FD+Ewmzk2m2cWkV674VX8mtgrBAiqwUK1X/ur0E5HFqEkobm078FPq5tyQFAonxU5mMeVixAfYdqh5jLabz7aesFPn9FmUGHc0sZn7+0XOY2vHcehuxpyGdjmbmv9l7Yyiq24udZoRajEfFGWKUcKmFbC+NChIjR1wYaTblYkhN1yQK6roSgiWv7wKjfNq4FeDOx8KcAwncAYBXMI13EIN6iDAwAu8wbv35L16H/O61rxFb0fwR97nD27qkUc=</latexit><latexit sha1_base64="s84kqqt315Hp39e4bdrCATvszSI=">AAAB9XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaWDbbTbt0swm7E7WE/g8vHhTx6n/x5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqxhsslrFuB9RwKRRvoEDJ24nmNAokbwWj66nfeuDaiFjd4TjhfkQHSoSCUbTSfRf5E+oo06ki55NeueJW3RnIMvFyUoEc9V75q9uPWRpxhUxSYzqem6CfUY2CST4pdVPDE8pGdMA7lioaceNns6sn5MQqfRLG2pZCMlN/T2Q0MmYcBbYzojg0i95U/M/rpBhe+plQSYpcsfmiMJUEYzKNgPSF5gzl2BLKtLC3EjakmjK0QZVsCN7iy8ukeVb13Kp361ZqV3kcRTiCYzgFDy6gBjdQhwYw0PAMr/DmPDovzrvzMW8tOPnMIfyB8/kDvlOSow==</latexit><latexit sha1_base64="ue3WBNuo2mSgITFm5zybLczT74E=">AAAB9XicbVBNS8NAEN3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaWDbbSbt0swm7E7WE/g8vHhTx6n/x5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqDg0ey1i3A2ZACgUNFCihnWhgUSChFYyup37rAbQRsbrDcQJ+xAZKhIIztNJ9F+EJdZTpVNHzSa9ccavuDHSZeDmpkBz1Xvmr2495GoFCLpkxHc9N0M+YRsElTErd1EDC+IgNoGOpYhEYP5tdPaEnVunTMNa2FNKZ+nsiY5Ex4yiwnRHDoVn0puJ/XifF8NLPhEpSBMXni8JUUozpNALaFxo4yrEljGthb6V8yDTjaIMq2RC8xZeXSfOs6rlV7/a8UrvK4yiSI3JMTolHLkiN3JA6aRBONHkmr+TNeXRenHfnY95acPKZQ/IHzucPv5OSpw==</latexit><latexit sha1_base64="ue3WBNuo2mSgITFm5zybLczT74E=">AAAB9XicbVBNS8NAEN3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaWDbbSbt0swm7E7WE/g8vHhTx6n/x5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqDg0ey1i3A2ZACgUNFCihnWhgUSChFYyup37rAbQRsbrDcQJ+xAZKhIIztNJ9F+EJdZTpVNHzSa9ccavuDHSZeDmpkBz1Xvmr2495GoFCLpkxHc9N0M+YRsElTErd1EDC+IgNoGOpYhEYP5tdPaEnVunTMNa2FNKZ+nsiY5Ex4yiwnRHDoVn0puJ/XifF8NLPhEpSBMXni8JUUozpNALaFxo4yrEljGthb6V8yDTjaIMq2RC8xZeXSfOs6rlV7/a8UrvK4yiSI3JMTolHLkiN3JA6aRBONHkmr+TNeXRenHfnY95acPKZQ/IHzucPv5OSpw==</latexit><latexit sha1_base64="ue3WBNuo2mSgITFm5zybLczT74E=">AAAB9XicbVBNS8NAEN3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaWDbbSbt0swm7E7WE/g8vHhTx6n/x5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqDg0ey1i3A2ZACgUNFCihnWhgUSChFYyup37rAbQRsbrDcQJ+xAZKhIIztNJ9F+EJdZTpVNHzSa9ccavuDHSZeDmpkBz1Xvmr2495GoFCLpkxHc9N0M+YRsElTErd1EDC+IgNoGOpYhEYP5tdPaEnVunTMNa2FNKZ+nsiY5Ex4yiwnRHDoVn0puJ/XifF8NLPhEpSBMXni8JUUozpNALaFxo4yrEljGthb6V8yDTjaIMq2RC8xZeXSfOs6rlV7/a8UrvK4yiSI3JMTolHLkiN3JA6aRBONHkmr+TNeXRenHfnY95acPKZQ/IHzucPv5OSpw==</latexit><latexit sha1_base64="ue3WBNuo2mSgITFm5zybLczT74E=">AAAB9XicbVBNS8NAEN3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaWDbbSbt0swm7E7WE/g8vHhTx6n/x5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqDg0ey1i3A2ZACgUNFCihnWhgUSChFYyup37rAbQRsbrDcQJ+xAZKhIIztNJ9F+EJdZTpVNHzSa9ccavuDHSZeDmpkBz1Xvmr2495GoFCLpkxHc9N0M+YRsElTErd1EDC+IgNoGOpYhEYP5tdPaEnVunTMNa2FNKZ+nsiY5Ex4yiwnRHDoVn0puJ/XifF8NLPhEpSBMXni8JUUozpNALaFxo4yrEljGthb6V8yDTjaIMq2RC8xZeXSfOs6rlV7/a8UrvK4yiSI3JMTolHLkiN3JA6aRBONHkmr+TNeXRenHfnY95acPKZQ/IHzucPv5OSpw==</latexit><latexit sha1_base64="ue3WBNuo2mSgITFm5zybLczT74E=">AAAB9XicbVBNS8NAEN3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaWDbbSbt0swm7E7WE/g8vHhTx6n/x5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqDg0ey1i3A2ZACgUNFCihnWhgUSChFYyup37rAbQRsbrDcQJ+xAZKhIIztNJ9F+EJdZTpVNHzSa9ccavuDHSZeDmpkBz1Xvmr2495GoFCLpkxHc9N0M+YRsElTErd1EDC+IgNoGOpYhEYP5tdPaEnVunTMNa2FNKZ+nsiY5Ex4yiwnRHDoVn0puJ/XifF8NLPhEpSBMXni8JUUozpNALaFxo4yrEljGthb6V8yDTjaIMq2RC8xZeXSfOs6rlV7/a8UrvK4yiSI3JMTolHLkiN3JA6aRBONHkmr+TNeXRenHfnY95acPKZQ/IHzucPv5OSpw==</latexit><latexit sha1_base64="ue3WBNuo2mSgITFm5zybLczT74E=">AAAB9XicbVBNS8NAEN3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaWDbbSbt0swm7E7WE/g8vHhTx6n/x5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqDg0ey1i3A2ZACgUNFCihnWhgUSChFYyup37rAbQRsbrDcQJ+xAZKhIIztNJ9F+EJdZTpVNHzSa9ccavuDHSZeDmpkBz1Xvmr2495GoFCLpkxHc9N0M+YRsElTErd1EDC+IgNoGOpYhEYP5tdPaEnVunTMNa2FNKZ+nsiY5Ex4yiwnRHDoVn0puJ/XifF8NLPhEpSBMXni8JUUozpNALaFxo4yrEljGthb6V8yDTjaIMq2RC8xZeXSfOs6rlV7/a8UrvK4yiSI3JMTolHLkiN3JA6aRBONHkmr+TNeXRenHfnY95acPKZQ/IHzucPv5OSpw==</latexit>
run 5 <latexit sha1_base64="9lCxLojEYNROqd5ZW2zY5IE7ONI=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4Kokoeix68VjBfkAby2Y7aZduNmF3opbQ/+HFgyJe/S/e/Ddu2xy09cHA470ZZuYFiRQGXffbWVpeWV1bL2wUN7e2d3ZLe/sNE6eaQ53HMtatgBmQQkEdBUpoJRpYFEhoBsPrid98AG1ErO5wlIAfsb4SoeAMrXTfQXhCHWU6VfR83C2V3Yo7BV0kXk7KJEetW/rq9GKeRqCQS2ZM23MT9DOmUXAJ42InNZAwPmR9aFuqWATGz6ZXj+mxVXo0jLUthXSq/p7IWGTMKApsZ8RwYOa9ifif104xvPQzoZIUQfHZojCVFGM6iYD2hAaOcmQJ41rYWykfMM042qCKNgRv/uVF0jiteG7Fuz0rV6/yOArkkByRE+KRC1IlN6RG6oQTTZ7JK3lzHp0X5935mLUuOfnMAfkD5/MHwRiSqA==</latexit><latexit sha1_base64="9lCxLojEYNROqd5ZW2zY5IE7ONI=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4Kokoeix68VjBfkAby2Y7aZduNmF3opbQ/+HFgyJe/S/e/Ddu2xy09cHA470ZZuYFiRQGXffbWVpeWV1bL2wUN7e2d3ZLe/sNE6eaQ53HMtatgBmQQkEdBUpoJRpYFEhoBsPrid98AG1ErO5wlIAfsb4SoeAMrXTfQXhCHWU6VfR83C2V3Yo7BV0kXk7KJEetW/rq9GKeRqCQS2ZM23MT9DOmUXAJ42InNZAwPmR9aFuqWATGz6ZXj+mxVXo0jLUthXSq/p7IWGTMKApsZ8RwYOa9ifif104xvPQzoZIUQfHZojCVFGM6iYD2hAaOcmQJ41rYWykfMM042qCKNgRv/uVF0jiteG7Fuz0rV6/yOArkkByRE+KRC1IlN6RG6oQTTZ7JK3lzHp0X5935mLUuOfnMAfkD5/MHwRiSqA==</latexit><latexit sha1_base64="9lCxLojEYNROqd5ZW2zY5IE7ONI=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4Kokoeix68VjBfkAby2Y7aZduNmF3opbQ/+HFgyJe/S/e/Ddu2xy09cHA470ZZuYFiRQGXffbWVpeWV1bL2wUN7e2d3ZLe/sNE6eaQ53HMtatgBmQQkEdBUpoJRpYFEhoBsPrid98AG1ErO5wlIAfsb4SoeAMrXTfQXhCHWU6VfR83C2V3Yo7BV0kXk7KJEetW/rq9GKeRqCQS2ZM23MT9DOmUXAJ42InNZAwPmR9aFuqWATGz6ZXj+mxVXo0jLUthXSq/p7IWGTMKApsZ8RwYOa9ifif104xvPQzoZIUQfHZojCVFGM6iYD2hAaOcmQJ41rYWykfMM042qCKNgRv/uVF0jiteG7Fuz0rV6/yOArkkByRE+KRC1IlN6RG6oQTTZ7JK3lzHp0X5935mLUuOfnMAfkD5/MHwRiSqA==</latexit><latexit sha1_base64="9lCxLojEYNROqd5ZW2zY5IE7ONI=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4Kokoeix68VjBfkAby2Y7aZduNmF3opbQ/+HFgyJe/S/e/Ddu2xy09cHA470ZZuYFiRQGXffbWVpeWV1bL2wUN7e2d3ZLe/sNE6eaQ53HMtatgBmQQkEdBUpoJRpYFEhoBsPrid98AG1ErO5wlIAfsb4SoeAMrXTfQXhCHWU6VfR83C2V3Yo7BV0kXk7KJEetW/rq9GKeRqCQS2ZM23MT9DOmUXAJ42InNZAwPmR9aFuqWATGz6ZXj+mxVXo0jLUthXSq/p7IWGTMKApsZ8RwYOa9ifif104xvPQzoZIUQfHZojCVFGM6iYD2hAaOcmQJ41rYWykfMM042qCKNgRv/uVF0jiteG7Fuz0rV6/yOArkkByRE+KRC1IlN6RG6oQTTZ7JK3lzHp0X5935mLUuOfnMAfkD5/MHwRiSqA==</latexit>
run 6 <latexit sha1_base64="z+40nnry1acL6Q+gUVhFrRezuLk=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KomIeix68VjBfkAby2Y7aZduNmF3opbQ/+HFgyJe/S/e/Ddu2xy09cHA470ZZuYFiRQGXffbWVpeWV1bL2wUN7e2d3ZLe/sNE6eaQ53HMtatgBmQQkEdBUpoJRpYFEhoBsPrid98AG1ErO5wlIAfsb4SoeAMrXTfQXhCHWU6VfR83C2V3Yo7BV0kXk7KJEetW/rq9GKeRqCQS2ZM23MT9DOmUXAJ42InNZAwPmR9aFuqWATGz6ZXj+mxVXo0jLUthXSq/p7IWGTMKApsZ8RwYOa9ifif104xvPQzoZIUQfHZojCVFGM6iYD2hAaOcmQJ41rYWykfMM042qCKNgRv/uVF0jiteG7Fuz0rV6/yOArkkByRE+KRC1IlN6RG6oQTTZ7JK3lzHp0X5935mLUuOfnMAfkD5/MHwp2SqQ==</latexit><latexit sha1_base64="z+40nnry1acL6Q+gUVhFrRezuLk=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KomIeix68VjBfkAby2Y7aZduNmF3opbQ/+HFgyJe/S/e/Ddu2xy09cHA470ZZuYFiRQGXffbWVpeWV1bL2wUN7e2d3ZLe/sNE6eaQ53HMtatgBmQQkEdBUpoJRpYFEhoBsPrid98AG1ErO5wlIAfsb4SoeAMrXTfQXhCHWU6VfR83C2V3Yo7BV0kXk7KJEetW/rq9GKeRqCQS2ZM23MT9DOmUXAJ42InNZAwPmR9aFuqWATGz6ZXj+mxVXo0jLUthXSq/p7IWGTMKApsZ8RwYOa9ifif104xvPQzoZIUQfHZojCVFGM6iYD2hAaOcmQJ41rYWykfMM042qCKNgRv/uVF0jiteG7Fuz0rV6/yOArkkByRE+KRC1IlN6RG6oQTTZ7JK3lzHp0X5935mLUuOfnMAfkD5/MHwp2SqQ==</latexit><latexit sha1_base64="z+40nnry1acL6Q+gUVhFrRezuLk=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KomIeix68VjBfkAby2Y7aZduNmF3opbQ/+HFgyJe/S/e/Ddu2xy09cHA470ZZuYFiRQGXffbWVpeWV1bL2wUN7e2d3ZLe/sNE6eaQ53HMtatgBmQQkEdBUpoJRpYFEhoBsPrid98AG1ErO5wlIAfsb4SoeAMrXTfQXhCHWU6VfR83C2V3Yo7BV0kXk7KJEetW/rq9GKeRqCQS2ZM23MT9DOmUXAJ42InNZAwPmR9aFuqWATGz6ZXj+mxVXo0jLUthXSq/p7IWGTMKApsZ8RwYOa9ifif104xvPQzoZIUQfHZojCVFGM6iYD2hAaOcmQJ41rYWykfMM042qCKNgRv/uVF0jiteG7Fuz0rV6/yOArkkByRE+KRC1IlN6RG6oQTTZ7JK3lzHp0X5935mLUuOfnMAfkD5/MHwp2SqQ==</latexit><latexit sha1_base64="z+40nnry1acL6Q+gUVhFrRezuLk=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KomIeix68VjBfkAby2Y7aZduNmF3opbQ/+HFgyJe/S/e/Ddu2xy09cHA470ZZuYFiRQGXffbWVpeWV1bL2wUN7e2d3ZLe/sNE6eaQ53HMtatgBmQQkEdBUpoJRpYFEhoBsPrid98AG1ErO5wlIAfsb4SoeAMrXTfQXhCHWU6VfR83C2V3Yo7BV0kXk7KJEetW/rq9GKeRqCQS2ZM23MT9DOmUXAJ42InNZAwPmR9aFuqWATGz6ZXj+mxVXo0jLUthXSq/p7IWGTMKApsZ8RwYOa9ifif104xvPQzoZIUQfHZojCVFGM6iYD2hAaOcmQJ41rYWykfMM042qCKNgRv/uVF0jiteG7Fuz0rV6/yOArkkByRE+KRC1IlN6RG6oQTTZ7JK3lzHp0X5935mLUuOfnMAfkD5/MHwp2SqQ==</latexit>
run 7 <latexit sha1_base64="EfNQARihNovNhGzbe22A/IikzaI=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KokI9Vj04rGC/YA2ls120i7dbMLuRC2h/8OLB0W8+l+8+W/ctjlo64OBx3szzMwLEikMuu63s7K6tr6xWdgqbu/s7u2XDg6bJk41hwaPZazbATMghYIGCpTQTjSwKJDQCkbXU7/1ANqIWN3hOAE/YgMlQsEZWum+i/CEOsp0qmh10iuV3Yo7A10mXk7KJEe9V/rq9mOeRqCQS2ZMx3MT9DOmUXAJk2I3NZAwPmID6FiqWATGz2ZXT+ipVfo0jLUthXSm/p7IWGTMOApsZ8RwaBa9qfif10kxvPQzoZIUQfH5ojCVFGM6jYD2hQaOcmwJ41rYWykfMs042qCKNgRv8eVl0jyveG7Fu70o167yOArkmJyQM+KRKqmRG1InDcKJJs/klbw5j86L8+58zFtXnHzmiPyB8/kDxCKSqg==</latexit><latexit sha1_base64="EfNQARihNovNhGzbe22A/IikzaI=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KokI9Vj04rGC/YA2ls120i7dbMLuRC2h/8OLB0W8+l+8+W/ctjlo64OBx3szzMwLEikMuu63s7K6tr6xWdgqbu/s7u2XDg6bJk41hwaPZazbATMghYIGCpTQTjSwKJDQCkbXU7/1ANqIWN3hOAE/YgMlQsEZWum+i/CEOsp0qmh10iuV3Yo7A10mXk7KJEe9V/rq9mOeRqCQS2ZMx3MT9DOmUXAJk2I3NZAwPmID6FiqWATGz2ZXT+ipVfo0jLUthXSm/p7IWGTMOApsZ8RwaBa9qfif10kxvPQzoZIUQfH5ojCVFGM6jYD2hQaOcmwJ41rYWykfMs042qCKNgRv8eVl0jyveG7Fu70o167yOArkmJyQM+KRKqmRG1InDcKJJs/klbw5j86L8+58zFtXnHzmiPyB8/kDxCKSqg==</latexit><latexit sha1_base64="EfNQARihNovNhGzbe22A/IikzaI=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KokI9Vj04rGC/YA2ls120i7dbMLuRC2h/8OLB0W8+l+8+W/ctjlo64OBx3szzMwLEikMuu63s7K6tr6xWdgqbu/s7u2XDg6bJk41hwaPZazbATMghYIGCpTQTjSwKJDQCkbXU7/1ANqIWN3hOAE/YgMlQsEZWum+i/CEOsp0qmh10iuV3Yo7A10mXk7KJEe9V/rq9mOeRqCQS2ZMx3MT9DOmUXAJk2I3NZAwPmID6FiqWATGz2ZXT+ipVfo0jLUthXSm/p7IWGTMOApsZ8RwaBa9qfif10kxvPQzoZIUQfH5ojCVFGM6jYD2hQaOcmwJ41rYWykfMs042qCKNgRv8eVl0jyveG7Fu70o167yOArkmJyQM+KRKqmRG1InDcKJJs/klbw5j86L8+58zFtXnHzmiPyB8/kDxCKSqg==</latexit><latexit sha1_base64="EfNQARihNovNhGzbe22A/IikzaI=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KokI9Vj04rGC/YA2ls120i7dbMLuRC2h/8OLB0W8+l+8+W/ctjlo64OBx3szzMwLEikMuu63s7K6tr6xWdgqbu/s7u2XDg6bJk41hwaPZazbATMghYIGCpTQTjSwKJDQCkbXU7/1ANqIWN3hOAE/YgMlQsEZWum+i/CEOsp0qmh10iuV3Yo7A10mXk7KJEe9V/rq9mOeRqCQS2ZMx3MT9DOmUXAJk2I3NZAwPmID6FiqWATGz2ZXT+ipVfo0jLUthXSm/p7IWGTMOApsZ8RwaBa9qfif10kxvPQzoZIUQfH5ojCVFGM6jYD2hQaOcmwJ41rYWykfMs042qCKNgRv8eVl0jyveG7Fu70o167yOArkmJyQM+KRKqmRG1InDcKJJs/klbw5j86L8+58zFtXnHzmiPyB8/kDxCKSqg==</latexit>
run 8 <latexit sha1_base64="JHfo/0HAVB9gKRMN0ZwZmDCOIBE=">AAAB9XicbVBNS8NAEJ34WetX1aOXxSJ4KokI9lj04rGC/YA2ls120y7dbMLuRC2h/8OLB0W8+l+8+W/ctjlo64OBx3szzMwLEikMuu63s7K6tr6xWdgqbu/s7u2XDg6bJk414w0Wy1i3A2q4FIo3UKDk7URzGgWSt4LR9dRvPXBtRKzucJxwP6IDJULBKFrpvov8CXWU6VSR6qRXKrsVdwayTLyclCFHvVf66vZjlkZcIZPUmI7nJuhnVKNgkk+K3dTwhLIRHfCOpYpG3PjZ7OoJObVKn4SxtqWQzNTfExmNjBlHge2MKA7NojcV//M6KYZVPxMqSZErNl8UppJgTKYRkL7QnKEcW0KZFvZWwoZUU4Y2qKINwVt8eZk0zyueW/FuL8q1qzyOAhzDCZyBB5dQgxuoQwMYaHiGV3hzHp0X5935mLeuOPnMEfyB8/kDxaeSqw==</latexit><latexit sha1_base64="JHfo/0HAVB9gKRMN0ZwZmDCOIBE=">AAAB9XicbVBNS8NAEJ34WetX1aOXxSJ4KokI9lj04rGC/YA2ls120y7dbMLuRC2h/8OLB0W8+l+8+W/ctjlo64OBx3szzMwLEikMuu63s7K6tr6xWdgqbu/s7u2XDg6bJk414w0Wy1i3A2q4FIo3UKDk7URzGgWSt4LR9dRvPXBtRKzucJxwP6IDJULBKFrpvov8CXWU6VSR6qRXKrsVdwayTLyclCFHvVf66vZjlkZcIZPUmI7nJuhnVKNgkk+K3dTwhLIRHfCOpYpG3PjZ7OoJObVKn4SxtqWQzNTfExmNjBlHge2MKA7NojcV//M6KYZVPxMqSZErNl8UppJgTKYRkL7QnKEcW0KZFvZWwoZUU4Y2qKINwVt8eZk0zyueW/FuL8q1qzyOAhzDCZyBB5dQgxuoQwMYaHiGV3hzHp0X5935mLeuOPnMEfyB8/kDxaeSqw==</latexit><latexit sha1_base64="JHfo/0HAVB9gKRMN0ZwZmDCOIBE=">AAAB9XicbVBNS8NAEJ34WetX1aOXxSJ4KokI9lj04rGC/YA2ls120y7dbMLuRC2h/8OLB0W8+l+8+W/ctjlo64OBx3szzMwLEikMuu63s7K6tr6xWdgqbu/s7u2XDg6bJk414w0Wy1i3A2q4FIo3UKDk7URzGgWSt4LR9dRvPXBtRKzucJxwP6IDJULBKFrpvov8CXWU6VSR6qRXKrsVdwayTLyclCFHvVf66vZjlkZcIZPUmI7nJuhnVKNgkk+K3dTwhLIRHfCOpYpG3PjZ7OoJObVKn4SxtqWQzNTfExmNjBlHge2MKA7NojcV//M6KYZVPxMqSZErNl8UppJgTKYRkL7QnKEcW0KZFvZWwoZUU4Y2qKINwVt8eZk0zyueW/FuL8q1qzyOAhzDCZyBB5dQgxuoQwMYaHiGV3hzHp0X5935mLeuOPnMEfyB8/kDxaeSqw==</latexit><latexit sha1_base64="JHfo/0HAVB9gKRMN0ZwZmDCOIBE=">AAAB9XicbVBNS8NAEJ34WetX1aOXxSJ4KokI9lj04rGC/YA2ls120y7dbMLuRC2h/8OLB0W8+l+8+W/ctjlo64OBx3szzMwLEikMuu63s7K6tr6xWdgqbu/s7u2XDg6bJk414w0Wy1i3A2q4FIo3UKDk7URzGgWSt4LR9dRvPXBtRKzucJxwP6IDJULBKFrpvov8CXWU6VSR6qRXKrsVdwayTLyclCFHvVf66vZjlkZcIZPUmI7nJuhnVKNgkk+K3dTwhLIRHfCOpYpG3PjZ7OoJObVKn4SxtqWQzNTfExmNjBlHge2MKA7NojcV//M6KYZVPxMqSZErNl8UppJgTKYRkL7QnKEcW0KZFvZWwoZUU4Y2qKINwVt8eZk0zyueW/FuL8q1qzyOAhzDCZyBB5dQgxuoQwMYaHiGV3hzHp0X5935mLeuOPnMEfyB8/kDxaeSqw==</latexit>
run 9 <latexit sha1_base64="ZKiO5wNI3nePcwucVRaceCXQs9A=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KokI6q3oxWMF+wFtLJvtpF262YTdiVpC/4cXD4p49b9489+4bXPQ1gcDj/dmmJkXJFIYdN1vZ2l5ZXVtvbBR3Nza3tkt7e03TJxqDnUey1i3AmZACgV1FCihlWhgUSChGQyvJ37zAbQRsbrDUQJ+xPpKhIIztNJ9B+EJdZTpVNHLcbdUdivuFHSReDkpkxy1bumr04t5GoFCLpkxbc9N0M+YRsEljIud1EDC+JD1oW2pYhEYP5tePabHVunRMNa2FNKp+nsiY5ExoyiwnRHDgZn3JuJ/XjvF8MLPhEpSBMVni8JUUozpJALaExo4ypEljGthb6V8wDTjaIMq2hC8+ZcXSeO04rkV7/asXL3K4yiQQ3JETohHzkmV3JAaqRNONHkmr+TNeXRenHfnY9a65OQzB+QPnM8fxyySrA==</latexit><latexit sha1_base64="ZKiO5wNI3nePcwucVRaceCXQs9A=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KokI6q3oxWMF+wFtLJvtpF262YTdiVpC/4cXD4p49b9489+4bXPQ1gcDj/dmmJkXJFIYdN1vZ2l5ZXVtvbBR3Nza3tkt7e03TJxqDnUey1i3AmZACgV1FCihlWhgUSChGQyvJ37zAbQRsbrDUQJ+xPpKhIIztNJ9B+EJdZTpVNHLcbdUdivuFHSReDkpkxy1bumr04t5GoFCLpkxbc9N0M+YRsEljIud1EDC+JD1oW2pYhEYP5tePabHVunRMNa2FNKp+nsiY5ExoyiwnRHDgZn3JuJ/XjvF8MLPhEpSBMVni8JUUozpJALaExo4ypEljGthb6V8wDTjaIMq2hC8+ZcXSeO04rkV7/asXL3K4yiQQ3JETohHzkmV3JAaqRNONHkmr+TNeXRenHfnY9a65OQzB+QPnM8fxyySrA==</latexit><latexit sha1_base64="ZKiO5wNI3nePcwucVRaceCXQs9A=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KokI6q3oxWMF+wFtLJvtpF262YTdiVpC/4cXD4p49b9489+4bXPQ1gcDj/dmmJkXJFIYdN1vZ2l5ZXVtvbBR3Nza3tkt7e03TJxqDnUey1i3AmZACgV1FCihlWhgUSChGQyvJ37zAbQRsbrDUQJ+xPpKhIIztNJ9B+EJdZTpVNHLcbdUdivuFHSReDkpkxy1bumr04t5GoFCLpkxbc9N0M+YRsEljIud1EDC+JD1oW2pYhEYP5tePabHVunRMNa2FNKp+nsiY5ExoyiwnRHDgZn3JuJ/XjvF8MLPhEpSBMVni8JUUozpJALaExo4ypEljGthb6V8wDTjaIMq2hC8+ZcXSeO04rkV7/asXL3K4yiQQ3JETohHzkmV3JAaqRNONHkmr+TNeXRenHfnY9a65OQzB+QPnM8fxyySrA==</latexit><latexit sha1_base64="ZKiO5wNI3nePcwucVRaceCXQs9A=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KokI6q3oxWMF+wFtLJvtpF262YTdiVpC/4cXD4p49b9489+4bXPQ1gcDj/dmmJkXJFIYdN1vZ2l5ZXVtvbBR3Nza3tkt7e03TJxqDnUey1i3AmZACgV1FCihlWhgUSChGQyvJ37zAbQRsbrDUQJ+xPpKhIIztNJ9B+EJdZTpVNHLcbdUdivuFHSReDkpkxy1bumr04t5GoFCLpkxbc9N0M+YRsEljIud1EDC+JD1oW2pYhEYP5tePabHVunRMNa2FNKp+nsiY5ExoyiwnRHDgZn3JuJ/XjvF8MLPhEpSBMVni8JUUozpJALaExo4ypEljGthb6V8wDTjaIMq2hC8+ZcXSeO04rkV7/asXL3K4yiQQ3JETohHzkmV3JAaqRNONHkmr+TNeXRenHfnY9a65OQzB+QPnM8fxyySrA==</latexit>
run 10 <latexit sha1_base64="rMZt6lLsG4QW0FzbkSd9wwyTKHE=">AAAB+HicbVBNSwMxEM3Wr1o/uurRS7AInsquCHosevFYwX5Au5RsmrahSXZJJmJd+ku8eFDEqz/Fm//GtN2Dtj4YeLw3w8y8OBXcQBB8e4W19Y3NreJ2aWd3b7/sHxw2TWI1ZQ2aiES3Y2KY4Io1gINg7VQzImPBWvH4Zua3Hpg2PFH3MElZJMlQ8QGnBJzU88tdYI+gZaatwmEw7fmVoBrMgVdJmJMKylHv+V/dfkKtZAqoIMZ0wiCFKCMaOBVsWupaw1JCx2TIOo4qIpmJsvnhU3zqlD4eJNqVAjxXf09kRBozkbHrlARGZtmbif95HQuDqyjjKrXAFF0sGliBIcGzFHCfa0ZBTBwhVHN3K6YjogkFl1XJhRAuv7xKmufVMKiGdxeV2nUeRxEdoxN0hkJ0iWroFtVRA1Fk0TN6RW/ek/fivXsfi9aCl88coT/wPn8ApdeTDw==</latexit><latexit sha1_base64="rMZt6lLsG4QW0FzbkSd9wwyTKHE=">AAAB+HicbVBNSwMxEM3Wr1o/uurRS7AInsquCHosevFYwX5Au5RsmrahSXZJJmJd+ku8eFDEqz/Fm//GtN2Dtj4YeLw3w8y8OBXcQBB8e4W19Y3NreJ2aWd3b7/sHxw2TWI1ZQ2aiES3Y2KY4Io1gINg7VQzImPBWvH4Zua3Hpg2PFH3MElZJMlQ8QGnBJzU88tdYI+gZaatwmEw7fmVoBrMgVdJmJMKylHv+V/dfkKtZAqoIMZ0wiCFKCMaOBVsWupaw1JCx2TIOo4qIpmJsvnhU3zqlD4eJNqVAjxXf09kRBozkbHrlARGZtmbif95HQuDqyjjKrXAFF0sGliBIcGzFHCfa0ZBTBwhVHN3K6YjogkFl1XJhRAuv7xKmufVMKiGdxeV2nUeRxEdoxN0hkJ0iWroFtVRA1Fk0TN6RW/ek/fivXsfi9aCl88coT/wPn8ApdeTDw==</latexit><latexit sha1_base64="rMZt6lLsG4QW0FzbkSd9wwyTKHE=">AAAB+HicbVBNSwMxEM3Wr1o/uurRS7AInsquCHosevFYwX5Au5RsmrahSXZJJmJd+ku8eFDEqz/Fm//GtN2Dtj4YeLw3w8y8OBXcQBB8e4W19Y3NreJ2aWd3b7/sHxw2TWI1ZQ2aiES3Y2KY4Io1gINg7VQzImPBWvH4Zua3Hpg2PFH3MElZJMlQ8QGnBJzU88tdYI+gZaatwmEw7fmVoBrMgVdJmJMKylHv+V/dfkKtZAqoIMZ0wiCFKCMaOBVsWupaw1JCx2TIOo4qIpmJsvnhU3zqlD4eJNqVAjxXf09kRBozkbHrlARGZtmbif95HQuDqyjjKrXAFF0sGliBIcGzFHCfa0ZBTBwhVHN3K6YjogkFl1XJhRAuv7xKmufVMKiGdxeV2nUeRxEdoxN0hkJ0iWroFtVRA1Fk0TN6RW/ek/fivXsfi9aCl88coT/wPn8ApdeTDw==</latexit><latexit sha1_base64="rMZt6lLsG4QW0FzbkSd9wwyTKHE=">AAAB+HicbVBNSwMxEM3Wr1o/uurRS7AInsquCHosevFYwX5Au5RsmrahSXZJJmJd+ku8eFDEqz/Fm//GtN2Dtj4YeLw3w8y8OBXcQBB8e4W19Y3NreJ2aWd3b7/sHxw2TWI1ZQ2aiES3Y2KY4Io1gINg7VQzImPBWvH4Zua3Hpg2PFH3MElZJMlQ8QGnBJzU88tdYI+gZaatwmEw7fmVoBrMgVdJmJMKylHv+V/dfkKtZAqoIMZ0wiCFKCMaOBVsWupaw1JCx2TIOo4qIpmJsvnhU3zqlD4eJNqVAjxXf09kRBozkbHrlARGZtmbif95HQuDqyjjKrXAFF0sGliBIcGzFHCfa0ZBTBwhVHN3K6YjogkFl1XJhRAuv7xKmufVMKiGdxeV2nUeRxEdoxN0hkJ0iWroFtVRA1Fk0TN6RW/ek/fivXsfi9aCl88coT/wPn8ApdeTDw==</latexit>
|a| <latexit sha1_base64="p4NdPauRcwGeLSvEXoY+yd5ipTI=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8eK9gPaUDbbSbt0swm7G6Gk/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU1HGqGDZYLGLVDqhGwSU2DDcC24lCGgUCW8Hodua3nlBpHstHM07Qj+hA8pAzaqz0MKGTXrniVt05yCrxclKBHPVe+avbj1kaoTRMUK07npsYP6PKcCZwWuqmGhPKRnSAHUsljVD72fzUKTmzSp+EsbIlDZmrvycyGmk9jgLbGVEz1MveTPzP66QmvPYzLpPUoGSLRWEqiInJ7G/S5wqZEWNLKFPc3krYkCrKjE2nZEPwll9eJc2LqudWvfvLSu0mj6MIJ3AK5+DBFdTgDurQAAYDeIZXeHOE8+K8Ox+L1oKTzxzDHzifP4Z+jfE=</latexit><latexit sha1_base64="p4NdPauRcwGeLSvEXoY+yd5ipTI=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8eK9gPaUDbbSbt0swm7G6Gk/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU1HGqGDZYLGLVDqhGwSU2DDcC24lCGgUCW8Hodua3nlBpHstHM07Qj+hA8pAzaqz0MKGTXrniVt05yCrxclKBHPVe+avbj1kaoTRMUK07npsYP6PKcCZwWuqmGhPKRnSAHUsljVD72fzUKTmzSp+EsbIlDZmrvycyGmk9jgLbGVEz1MveTPzP66QmvPYzLpPUoGSLRWEqiInJ7G/S5wqZEWNLKFPc3krYkCrKjE2nZEPwll9eJc2LqudWvfvLSu0mj6MIJ3AK5+DBFdTgDurQAAYDeIZXeHOE8+K8Ox+L1oKTzxzDHzifP4Z+jfE=</latexit><latexit sha1_base64="p4NdPauRcwGeLSvEXoY+yd5ipTI=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8eK9gPaUDbbSbt0swm7G6Gk/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU1HGqGDZYLGLVDqhGwSU2DDcC24lCGgUCW8Hodua3nlBpHstHM07Qj+hA8pAzaqz0MKGTXrniVt05yCrxclKBHPVe+avbj1kaoTRMUK07npsYP6PKcCZwWuqmGhPKRnSAHUsljVD72fzUKTmzSp+EsbIlDZmrvycyGmk9jgLbGVEz1MveTPzP66QmvPYzLpPUoGSLRWEqiInJ7G/S5wqZEWNLKFPc3krYkCrKjE2nZEPwll9eJc2LqudWvfvLSu0mj6MIJ3AK5+DBFdTgDurQAAYDeIZXeHOE8+K8Ox+L1oKTzxzDHzifP4Z+jfE=</latexit><latexit sha1_base64="p4NdPauRcwGeLSvEXoY+yd5ipTI=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8eK9gPaUDbbSbt0swm7G6Gk/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU1HGqGDZYLGLVDqhGwSU2DDcC24lCGgUCW8Hodua3nlBpHstHM07Qj+hA8pAzaqz0MKGTXrniVt05yCrxclKBHPVe+avbj1kaoTRMUK07npsYP6PKcCZwWuqmGhPKRnSAHUsljVD72fzUKTmzSp+EsbIlDZmrvycyGmk9jgLbGVEz1MveTPzP66QmvPYzLpPUoGSLRWEqiInJ7G/S5wqZEWNLKFPc3krYkCrKjE2nZEPwll9eJc2LqudWvfvLSu0mj6MIJ3AK5+DBFdTgDurQAAYDeIZXeHOE8+K8Ox+L1oKTzxzDHzifP4Z+jfE=</latexit>
|a| <latexit sha1_base64="p4NdPauRcwGeLSvEXoY+yd5ipTI=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8eK9gPaUDbbSbt0swm7G6Gk/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU1HGqGDZYLGLVDqhGwSU2DDcC24lCGgUCW8Hodua3nlBpHstHM07Qj+hA8pAzaqz0MKGTXrniVt05yCrxclKBHPVe+avbj1kaoTRMUK07npsYP6PKcCZwWuqmGhPKRnSAHUsljVD72fzUKTmzSp+EsbIlDZmrvycyGmk9jgLbGVEz1MveTPzP66QmvPYzLpPUoGSLRWEqiInJ7G/S5wqZEWNLKFPc3krYkCrKjE2nZEPwll9eJc2LqudWvfvLSu0mj6MIJ3AK5+DBFdTgDurQAAYDeIZXeHOE8+K8Ox+L1oKTzxzDHzifP4Z+jfE=</latexit><latexit sha1_base64="p4NdPauRcwGeLSvEXoY+yd5ipTI=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8eK9gPaUDbbSbt0swm7G6Gk/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU1HGqGDZYLGLVDqhGwSU2DDcC24lCGgUCW8Hodua3nlBpHstHM07Qj+hA8pAzaqz0MKGTXrniVt05yCrxclKBHPVe+avbj1kaoTRMUK07npsYP6PKcCZwWuqmGhPKRnSAHUsljVD72fzUKTmzSp+EsbIlDZmrvycyGmk9jgLbGVEz1MveTPzP66QmvPYzLpPUoGSLRWEqiInJ7G/S5wqZEWNLKFPc3krYkCrKjE2nZEPwll9eJc2LqudWvfvLSu0mj6MIJ3AK5+DBFdTgDurQAAYDeIZXeHOE8+K8Ox+L1oKTzxzDHzifP4Z+jfE=</latexit><latexit sha1_base64="p4NdPauRcwGeLSvEXoY+yd5ipTI=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8eK9gPaUDbbSbt0swm7G6Gk/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU1HGqGDZYLGLVDqhGwSU2DDcC24lCGgUCW8Hodua3nlBpHstHM07Qj+hA8pAzaqz0MKGTXrniVt05yCrxclKBHPVe+avbj1kaoTRMUK07npsYP6PKcCZwWuqmGhPKRnSAHUsljVD72fzUKTmzSp+EsbIlDZmrvycyGmk9jgLbGVEz1MveTPzP66QmvPYzLpPUoGSLRWEqiInJ7G/S5wqZEWNLKFPc3krYkCrKjE2nZEPwll9eJc2LqudWvfvLSu0mj6MIJ3AK5+DBFdTgDurQAAYDeIZXeHOE8+K8Ox+L1oKTzxzDHzifP4Z+jfE=</latexit><latexit sha1_base64="p4NdPauRcwGeLSvEXoY+yd5ipTI=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8eK9gPaUDbbSbt0swm7G6Gk/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU1HGqGDZYLGLVDqhGwSU2DDcC24lCGgUCW8Hodua3nlBpHstHM07Qj+hA8pAzaqz0MKGTXrniVt05yCrxclKBHPVe+avbj1kaoTRMUK07npsYP6PKcCZwWuqmGhPKRnSAHUsljVD72fzUKTmzSp+EsbIlDZmrvycyGmk9jgLbGVEz1MveTPzP66QmvPYzLpPUoGSLRWEqiInJ7G/S5wqZEWNLKFPc3krYkCrKjE2nZEPwll9eJc2LqudWvfvLSu0mj6MIJ3AK5+DBFdTgDurQAAYDeIZXeHOE8+K8Ox+L1oKTzxzDHzifP4Z+jfE=</latexit>
Figure 14: We repeat Figure 6 for all 10 seeds of the both pnorm agent of Table 2. Note that the action distribution does not saturate.
run 1 <latexit sha1_base64="K2LHkVIbGBbhAACjOdf7sVnMn+4=">AAAB9XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaWDbbTbt0swm7E7WE/g8vHhTx6n/x5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqxhsslrFuB9RwKRRvoEDJ24nmNAokbwWj66nfeuDaiFjd4TjhfkQHSoSCUbTSfRf5E+oo06ki3qRXrrhVdwayTLycVCBHvVf+6vZjlkZcIZPUmI7nJuhnVKNgkk9K3dTwhLIRHfCOpYpG3PjZ7OoJObFKn4SxtqWQzNTfExmNjBlHge2MKA7NojcV//M6KYaXfiZUkiJXbL4oTCXBmEwjIH2hOUM5toQyLeythA2ppgxtUCUbgrf48jJpnlU9t+rdnldqV3kcRTiCYzgFDy6gBjdQhwYw0PAMr/DmPDovzrvzMW8tOPnMIfyB8/kDuwSSpA==</latexit><latexit sha1_base64="K2LHkVIbGBbhAACjOdf7sVnMn+4=">AAAB9XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaWDbbTbt0swm7E7WE/g8vHhTx6n/x5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqxhsslrFuB9RwKRRvoEDJ24nmNAokbwWj66nfeuDaiFjd4TjhfkQHSoSCUbTSfRf5E+oo06ki3qRXrrhVdwayTLycVCBHvVf+6vZjlkZcIZPUmI7nJuhnVKNgkk9K3dTwhLIRHfCOpYpG3PjZ7OoJObFKn4SxtqWQzNTfExmNjBlHge2MKA7NojcV//M6KYaXfiZUkiJXbL4oTCXBmEwjIH2hOUM5toQyLeythA2ppgxtUCUbgrf48jJpnlU9t+rdnldqV3kcRTiCYzgFDy6gBjdQhwYw0PAMr/DmPDovzrvzMW8tOPnMIfyB8/kDuwSSpA==</latexit><latexit sha1_base64="K2LHkVIbGBbhAACjOdf7sVnMn+4=">AAAB9XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaWDbbTbt0swm7E7WE/g8vHhTx6n/x5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqxhsslrFuB9RwKRRvoEDJ24nmNAokbwWj66nfeuDaiFjd4TjhfkQHSoSCUbTSfRf5E+oo06ki3qRXrrhVdwayTLycVCBHvVf+6vZjlkZcIZPUmI7nJuhnVKNgkk9K3dTwhLIRHfCOpYpG3PjZ7OoJObFKn4SxtqWQzNTfExmNjBlHge2MKA7NojcV//M6KYaXfiZUkiJXbL4oTCXBmEwjIH2hOUM5toQyLeythA2ppgxtUCUbgrf48jJpnlU9t+rdnldqV3kcRTiCYzgFDy6gBjdQhwYw0PAMr/DmPDovzrvzMW8tOPnMIfyB8/kDuwSSpA==</latexit><latexit sha1_base64="K2LHkVIbGBbhAACjOdf7sVnMn+4=">AAAB9XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaWDbbTbt0swm7E7WE/g8vHhTx6n/x5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqxhsslrFuB9RwKRRvoEDJ24nmNAokbwWj66nfeuDaiFjd4TjhfkQHSoSCUbTSfRf5E+oo06ki3qRXrrhVdwayTLycVCBHvVf+6vZjlkZcIZPUmI7nJuhnVKNgkk9K3dTwhLIRHfCOpYpG3PjZ7OoJObFKn4SxtqWQzNTfExmNjBlHge2MKA7NojcV//M6KYaXfiZUkiJXbL4oTCXBmEwjIH2hOUM5toQyLeythA2ppgxtUCUbgrf48jJpnlU9t+rdnldqV3kcRTiCYzgFDy6gBjdQhwYw0PAMr/DmPDovzrvzMW8tOPnMIfyB8/kDuwSSpA==</latexit>
run 2 <latexit sha1_base64="5Dgsnw1SpIG9RSwzoiyubX6nkEQ=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KkkR9Fj04rGC/YA2ls120i7dbMLuRC2h/8OLB0W8+l+8+W/ctjlo64OBx3szzMwLEikMuu63s7K6tr6xWdgqbu/s7u2XDg6bJk41hwaPZazbATMghYIGCpTQTjSwKJDQCkbXU7/1ANqIWN3hOAE/YgMlQsEZWum+i/CEOsp0qmh10iuV3Yo7A10mXk7KJEe9V/rq9mOeRqCQS2ZMx3MT9DOmUXAJk2I3NZAwPmID6FiqWATGz2ZXT+ipVfo0jLUthXSm/p7IWGTMOApsZ8RwaBa9qfif10kxvPQzoZIUQfH5ojCVFGM6jYD2hQaOcmwJ41rYWykfMs042qCKNgRv8eVl0qxWPLfi3Z6Xa1d5HAVyTE7IGfHIBamRG1InDcKJJs/klbw5j86L8+58zFtXnHzmiPyB8/kDvImSpQ==</latexit><latexit sha1_base64="5Dgsnw1SpIG9RSwzoiyubX6nkEQ=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KkkR9Fj04rGC/YA2ls120i7dbMLuRC2h/8OLB0W8+l+8+W/ctjlo64OBx3szzMwLEikMuu63s7K6tr6xWdgqbu/s7u2XDg6bJk41hwaPZazbATMghYIGCpTQTjSwKJDQCkbXU7/1ANqIWN3hOAE/YgMlQsEZWum+i/CEOsp0qmh10iuV3Yo7A10mXk7KJEe9V/rq9mOeRqCQS2ZMx3MT9DOmUXAJk2I3NZAwPmID6FiqWATGz2ZXT+ipVfo0jLUthXSm/p7IWGTMOApsZ8RwaBa9qfif10kxvPQzoZIUQfH5ojCVFGM6jYD2hQaOcmwJ41rYWykfMs042qCKNgRv8eVl0qxWPLfi3Z6Xa1d5HAVyTE7IGfHIBamRG1InDcKJJs/klbw5j86L8+58zFtXnHzmiPyB8/kDvImSpQ==</latexit><latexit sha1_base64="5Dgsnw1SpIG9RSwzoiyubX6nkEQ=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KkkR9Fj04rGC/YA2ls120i7dbMLuRC2h/8OLB0W8+l+8+W/ctjlo64OBx3szzMwLEikMuu63s7K6tr6xWdgqbu/s7u2XDg6bJk41hwaPZazbATMghYIGCpTQTjSwKJDQCkbXU7/1ANqIWN3hOAE/YgMlQsEZWum+i/CEOsp0qmh10iuV3Yo7A10mXk7KJEe9V/rq9mOeRqCQS2ZMx3MT9DOmUXAJk2I3NZAwPmID6FiqWATGz2ZXT+ipVfo0jLUthXSm/p7IWGTMOApsZ8RwaBa9qfif10kxvPQzoZIUQfH5ojCVFGM6jYD2hQaOcmwJ41rYWykfMs042qCKNgRv8eVl0qxWPLfi3Z6Xa1d5HAVyTE7IGfHIBamRG1InDcKJJs/klbw5j86L8+58zFtXnHzmiPyB8/kDvImSpQ==</latexit><latexit sha1_base64="5Dgsnw1SpIG9RSwzoiyubX6nkEQ=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KkkR9Fj04rGC/YA2ls120i7dbMLuRC2h/8OLB0W8+l+8+W/ctjlo64OBx3szzMwLEikMuu63s7K6tr6xWdgqbu/s7u2XDg6bJk41hwaPZazbATMghYIGCpTQTjSwKJDQCkbXU7/1ANqIWN3hOAE/YgMlQsEZWum+i/CEOsp0qmh10iuV3Yo7A10mXk7KJEe9V/rq9mOeRqCQS2ZMx3MT9DOmUXAJk2I3NZAwPmID6FiqWATGz2ZXT+ipVfo0jLUthXSm/p7IWGTMOApsZ8RwaBa9qfif10kxvPQzoZIUQfH5ojCVFGM6jYD2hQaOcmwJ41rYWykfMs042qCKNgRv8eVl0qxWPLfi3Z6Xa1d5HAVyTE7IGfHIBamRG1InDcKJJs/klbw5j86L8+58zFtXnHzmiPyB8/kDvImSpQ==</latexit>
run 3 <latexit sha1_base64="ozX/0t3JPmydgwQzsf6ljZW+R6o=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KokKeix68VjBfkAby2Y7aZduNmF3opbQ/+HFgyJe/S/e/Ddu2xy09cHA470ZZuYFiRQGXffbWVpeWV1bL2wUN7e2d3ZLe/sNE6eaQ53HMtatgBmQQkEdBUpoJRpYFEhoBsPrid98AG1ErO5wlIAfsb4SoeAMrXTfQXhCHWU6VfRs3C2V3Yo7BV0kXk7KJEetW/rq9GKeRqCQS2ZM23MT9DOmUXAJ42InNZAwPmR9aFuqWATGz6ZXj+mxVXo0jLUthXSq/p7IWGTMKApsZ8RwYOa9ifif104xvPQzoZIUQfHZojCVFGM6iYD2hAaOcmQJ41rYWykfMM042qCKNgRv/uVF0jiteG7Fuz0vV6/yOArkkByRE+KRC1IlN6RG6oQTTZ7JK3lzHp0X5935mLUuOfnMAfkD5/MHvg6Spg==</latexit><latexit sha1_base64="ozX/0t3JPmydgwQzsf6ljZW+R6o=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KokKeix68VjBfkAby2Y7aZduNmF3opbQ/+HFgyJe/S/e/Ddu2xy09cHA470ZZuYFiRQGXffbWVpeWV1bL2wUN7e2d3ZLe/sNE6eaQ53HMtatgBmQQkEdBUpoJRpYFEhoBsPrid98AG1ErO5wlIAfsb4SoeAMrXTfQXhCHWU6VfRs3C2V3Yo7BV0kXk7KJEetW/rq9GKeRqCQS2ZM23MT9DOmUXAJ42InNZAwPmR9aFuqWATGz6ZXj+mxVXo0jLUthXSq/p7IWGTMKApsZ8RwYOa9ifif104xvPQzoZIUQfHZojCVFGM6iYD2hAaOcmQJ41rYWykfMM042qCKNgRv/uVF0jiteG7Fuz0vV6/yOArkkByRE+KRC1IlN6RG6oQTTZ7JK3lzHp0X5935mLUuOfnMAfkD5/MHvg6Spg==</latexit><latexit sha1_base64="ozX/0t3JPmydgwQzsf6ljZW+R6o=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KokKeix68VjBfkAby2Y7aZduNmF3opbQ/+HFgyJe/S/e/Ddu2xy09cHA470ZZuYFiRQGXffbWVpeWV1bL2wUN7e2d3ZLe/sNE6eaQ53HMtatgBmQQkEdBUpoJRpYFEhoBsPrid98AG1ErO5wlIAfsb4SoeAMrXTfQXhCHWU6VfRs3C2V3Yo7BV0kXk7KJEetW/rq9GKeRqCQS2ZM23MT9DOmUXAJ42InNZAwPmR9aFuqWATGz6ZXj+mxVXo0jLUthXSq/p7IWGTMKApsZ8RwYOa9ifif104xvPQzoZIUQfHZojCVFGM6iYD2hAaOcmQJ41rYWykfMM042qCKNgRv/uVF0jiteG7Fuz0vV6/yOArkkByRE+KRC1IlN6RG6oQTTZ7JK3lzHp0X5935mLUuOfnMAfkD5/MHvg6Spg==</latexit><latexit sha1_base64="ozX/0t3JPmydgwQzsf6ljZW+R6o=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KokKeix68VjBfkAby2Y7aZduNmF3opbQ/+HFgyJe/S/e/Ddu2xy09cHA470ZZuYFiRQGXffbWVpeWV1bL2wUN7e2d3ZLe/sNE6eaQ53HMtatgBmQQkEdBUpoJRpYFEhoBsPrid98AG1ErO5wlIAfsb4SoeAMrXTfQXhCHWU6VfRs3C2V3Yo7BV0kXk7KJEetW/rq9GKeRqCQS2ZM23MT9DOmUXAJ42InNZAwPmR9aFuqWATGz6ZXj+mxVXo0jLUthXSq/p7IWGTMKApsZ8RwYOa9ifif104xvPQzoZIUQfHZojCVFGM6iYD2hAaOcmQJ41rYWykfMM042qCKNgRv/uVF0jiteG7Fuz0vV6/yOArkkByRE+KRC1IlN6RG6oQTTZ7JK3lzHp0X5935mLUuOfnMAfkD5/MHvg6Spg==</latexit>
run 4 <latexit sha1_base64="ue3WBNuo2mSgITFm5zybLczT74E=">AAAB9XicbVBNS8NAEN3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaWDbbSbt0swm7E7WE/g8vHhTx6n/x5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqDg0ey1i3A2ZACgUNFCihnWhgUSChFYyup37rAbQRsbrDcQJ+xAZKhIIztNJ9F+EJdZTpVNHzSa9ccavuDHSZeDmpkBz1Xvmr2495GoFCLpkxHc9N0M+YRsElTErd1EDC+IgNoGOpYhEYP5tdPaEnVunTMNa2FNKZ+nsiY5Ex4yiwnRHDoVn0puJ/XifF8NLPhEpSBMXni8JUUozpNALaFxo4yrEljGthb6V8yDTjaIMq2RC8xZeXSfOs6rlV7/a8UrvK4yiSI3JMTolHLkiN3JA6aRBONHkmr+TNeXRenHfnY95acPKZQ/IHzucPv5OSpw==</latexit><latexit sha1_base64="ue3WBNuo2mSgITFm5zybLczT74E=">AAAB9XicbVBNS8NAEN3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaWDbbSbt0swm7E7WE/g8vHhTx6n/x5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqDg0ey1i3A2ZACgUNFCihnWhgUSChFYyup37rAbQRsbrDcQJ+xAZKhIIztNJ9F+EJdZTpVNHzSa9ccavuDHSZeDmpkBz1Xvmr2495GoFCLpkxHc9N0M+YRsElTErd1EDC+IgNoGOpYhEYP5tdPaEnVunTMNa2FNKZ+nsiY5Ex4yiwnRHDoVn0puJ/XifF8NLPhEpSBMXni8JUUozpNALaFxo4yrEljGthb6V8yDTjaIMq2RC8xZeXSfOs6rlV7/a8UrvK4yiSI3JMTolHLkiN3JA6aRBONHkmr+TNeXRenHfnY95acPKZQ/IHzucPv5OSpw==</latexit><latexit sha1_base64="ue3WBNuo2mSgITFm5zybLczT74E=">AAAB9XicbVBNS8NAEN3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaWDbbSbt0swm7E7WE/g8vHhTx6n/x5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqDg0ey1i3A2ZACgUNFCihnWhgUSChFYyup37rAbQRsbrDcQJ+xAZKhIIztNJ9F+EJdZTpVNHzSa9ccavuDHSZeDmpkBz1Xvmr2495GoFCLpkxHc9N0M+YRsElTErd1EDC+IgNoGOpYhEYP5tdPaEnVunTMNa2FNKZ+nsiY5Ex4yiwnRHDoVn0puJ/XifF8NLPhEpSBMXni8JUUozpNALaFxo4yrEljGthb6V8yDTjaIMq2RC8xZeXSfOs6rlV7/a8UrvK4yiSI3JMTolHLkiN3JA6aRBONHkmr+TNeXRenHfnY95acPKZQ/IHzucPv5OSpw==</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="/tomWnPbbdkqRAl0U8LNJQYXIBo=">AAAB6nicbZBLSwMxFIXv+Ky1anXrJlgEV2VGBF0KblxWsA9oa8mkd9rQTGZI7qhl6P9w40IRf5A7/43pY6GtBwIf5yTcmxOmSlry/W9vbX1jc2u7sFPcLe3tH5QPSw2bZEZgXSQqMa2QW1RSY50kKWylBnkcKmyGo5tp3nxEY2Wi72mcYjfmAy0jKTg566FD+Ewmzk2m2cWkV674VX8mtgrBAiqwUK1X/ur0E5HFqEkobm078FPq5tyQFAonxU5mMeVixAfYdqh5jLabz7aesFPn9FmUGHc0sZn7+0XOY2vHcehuxpyGdjmbmv9l7Yyiq24udZoRajEfFGWKUcKmFbC+NChIjR1wYaTblYkhN1yQK6roSgiWv7wKjfNq4FeDOx8KcAwncAYBXMI13EIN6iDAwAu8wbv35L16H/O61rxFb0fwR97nD27qkUc=</latexit><latexit sha1_base64="/tomWnPbbdkqRAl0U8LNJQYXIBo=">AAAB6nicbZBLSwMxFIXv+Ky1anXrJlgEV2VGBF0KblxWsA9oa8mkd9rQTGZI7qhl6P9w40IRf5A7/43pY6GtBwIf5yTcmxOmSlry/W9vbX1jc2u7sFPcLe3tH5QPSw2bZEZgXSQqMa2QW1RSY50kKWylBnkcKmyGo5tp3nxEY2Wi72mcYjfmAy0jKTg566FD+Ewmzk2m2cWkV674VX8mtgrBAiqwUK1X/ur0E5HFqEkobm078FPq5tyQFAonxU5mMeVixAfYdqh5jLabz7aesFPn9FmUGHc0sZn7+0XOY2vHcehuxpyGdjmbmv9l7Yyiq24udZoRajEfFGWKUcKmFbC+NChIjR1wYaTblYkhN1yQK6roSgiWv7wKjfNq4FeDOx8KcAwncAYBXMI13EIN6iDAwAu8wbv35L16H/O61rxFb0fwR97nD27qkUc=</latexit><latexit sha1_base64="s84kqqt315Hp39e4bdrCATvszSI=">AAAB9XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaWDbbTbt0swm7E7WE/g8vHhTx6n/x5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqxhsslrFuB9RwKRRvoEDJ24nmNAokbwWj66nfeuDaiFjd4TjhfkQHSoSCUbTSfRf5E+oo06ki55NeueJW3RnIMvFyUoEc9V75q9uPWRpxhUxSYzqem6CfUY2CST4pdVPDE8pGdMA7lioaceNns6sn5MQqfRLG2pZCMlN/T2Q0MmYcBbYzojg0i95U/M/rpBhe+plQSYpcsfmiMJUEYzKNgPSF5gzl2BLKtLC3EjakmjK0QZVsCN7iy8ukeVb13Kp361ZqV3kcRTiCYzgFDy6gBjdQhwYw0PAMr/DmPDovzrvzMW8tOPnMIfyB8/kDvlOSow==</latexit><latexit sha1_base64="ue3WBNuo2mSgITFm5zybLczT74E=">AAAB9XicbVBNS8NAEN3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaWDbbSbt0swm7E7WE/g8vHhTx6n/x5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqDg0ey1i3A2ZACgUNFCihnWhgUSChFYyup37rAbQRsbrDcQJ+xAZKhIIztNJ9F+EJdZTpVNHzSa9ccavuDHSZeDmpkBz1Xvmr2495GoFCLpkxHc9N0M+YRsElTErd1EDC+IgNoGOpYhEYP5tdPaEnVunTMNa2FNKZ+nsiY5Ex4yiwnRHDoVn0puJ/XifF8NLPhEpSBMXni8JUUozpNALaFxo4yrEljGthb6V8yDTjaIMq2RC8xZeXSfOs6rlV7/a8UrvK4yiSI3JMTolHLkiN3JA6aRBONHkmr+TNeXRenHfnY95acPKZQ/IHzucPv5OSpw==</latexit><latexit sha1_base64="ue3WBNuo2mSgITFm5zybLczT74E=">AAAB9XicbVBNS8NAEN3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaWDbbSbt0swm7E7WE/g8vHhTx6n/x5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqDg0ey1i3A2ZACgUNFCihnWhgUSChFYyup37rAbQRsbrDcQJ+xAZKhIIztNJ9F+EJdZTpVNHzSa9ccavuDHSZeDmpkBz1Xvmr2495GoFCLpkxHc9N0M+YRsElTErd1EDC+IgNoGOpYhEYP5tdPaEnVunTMNa2FNKZ+nsiY5Ex4yiwnRHDoVn0puJ/XifF8NLPhEpSBMXni8JUUozpNALaFxo4yrEljGthb6V8yDTjaIMq2RC8xZeXSfOs6rlV7/a8UrvK4yiSI3JMTolHLkiN3JA6aRBONHkmr+TNeXRenHfnY95acPKZQ/IHzucPv5OSpw==</latexit><latexit sha1_base64="ue3WBNuo2mSgITFm5zybLczT74E=">AAAB9XicbVBNS8NAEN3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaWDbbSbt0swm7E7WE/g8vHhTx6n/x5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqDg0ey1i3A2ZACgUNFCihnWhgUSChFYyup37rAbQRsbrDcQJ+xAZKhIIztNJ9F+EJdZTpVNHzSa9ccavuDHSZeDmpkBz1Xvmr2495GoFCLpkxHc9N0M+YRsElTErd1EDC+IgNoGOpYhEYP5tdPaEnVunTMNa2FNKZ+nsiY5Ex4yiwnRHDoVn0puJ/XifF8NLPhEpSBMXni8JUUozpNALaFxo4yrEljGthb6V8yDTjaIMq2RC8xZeXSfOs6rlV7/a8UrvK4yiSI3JMTolHLkiN3JA6aRBONHkmr+TNeXRenHfnY95acPKZQ/IHzucPv5OSpw==</latexit><latexit sha1_base64="ue3WBNuo2mSgITFm5zybLczT74E=">AAAB9XicbVBNS8NAEN3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaWDbbSbt0swm7E7WE/g8vHhTx6n/x5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqDg0ey1i3A2ZACgUNFCihnWhgUSChFYyup37rAbQRsbrDcQJ+xAZKhIIztNJ9F+EJdZTpVNHzSa9ccavuDHSZeDmpkBz1Xvmr2495GoFCLpkxHc9N0M+YRsElTErd1EDC+IgNoGOpYhEYP5tdPaEnVunTMNa2FNKZ+nsiY5Ex4yiwnRHDoVn0puJ/XifF8NLPhEpSBMXni8JUUozpNALaFxo4yrEljGthb6V8yDTjaIMq2RC8xZeXSfOs6rlV7/a8UrvK4yiSI3JMTolHLkiN3JA6aRBONHkmr+TNeXRenHfnY95acPKZQ/IHzucPv5OSpw==</latexit><latexit sha1_base64="ue3WBNuo2mSgITFm5zybLczT74E=">AAAB9XicbVBNS8NAEN3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaWDbbSbt0swm7E7WE/g8vHhTx6n/x5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqDg0ey1i3A2ZACgUNFCihnWhgUSChFYyup37rAbQRsbrDcQJ+xAZKhIIztNJ9F+EJdZTpVNHzSa9ccavuDHSZeDmpkBz1Xvmr2495GoFCLpkxHc9N0M+YRsElTErd1EDC+IgNoGOpYhEYP5tdPaEnVunTMNa2FNKZ+nsiY5Ex4yiwnRHDoVn0puJ/XifF8NLPhEpSBMXni8JUUozpNALaFxo4yrEljGthb6V8yDTjaIMq2RC8xZeXSfOs6rlV7/a8UrvK4yiSI3JMTolHLkiN3JA6aRBONHkmr+TNeXRenHfnY95acPKZQ/IHzucPv5OSpw==</latexit><latexit sha1_base64="ue3WBNuo2mSgITFm5zybLczT74E=">AAAB9XicbVBNS8NAEN3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaWDbbSbt0swm7E7WE/g8vHhTx6n/x5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqDg0ey1i3A2ZACgUNFCihnWhgUSChFYyup37rAbQRsbrDcQJ+xAZKhIIztNJ9F+EJdZTpVNHzSa9ccavuDHSZeDmpkBz1Xvmr2495GoFCLpkxHc9N0M+YRsElTErd1EDC+IgNoGOpYhEYP5tdPaEnVunTMNa2FNKZ+nsiY5Ex4yiwnRHDoVn0puJ/XifF8NLPhEpSBMXni8JUUozpNALaFxo4yrEljGthb6V8yDTjaIMq2RC8xZeXSfOs6rlV7/a8UrvK4yiSI3JMTolHLkiN3JA6aRBONHkmr+TNeXRenHfnY95acPKZQ/IHzucPv5OSpw==</latexit>
run 5 <latexit sha1_base64="9lCxLojEYNROqd5ZW2zY5IE7ONI=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4Kokoeix68VjBfkAby2Y7aZduNmF3opbQ/+HFgyJe/S/e/Ddu2xy09cHA470ZZuYFiRQGXffbWVpeWV1bL2wUN7e2d3ZLe/sNE6eaQ53HMtatgBmQQkEdBUpoJRpYFEhoBsPrid98AG1ErO5wlIAfsb4SoeAMrXTfQXhCHWU6VfR83C2V3Yo7BV0kXk7KJEetW/rq9GKeRqCQS2ZM23MT9DOmUXAJ42InNZAwPmR9aFuqWATGz6ZXj+mxVXo0jLUthXSq/p7IWGTMKApsZ8RwYOa9ifif104xvPQzoZIUQfHZojCVFGM6iYD2hAaOcmQJ41rYWykfMM042qCKNgRv/uVF0jiteG7Fuz0rV6/yOArkkByRE+KRC1IlN6RG6oQTTZ7JK3lzHp0X5935mLUuOfnMAfkD5/MHwRiSqA==</latexit><latexit sha1_base64="9lCxLojEYNROqd5ZW2zY5IE7ONI=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4Kokoeix68VjBfkAby2Y7aZduNmF3opbQ/+HFgyJe/S/e/Ddu2xy09cHA470ZZuYFiRQGXffbWVpeWV1bL2wUN7e2d3ZLe/sNE6eaQ53HMtatgBmQQkEdBUpoJRpYFEhoBsPrid98AG1ErO5wlIAfsb4SoeAMrXTfQXhCHWU6VfR83C2V3Yo7BV0kXk7KJEetW/rq9GKeRqCQS2ZM23MT9DOmUXAJ42InNZAwPmR9aFuqWATGz6ZXj+mxVXo0jLUthXSq/p7IWGTMKApsZ8RwYOa9ifif104xvPQzoZIUQfHZojCVFGM6iYD2hAaOcmQJ41rYWykfMM042qCKNgRv/uVF0jiteG7Fuz0rV6/yOArkkByRE+KRC1IlN6RG6oQTTZ7JK3lzHp0X5935mLUuOfnMAfkD5/MHwRiSqA==</latexit><latexit sha1_base64="9lCxLojEYNROqd5ZW2zY5IE7ONI=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4Kokoeix68VjBfkAby2Y7aZduNmF3opbQ/+HFgyJe/S/e/Ddu2xy09cHA470ZZuYFiRQGXffbWVpeWV1bL2wUN7e2d3ZLe/sNE6eaQ53HMtatgBmQQkEdBUpoJRpYFEhoBsPrid98AG1ErO5wlIAfsb4SoeAMrXTfQXhCHWU6VfR83C2V3Yo7BV0kXk7KJEetW/rq9GKeRqCQS2ZM23MT9DOmUXAJ42InNZAwPmR9aFuqWATGz6ZXj+mxVXo0jLUthXSq/p7IWGTMKApsZ8RwYOa9ifif104xvPQzoZIUQfHZojCVFGM6iYD2hAaOcmQJ41rYWykfMM042qCKNgRv/uVF0jiteG7Fuz0rV6/yOArkkByRE+KRC1IlN6RG6oQTTZ7JK3lzHp0X5935mLUuOfnMAfkD5/MHwRiSqA==</latexit><latexit sha1_base64="9lCxLojEYNROqd5ZW2zY5IE7ONI=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4Kokoeix68VjBfkAby2Y7aZduNmF3opbQ/+HFgyJe/S/e/Ddu2xy09cHA470ZZuYFiRQGXffbWVpeWV1bL2wUN7e2d3ZLe/sNE6eaQ53HMtatgBmQQkEdBUpoJRpYFEhoBsPrid98AG1ErO5wlIAfsb4SoeAMrXTfQXhCHWU6VfR83C2V3Yo7BV0kXk7KJEetW/rq9GKeRqCQS2ZM23MT9DOmUXAJ42InNZAwPmR9aFuqWATGz6ZXj+mxVXo0jLUthXSq/p7IWGTMKApsZ8RwYOa9ifif104xvPQzoZIUQfHZojCVFGM6iYD2hAaOcmQJ41rYWykfMM042qCKNgRv/uVF0jiteG7Fuz0rV6/yOArkkByRE+KRC1IlN6RG6oQTTZ7JK3lzHp0X5935mLUuOfnMAfkD5/MHwRiSqA==</latexit>
run 6 <latexit sha1_base64="z+40nnry1acL6Q+gUVhFrRezuLk=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KomIeix68VjBfkAby2Y7aZduNmF3opbQ/+HFgyJe/S/e/Ddu2xy09cHA470ZZuYFiRQGXffbWVpeWV1bL2wUN7e2d3ZLe/sNE6eaQ53HMtatgBmQQkEdBUpoJRpYFEhoBsPrid98AG1ErO5wlIAfsb4SoeAMrXTfQXhCHWU6VfR83C2V3Yo7BV0kXk7KJEetW/rq9GKeRqCQS2ZM23MT9DOmUXAJ42InNZAwPmR9aFuqWATGz6ZXj+mxVXo0jLUthXSq/p7IWGTMKApsZ8RwYOa9ifif104xvPQzoZIUQfHZojCVFGM6iYD2hAaOcmQJ41rYWykfMM042qCKNgRv/uVF0jiteG7Fuz0rV6/yOArkkByRE+KRC1IlN6RG6oQTTZ7JK3lzHp0X5935mLUuOfnMAfkD5/MHwp2SqQ==</latexit><latexit sha1_base64="z+40nnry1acL6Q+gUVhFrRezuLk=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KomIeix68VjBfkAby2Y7aZduNmF3opbQ/+HFgyJe/S/e/Ddu2xy09cHA470ZZuYFiRQGXffbWVpeWV1bL2wUN7e2d3ZLe/sNE6eaQ53HMtatgBmQQkEdBUpoJRpYFEhoBsPrid98AG1ErO5wlIAfsb4SoeAMrXTfQXhCHWU6VfR83C2V3Yo7BV0kXk7KJEetW/rq9GKeRqCQS2ZM23MT9DOmUXAJ42InNZAwPmR9aFuqWATGz6ZXj+mxVXo0jLUthXSq/p7IWGTMKApsZ8RwYOa9ifif104xvPQzoZIUQfHZojCVFGM6iYD2hAaOcmQJ41rYWykfMM042qCKNgRv/uVF0jiteG7Fuz0rV6/yOArkkByRE+KRC1IlN6RG6oQTTZ7JK3lzHp0X5935mLUuOfnMAfkD5/MHwp2SqQ==</latexit><latexit sha1_base64="z+40nnry1acL6Q+gUVhFrRezuLk=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KomIeix68VjBfkAby2Y7aZduNmF3opbQ/+HFgyJe/S/e/Ddu2xy09cHA470ZZuYFiRQGXffbWVpeWV1bL2wUN7e2d3ZLe/sNE6eaQ53HMtatgBmQQkEdBUpoJRpYFEhoBsPrid98AG1ErO5wlIAfsb4SoeAMrXTfQXhCHWU6VfR83C2V3Yo7BV0kXk7KJEetW/rq9GKeRqCQS2ZM23MT9DOmUXAJ42InNZAwPmR9aFuqWATGz6ZXj+mxVXo0jLUthXSq/p7IWGTMKApsZ8RwYOa9ifif104xvPQzoZIUQfHZojCVFGM6iYD2hAaOcmQJ41rYWykfMM042qCKNgRv/uVF0jiteG7Fuz0rV6/yOArkkByRE+KRC1IlN6RG6oQTTZ7JK3lzHp0X5935mLUuOfnMAfkD5/MHwp2SqQ==</latexit><latexit sha1_base64="z+40nnry1acL6Q+gUVhFrRezuLk=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KomIeix68VjBfkAby2Y7aZduNmF3opbQ/+HFgyJe/S/e/Ddu2xy09cHA470ZZuYFiRQGXffbWVpeWV1bL2wUN7e2d3ZLe/sNE6eaQ53HMtatgBmQQkEdBUpoJRpYFEhoBsPrid98AG1ErO5wlIAfsb4SoeAMrXTfQXhCHWU6VfR83C2V3Yo7BV0kXk7KJEetW/rq9GKeRqCQS2ZM23MT9DOmUXAJ42InNZAwPmR9aFuqWATGz6ZXj+mxVXo0jLUthXSq/p7IWGTMKApsZ8RwYOa9ifif104xvPQzoZIUQfHZojCVFGM6iYD2hAaOcmQJ41rYWykfMM042qCKNgRv/uVF0jiteG7Fuz0rV6/yOArkkByRE+KRC1IlN6RG6oQTTZ7JK3lzHp0X5935mLUuOfnMAfkD5/MHwp2SqQ==</latexit>
run 7 <latexit sha1_base64="EfNQARihNovNhGzbe22A/IikzaI=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KokI9Vj04rGC/YA2ls120i7dbMLuRC2h/8OLB0W8+l+8+W/ctjlo64OBx3szzMwLEikMuu63s7K6tr6xWdgqbu/s7u2XDg6bJk41hwaPZazbATMghYIGCpTQTjSwKJDQCkbXU7/1ANqIWN3hOAE/YgMlQsEZWum+i/CEOsp0qmh10iuV3Yo7A10mXk7KJEe9V/rq9mOeRqCQS2ZMx3MT9DOmUXAJk2I3NZAwPmID6FiqWATGz2ZXT+ipVfo0jLUthXSm/p7IWGTMOApsZ8RwaBa9qfif10kxvPQzoZIUQfH5ojCVFGM6jYD2hQaOcmwJ41rYWykfMs042qCKNgRv8eVl0jyveG7Fu70o167yOArkmJyQM+KRKqmRG1InDcKJJs/klbw5j86L8+58zFtXnHzmiPyB8/kDxCKSqg==</latexit><latexit sha1_base64="EfNQARihNovNhGzbe22A/IikzaI=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KokI9Vj04rGC/YA2ls120i7dbMLuRC2h/8OLB0W8+l+8+W/ctjlo64OBx3szzMwLEikMuu63s7K6tr6xWdgqbu/s7u2XDg6bJk41hwaPZazbATMghYIGCpTQTjSwKJDQCkbXU7/1ANqIWN3hOAE/YgMlQsEZWum+i/CEOsp0qmh10iuV3Yo7A10mXk7KJEe9V/rq9mOeRqCQS2ZMx3MT9DOmUXAJk2I3NZAwPmID6FiqWATGz2ZXT+ipVfo0jLUthXSm/p7IWGTMOApsZ8RwaBa9qfif10kxvPQzoZIUQfH5ojCVFGM6jYD2hQaOcmwJ41rYWykfMs042qCKNgRv8eVl0jyveG7Fu70o167yOArkmJyQM+KRKqmRG1InDcKJJs/klbw5j86L8+58zFtXnHzmiPyB8/kDxCKSqg==</latexit><latexit sha1_base64="EfNQARihNovNhGzbe22A/IikzaI=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KokI9Vj04rGC/YA2ls120i7dbMLuRC2h/8OLB0W8+l+8+W/ctjlo64OBx3szzMwLEikMuu63s7K6tr6xWdgqbu/s7u2XDg6bJk41hwaPZazbATMghYIGCpTQTjSwKJDQCkbXU7/1ANqIWN3hOAE/YgMlQsEZWum+i/CEOsp0qmh10iuV3Yo7A10mXk7KJEe9V/rq9mOeRqCQS2ZMx3MT9DOmUXAJk2I3NZAwPmID6FiqWATGz2ZXT+ipVfo0jLUthXSm/p7IWGTMOApsZ8RwaBa9qfif10kxvPQzoZIUQfH5ojCVFGM6jYD2hQaOcmwJ41rYWykfMs042qCKNgRv8eVl0jyveG7Fu70o167yOArkmJyQM+KRKqmRG1InDcKJJs/klbw5j86L8+58zFtXnHzmiPyB8/kDxCKSqg==</latexit><latexit sha1_base64="EfNQARihNovNhGzbe22A/IikzaI=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KokI9Vj04rGC/YA2ls120i7dbMLuRC2h/8OLB0W8+l+8+W/ctjlo64OBx3szzMwLEikMuu63s7K6tr6xWdgqbu/s7u2XDg6bJk41hwaPZazbATMghYIGCpTQTjSwKJDQCkbXU7/1ANqIWN3hOAE/YgMlQsEZWum+i/CEOsp0qmh10iuV3Yo7A10mXk7KJEe9V/rq9mOeRqCQS2ZMx3MT9DOmUXAJk2I3NZAwPmID6FiqWATGz2ZXT+ipVfo0jLUthXSm/p7IWGTMOApsZ8RwaBa9qfif10kxvPQzoZIUQfH5ojCVFGM6jYD2hQaOcmwJ41rYWykfMs042qCKNgRv8eVl0jyveG7Fu70o167yOArkmJyQM+KRKqmRG1InDcKJJs/klbw5j86L8+58zFtXnHzmiPyB8/kDxCKSqg==</latexit>
run 8 <latexit sha1_base64="JHfo/0HAVB9gKRMN0ZwZmDCOIBE=">AAAB9XicbVBNS8NAEJ34WetX1aOXxSJ4KokI9lj04rGC/YA2ls120y7dbMLuRC2h/8OLB0W8+l+8+W/ctjlo64OBx3szzMwLEikMuu63s7K6tr6xWdgqbu/s7u2XDg6bJk414w0Wy1i3A2q4FIo3UKDk7URzGgWSt4LR9dRvPXBtRKzucJxwP6IDJULBKFrpvov8CXWU6VSR6qRXKrsVdwayTLyclCFHvVf66vZjlkZcIZPUmI7nJuhnVKNgkk+K3dTwhLIRHfCOpYpG3PjZ7OoJObVKn4SxtqWQzNTfExmNjBlHge2MKA7NojcV//M6KYZVPxMqSZErNl8UppJgTKYRkL7QnKEcW0KZFvZWwoZUU4Y2qKINwVt8eZk0zyueW/FuL8q1qzyOAhzDCZyBB5dQgxuoQwMYaHiGV3hzHp0X5935mLeuOPnMEfyB8/kDxaeSqw==</latexit><latexit sha1_base64="JHfo/0HAVB9gKRMN0ZwZmDCOIBE=">AAAB9XicbVBNS8NAEJ34WetX1aOXxSJ4KokI9lj04rGC/YA2ls120y7dbMLuRC2h/8OLB0W8+l+8+W/ctjlo64OBx3szzMwLEikMuu63s7K6tr6xWdgqbu/s7u2XDg6bJk414w0Wy1i3A2q4FIo3UKDk7URzGgWSt4LR9dRvPXBtRKzucJxwP6IDJULBKFrpvov8CXWU6VSR6qRXKrsVdwayTLyclCFHvVf66vZjlkZcIZPUmI7nJuhnVKNgkk+K3dTwhLIRHfCOpYpG3PjZ7OoJObVKn4SxtqWQzNTfExmNjBlHge2MKA7NojcV//M6KYZVPxMqSZErNl8UppJgTKYRkL7QnKEcW0KZFvZWwoZUU4Y2qKINwVt8eZk0zyueW/FuL8q1qzyOAhzDCZyBB5dQgxuoQwMYaHiGV3hzHp0X5935mLeuOPnMEfyB8/kDxaeSqw==</latexit><latexit sha1_base64="JHfo/0HAVB9gKRMN0ZwZmDCOIBE=">AAAB9XicbVBNS8NAEJ34WetX1aOXxSJ4KokI9lj04rGC/YA2ls120y7dbMLuRC2h/8OLB0W8+l+8+W/ctjlo64OBx3szzMwLEikMuu63s7K6tr6xWdgqbu/s7u2XDg6bJk414w0Wy1i3A2q4FIo3UKDk7URzGgWSt4LR9dRvPXBtRKzucJxwP6IDJULBKFrpvov8CXWU6VSR6qRXKrsVdwayTLyclCFHvVf66vZjlkZcIZPUmI7nJuhnVKNgkk+K3dTwhLIRHfCOpYpG3PjZ7OoJObVKn4SxtqWQzNTfExmNjBlHge2MKA7NojcV//M6KYZVPxMqSZErNl8UppJgTKYRkL7QnKEcW0KZFvZWwoZUU4Y2qKINwVt8eZk0zyueW/FuL8q1qzyOAhzDCZyBB5dQgxuoQwMYaHiGV3hzHp0X5935mLeuOPnMEfyB8/kDxaeSqw==</latexit><latexit sha1_base64="JHfo/0HAVB9gKRMN0ZwZmDCOIBE=">AAAB9XicbVBNS8NAEJ34WetX1aOXxSJ4KokI9lj04rGC/YA2ls120y7dbMLuRC2h/8OLB0W8+l+8+W/ctjlo64OBx3szzMwLEikMuu63s7K6tr6xWdgqbu/s7u2XDg6bJk414w0Wy1i3A2q4FIo3UKDk7URzGgWSt4LR9dRvPXBtRKzucJxwP6IDJULBKFrpvov8CXWU6VSR6qRXKrsVdwayTLyclCFHvVf66vZjlkZcIZPUmI7nJuhnVKNgkk+K3dTwhLIRHfCOpYpG3PjZ7OoJObVKn4SxtqWQzNTfExmNjBlHge2MKA7NojcV//M6KYZVPxMqSZErNl8UppJgTKYRkL7QnKEcW0KZFvZWwoZUU4Y2qKINwVt8eZk0zyueW/FuL8q1qzyOAhzDCZyBB5dQgxuoQwMYaHiGV3hzHp0X5935mLeuOPnMEfyB8/kDxaeSqw==</latexit>
run 9 <latexit sha1_base64="ZKiO5wNI3nePcwucVRaceCXQs9A=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KokI6q3oxWMF+wFtLJvtpF262YTdiVpC/4cXD4p49b9489+4bXPQ1gcDj/dmmJkXJFIYdN1vZ2l5ZXVtvbBR3Nza3tkt7e03TJxqDnUey1i3AmZACgV1FCihlWhgUSChGQyvJ37zAbQRsbrDUQJ+xPpKhIIztNJ9B+EJdZTpVNHLcbdUdivuFHSReDkpkxy1bumr04t5GoFCLpkxbc9N0M+YRsEljIud1EDC+JD1oW2pYhEYP5tePabHVunRMNa2FNKp+nsiY5ExoyiwnRHDgZn3JuJ/XjvF8MLPhEpSBMVni8JUUozpJALaExo4ypEljGthb6V8wDTjaIMq2hC8+ZcXSeO04rkV7/asXL3K4yiQQ3JETohHzkmV3JAaqRNONHkmr+TNeXRenHfnY9a65OQzB+QPnM8fxyySrA==</latexit><latexit sha1_base64="ZKiO5wNI3nePcwucVRaceCXQs9A=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KokI6q3oxWMF+wFtLJvtpF262YTdiVpC/4cXD4p49b9489+4bXPQ1gcDj/dmmJkXJFIYdN1vZ2l5ZXVtvbBR3Nza3tkt7e03TJxqDnUey1i3AmZACgV1FCihlWhgUSChGQyvJ37zAbQRsbrDUQJ+xPpKhIIztNJ9B+EJdZTpVNHLcbdUdivuFHSReDkpkxy1bumr04t5GoFCLpkxbc9N0M+YRsEljIud1EDC+JD1oW2pYhEYP5tePabHVunRMNa2FNKp+nsiY5ExoyiwnRHDgZn3JuJ/XjvF8MLPhEpSBMVni8JUUozpJALaExo4ypEljGthb6V8wDTjaIMq2hC8+ZcXSeO04rkV7/asXL3K4yiQQ3JETohHzkmV3JAaqRNONHkmr+TNeXRenHfnY9a65OQzB+QPnM8fxyySrA==</latexit><latexit sha1_base64="ZKiO5wNI3nePcwucVRaceCXQs9A=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KokI6q3oxWMF+wFtLJvtpF262YTdiVpC/4cXD4p49b9489+4bXPQ1gcDj/dmmJkXJFIYdN1vZ2l5ZXVtvbBR3Nza3tkt7e03TJxqDnUey1i3AmZACgV1FCihlWhgUSChGQyvJ37zAbQRsbrDUQJ+xPpKhIIztNJ9B+EJdZTpVNHLcbdUdivuFHSReDkpkxy1bumr04t5GoFCLpkxbc9N0M+YRsEljIud1EDC+JD1oW2pYhEYP5tePabHVunRMNa2FNKp+nsiY5ExoyiwnRHDgZn3JuJ/XjvF8MLPhEpSBMVni8JUUozpJALaExo4ypEljGthb6V8wDTjaIMq2hC8+ZcXSeO04rkV7/asXL3K4yiQQ3JETohHzkmV3JAaqRNONHkmr+TNeXRenHfnY9a65OQzB+QPnM8fxyySrA==</latexit><latexit sha1_base64="ZKiO5wNI3nePcwucVRaceCXQs9A=">AAAB9XicbVBNS8NAEN34WetX1aOXxSJ4KokI6q3oxWMF+wFtLJvtpF262YTdiVpC/4cXD4p49b9489+4bXPQ1gcDj/dmmJkXJFIYdN1vZ2l5ZXVtvbBR3Nza3tkt7e03TJxqDnUey1i3AmZACgV1FCihlWhgUSChGQyvJ37zAbQRsbrDUQJ+xPpKhIIztNJ9B+EJdZTpVNHLcbdUdivuFHSReDkpkxy1bumr04t5GoFCLpkxbc9N0M+YRsEljIud1EDC+JD1oW2pYhEYP5tePabHVunRMNa2FNKp+nsiY5ExoyiwnRHDgZn3JuJ/XjvF8MLPhEpSBMVni8JUUozpJALaExo4ypEljGthb6V8wDTjaIMq2hC8+ZcXSeO04rkV7/asXL3K4yiQQ3JETohHzkmV3JAaqRNONHkmr+TNeXRenHfnY9a65OQzB+QPnM8fxyySrA==</latexit>
run 10 <latexit sha1_base64="rMZt6lLsG4QW0FzbkSd9wwyTKHE=">AAAB+HicbVBNSwMxEM3Wr1o/uurRS7AInsquCHosevFYwX5Au5RsmrahSXZJJmJd+ku8eFDEqz/Fm//GtN2Dtj4YeLw3w8y8OBXcQBB8e4W19Y3NreJ2aWd3b7/sHxw2TWI1ZQ2aiES3Y2KY4Io1gINg7VQzImPBWvH4Zua3Hpg2PFH3MElZJMlQ8QGnBJzU88tdYI+gZaatwmEw7fmVoBrMgVdJmJMKylHv+V/dfkKtZAqoIMZ0wiCFKCMaOBVsWupaw1JCx2TIOo4qIpmJsvnhU3zqlD4eJNqVAjxXf09kRBozkbHrlARGZtmbif95HQuDqyjjKrXAFF0sGliBIcGzFHCfa0ZBTBwhVHN3K6YjogkFl1XJhRAuv7xKmufVMKiGdxeV2nUeRxEdoxN0hkJ0iWroFtVRA1Fk0TN6RW/ek/fivXsfi9aCl88coT/wPn8ApdeTDw==</latexit><latexit sha1_base64="rMZt6lLsG4QW0FzbkSd9wwyTKHE=">AAAB+HicbVBNSwMxEM3Wr1o/uurRS7AInsquCHosevFYwX5Au5RsmrahSXZJJmJd+ku8eFDEqz/Fm//GtN2Dtj4YeLw3w8y8OBXcQBB8e4W19Y3NreJ2aWd3b7/sHxw2TWI1ZQ2aiES3Y2KY4Io1gINg7VQzImPBWvH4Zua3Hpg2PFH3MElZJMlQ8QGnBJzU88tdYI+gZaatwmEw7fmVoBrMgVdJmJMKylHv+V/dfkKtZAqoIMZ0wiCFKCMaOBVsWupaw1JCx2TIOo4qIpmJsvnhU3zqlD4eJNqVAjxXf09kRBozkbHrlARGZtmbif95HQuDqyjjKrXAFF0sGliBIcGzFHCfa0ZBTBwhVHN3K6YjogkFl1XJhRAuv7xKmufVMKiGdxeV2nUeRxEdoxN0hkJ0iWroFtVRA1Fk0TN6RW/ek/fivXsfi9aCl88coT/wPn8ApdeTDw==</latexit><latexit sha1_base64="rMZt6lLsG4QW0FzbkSd9wwyTKHE=">AAAB+HicbVBNSwMxEM3Wr1o/uurRS7AInsquCHosevFYwX5Au5RsmrahSXZJJmJd+ku8eFDEqz/Fm//GtN2Dtj4YeLw3w8y8OBXcQBB8e4W19Y3NreJ2aWd3b7/sHxw2TWI1ZQ2aiES3Y2KY4Io1gINg7VQzImPBWvH4Zua3Hpg2PFH3MElZJMlQ8QGnBJzU88tdYI+gZaatwmEw7fmVoBrMgVdJmJMKylHv+V/dfkKtZAqoIMZ0wiCFKCMaOBVsWupaw1JCx2TIOo4qIpmJsvnhU3zqlD4eJNqVAjxXf09kRBozkbHrlARGZtmbif95HQuDqyjjKrXAFF0sGliBIcGzFHCfa0ZBTBwhVHN3K6YjogkFl1XJhRAuv7xKmufVMKiGdxeV2nUeRxEdoxN0hkJ0iWroFtVRA1Fk0TN6RW/ek/fivXsfi9aCl88coT/wPn8ApdeTDw==</latexit><latexit sha1_base64="rMZt6lLsG4QW0FzbkSd9wwyTKHE=">AAAB+HicbVBNSwMxEM3Wr1o/uurRS7AInsquCHosevFYwX5Au5RsmrahSXZJJmJd+ku8eFDEqz/Fm//GtN2Dtj4YeLw3w8y8OBXcQBB8e4W19Y3NreJ2aWd3b7/sHxw2TWI1ZQ2aiES3Y2KY4Io1gINg7VQzImPBWvH4Zua3Hpg2PFH3MElZJMlQ8QGnBJzU88tdYI+gZaatwmEw7fmVoBrMgVdJmJMKylHv+V/dfkKtZAqoIMZ0wiCFKCMaOBVsWupaw1JCx2TIOo4qIpmJsvnhU3zqlD4eJNqVAjxXf09kRBozkbHrlARGZtmbif95HQuDqyjjKrXAFF0sGliBIcGzFHCfa0ZBTBwhVHN3K6YjogkFl1XJhRAuv7xKmufVMKiGdxeV2nUeRxEdoxN0hkJ0iWroFtVRA1Fk0TN6RW/ek/fivXsfi9aCl88coT/wPn8ApdeTDw==</latexit>
|a| <latexit sha1_base64="p4NdPauRcwGeLSvEXoY+yd5ipTI=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8eK9gPaUDbbSbt0swm7G6Gk/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU1HGqGDZYLGLVDqhGwSU2DDcC24lCGgUCW8Hodua3nlBpHstHM07Qj+hA8pAzaqz0MKGTXrniVt05yCrxclKBHPVe+avbj1kaoTRMUK07npsYP6PKcCZwWuqmGhPKRnSAHUsljVD72fzUKTmzSp+EsbIlDZmrvycyGmk9jgLbGVEz1MveTPzP66QmvPYzLpPUoGSLRWEqiInJ7G/S5wqZEWNLKFPc3krYkCrKjE2nZEPwll9eJc2LqudWvfvLSu0mj6MIJ3AK5+DBFdTgDurQAAYDeIZXeHOE8+K8Ox+L1oKTzxzDHzifP4Z+jfE=</latexit><latexit sha1_base64="p4NdPauRcwGeLSvEXoY+yd5ipTI=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8eK9gPaUDbbSbt0swm7G6Gk/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU1HGqGDZYLGLVDqhGwSU2DDcC24lCGgUCW8Hodua3nlBpHstHM07Qj+hA8pAzaqz0MKGTXrniVt05yCrxclKBHPVe+avbj1kaoTRMUK07npsYP6PKcCZwWuqmGhPKRnSAHUsljVD72fzUKTmzSp+EsbIlDZmrvycyGmk9jgLbGVEz1MveTPzP66QmvPYzLpPUoGSLRWEqiInJ7G/S5wqZEWNLKFPc3krYkCrKjE2nZEPwll9eJc2LqudWvfvLSu0mj6MIJ3AK5+DBFdTgDurQAAYDeIZXeHOE8+K8Ox+L1oKTzxzDHzifP4Z+jfE=</latexit><latexit sha1_base64="p4NdPauRcwGeLSvEXoY+yd5ipTI=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8eK9gPaUDbbSbt0swm7G6Gk/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU1HGqGDZYLGLVDqhGwSU2DDcC24lCGgUCW8Hodua3nlBpHstHM07Qj+hA8pAzaqz0MKGTXrniVt05yCrxclKBHPVe+avbj1kaoTRMUK07npsYP6PKcCZwWuqmGhPKRnSAHUsljVD72fzUKTmzSp+EsbIlDZmrvycyGmk9jgLbGVEz1MveTPzP66QmvPYzLpPUoGSLRWEqiInJ7G/S5wqZEWNLKFPc3krYkCrKjE2nZEPwll9eJc2LqudWvfvLSu0mj6MIJ3AK5+DBFdTgDurQAAYDeIZXeHOE8+K8Ox+L1oKTzxzDHzifP4Z+jfE=</latexit><latexit sha1_base64="p4NdPauRcwGeLSvEXoY+yd5ipTI=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8eK9gPaUDbbSbt0swm7G6Gk/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU1HGqGDZYLGLVDqhGwSU2DDcC24lCGgUCW8Hodua3nlBpHstHM07Qj+hA8pAzaqz0MKGTXrniVt05yCrxclKBHPVe+avbj1kaoTRMUK07npsYP6PKcCZwWuqmGhPKRnSAHUsljVD72fzUKTmzSp+EsbIlDZmrvycyGmk9jgLbGVEz1MveTPzP66QmvPYzLpPUoGSLRWEqiInJ7G/S5wqZEWNLKFPc3krYkCrKjE2nZEPwll9eJc2LqudWvfvLSu0mj6MIJ3AK5+DBFdTgDurQAAYDeIZXeHOE8+K8Ox+L1oKTzxzDHzifP4Z+jfE=</latexit>
|a| <latexit sha1_base64="p4NdPauRcwGeLSvEXoY+yd5ipTI=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8eK9gPaUDbbSbt0swm7G6Gk/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU1HGqGDZYLGLVDqhGwSU2DDcC24lCGgUCW8Hodua3nlBpHstHM07Qj+hA8pAzaqz0MKGTXrniVt05yCrxclKBHPVe+avbj1kaoTRMUK07npsYP6PKcCZwWuqmGhPKRnSAHUsljVD72fzUKTmzSp+EsbIlDZmrvycyGmk9jgLbGVEz1MveTPzP66QmvPYzLpPUoGSLRWEqiInJ7G/S5wqZEWNLKFPc3krYkCrKjE2nZEPwll9eJc2LqudWvfvLSu0mj6MIJ3AK5+DBFdTgDurQAAYDeIZXeHOE8+K8Ox+L1oKTzxzDHzifP4Z+jfE=</latexit><latexit sha1_base64="p4NdPauRcwGeLSvEXoY+yd5ipTI=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8eK9gPaUDbbSbt0swm7G6Gk/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU1HGqGDZYLGLVDqhGwSU2DDcC24lCGgUCW8Hodua3nlBpHstHM07Qj+hA8pAzaqz0MKGTXrniVt05yCrxclKBHPVe+avbj1kaoTRMUK07npsYP6PKcCZwWuqmGhPKRnSAHUsljVD72fzUKTmzSp+EsbIlDZmrvycyGmk9jgLbGVEz1MveTPzP66QmvPYzLpPUoGSLRWEqiInJ7G/S5wqZEWNLKFPc3krYkCrKjE2nZEPwll9eJc2LqudWvfvLSu0mj6MIJ3AK5+DBFdTgDurQAAYDeIZXeHOE8+K8Ox+L1oKTzxzDHzifP4Z+jfE=</latexit><latexit sha1_base64="p4NdPauRcwGeLSvEXoY+yd5ipTI=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8eK9gPaUDbbSbt0swm7G6Gk/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU1HGqGDZYLGLVDqhGwSU2DDcC24lCGgUCW8Hodua3nlBpHstHM07Qj+hA8pAzaqz0MKGTXrniVt05yCrxclKBHPVe+avbj1kaoTRMUK07npsYP6PKcCZwWuqmGhPKRnSAHUsljVD72fzUKTmzSp+EsbIlDZmrvycyGmk9jgLbGVEz1MveTPzP66QmvPYzLpPUoGSLRWEqiInJ7G/S5wqZEWNLKFPc3krYkCrKjE2nZEPwll9eJc2LqudWvfvLSu0mj6MIJ3AK5+DBFdTgDurQAAYDeIZXeHOE8+K8Ox+L1oKTzxzDHzifP4Z+jfE=</latexit><latexit sha1_base64="p4NdPauRcwGeLSvEXoY+yd5ipTI=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8eK9gPaUDbbSbt0swm7G6Gk/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU1HGqGDZYLGLVDqhGwSU2DDcC24lCGgUCW8Hodua3nlBpHstHM07Qj+hA8pAzaqz0MKGTXrniVt05yCrxclKBHPVe+avbj1kaoTRMUK07npsYP6PKcCZwWuqmGhPKRnSAHUsljVD72fzUKTmzSp+EsbIlDZmrvycyGmk9jgLbGVEz1MveTPzP66QmvPYzLpPUoGSLRWEqiInJ7G/S5wqZEWNLKFPc3krYkCrKjE2nZEPwll9eJc2LqudWvfvLSu0mj6MIJ3AK5+DBFdTgDurQAAYDeIZXeHOE8+K8Ox+L1oKTzxzDHzifP4Z+jfE=</latexit>
Figure 15: We repeat Figure 6 for all 10 seeds of the penalty agent of Table 2. Note that the action distribution does not saturate.
5 original tasks 16 additional tasks
Figure 16: Performance profiles as computer by the rliable library of Agarwal et al. (2021). (Left) profiles over the original 5 tasks of Table 3, with 10 seeds for combined++ and 40 seeds for the two other methods. (Right) profiles over the additional 16 tasks of Figure 5, with 10 seeds per method. Our combined agents improve performance, especially for the lowest quantiles. The effect is most dramatic for the 5 original tasks, this is natural since we initially selected these as they suffered from high variance. Also note that the baseline is the recent SOTA DRQv2 agent of Yarats et al. (2021b), which is very competitive – so any improvements to it can be very significant.
B IMPROVEMENTS FOR SPARSE TASKS
In Figure 5, consider the five tasks where the combined agent suffers from the highest standard deviation. Four of these tasks have sparse rewards. This suggests that sparsity is a major driver of variance. To further decrease the variance, we now investigate fixes for four sparse tasks – cartpole swingup sparse, pendulum swingup, reacher hard, and finger turn hard. We will consider three additional modifications which aim to specifically combat reward sparsity:
Avoiding training without rewards. For the sparse tasks, an agent might not see any rewards during e.g. the first 100 episodes. Perhaps training in the absence of any rewards harms the network weights. The fix we consider is simple, do not use any gradient updates before at least one non-zero reward has been observed.
Increasing self-supervised learning. Since the agent might potentially not observe any rewards for 100 episodes, we consider extending the self-supervised learning to ensure some supervisory signal. We simply extend the self-supervised time from 10k steps to the entire training process.
Removing clipped double q-learning. The agent of Yarats et al. (2021a) employs clipped double q-learning (Fujimoto et al., 2018) – i.e. the agent uses two critic networks Q1, Q2 and takes Q = min(Q1, Q2) to avoid overestimation. We hypothesise that the min-operator causes excessive pessimism in sparse environments where reward often are zero. For example, ifQ1 = 0 andQ2 > 0 we have Q = min(Q1, Q2) = 0. Thus, if one Q-network outputs only zeros and the rewards are almost only zeros, the critic target of eq. (1) will be mostly zeros and the other Q-network will be encouraged to output mostly zeros too. Thus, the Q-network could potentially get stuck with mostly outputting zeros, an issue shown in Figure 18. We thus propose to use Qa = avg(Q1, Q2). Through ablation experiments in Table 6, we find that using Qa = avg(Q1, Q2) only for the critic loss (i.e. eq. (1)) but retaining Q = min(Q1, Q2) for the actor update, works the best. We call this strategy asymmetrically clipped double Q-learning.
Results for these experiments are shown in Table 7. We see that decreasing pessimism and increasing self-supervised learning help the most. Increasing self-supervised learning helps the most for cartpole swingup sparse and pendulum swingup. Note that these two tasks were not used in the ablation experiments of Table 2. Additionally, whereas feature learning was not the bottleneck for the baseline agent, it might be the bottleneck for the combined agent. Avoiding training without rewards does not seem to improve performance significantly. We conclude that increased self-supervised learning and asymmetrically clipped double Q-learning improve performance for tasks with sparse rewards. We add these two modifications to the combined agent and call the resulting agent combined++. Its performance across all 21 tasks is shown in Figure 5. It achieves even better results, and its average standard deviation is smaller than the baseline by a factor > 3.

heading: C NORMALIZATION FOR THE CRITIC
text: We now study why penultimate normalization is helpful for the critic network. We consider three agents from Table 2: baseline, actor pnorm, and both pnorm. The both pnorm agent uses penultimate normalization for the critic, whereas the others do not. We plot two additional quantities during early training for the reacher hard task, using 10 new seeds. In Figure 19 we show the average Q-values, measured over training batches, achieved by actions generated by the actor network. We see that the curves are much smoother when normalizing the critic, whereas they oscillate aggressively otherwise. Quickly oscillating Q-values makes it hard for the actor to learn a good policy. We hypothesize that stabilizing the Q-values over time is the primary mechanism by which critic normalization helps. In Figure 20 we show the average absolute difference between Q-values before and after a gradient update, as measured over training batches. We see that normalizing the critic often leads to Q-value changes which are smaller and more stable over time. This in turn likely stabilizes learning as per Figure 19. Whereas the Q-value changes can trivially be made smaller by decreasing the learning rate, changes in Q-values depend non-linearly on the learning rate. Furthermore, too low learning rates often hurt generalization (Li et al., 2019). For increased legibility, the curves are smoothed by averaging with a window of width ten.
baseline
actor pnorm
both pnorm
Q <latexit sha1_base64="klm9NPgBdSUH55vJFmY1NAsBu6k=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GNRDx5bsB/QhrLZTtqlm03c3Qgl9E948aCIV/+ON/+N2zYHbX0w8Hhvhpl5QSK4Nq777RTW1jc2t4rbpZ3dvf2D8uFRS8epYthksYhVJ6AaBZfYNNwI7CQKaRQIbAfj25nffkKleSwfzCRBP6JDyUPOqLFSp3eHwlDS6JcrbtWdg6wSLycVyFHvl796g5ilEUrDBNW667mJ8TOqDGcCp6VeqjGhbEyH2LVU0gi1n83vnZIzqwxIGCtb0pC5+nsio5HWkyiwnRE1I73szcT/vG5qwms/4zJJDUq2WBSmgpiYzJ4nA66QGTGxhDLF7a2EjaiizNiISjYEb/nlVdK6qHpu1WtcVmo3eRxFOIFTOAcPrqAG91CHJjAQ8Ayv8OY8Oi/Ou/OxaC04+cwx/IHz+QNaaY+B</latexit><latexit sha1_base64="klm9NPgBdSUH55vJFmY1NAsBu6k=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GNRDx5bsB/QhrLZTtqlm03c3Qgl9E948aCIV/+ON/+N2zYHbX0w8Hhvhpl5QSK4Nq777RTW1jc2t4rbpZ3dvf2D8uFRS8epYthksYhVJ6AaBZfYNNwI7CQKaRQIbAfj25nffkKleSwfzCRBP6JDyUPOqLFSp3eHwlDS6JcrbtWdg6wSLycVyFHvl796g5ilEUrDBNW667mJ8TOqDGcCp6VeqjGhbEyH2LVU0gi1n83vnZIzqwxIGCtb0pC5+nsio5HWkyiwnRE1I73szcT/vG5qwms/4zJJDUq2WBSmgpiYzJ4nA66QGTGxhDLF7a2EjaiizNiISjYEb/nlVdK6qHpu1WtcVmo3eRxFOIFTOAcPrqAG91CHJjAQ8Ayv8OY8Oi/Ou/OxaC04+cwx/IHz+QNaaY+B</latexit><latexit sha1_base64="klm9NPgBdSUH55vJFmY1NAsBu6k=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GNRDx5bsB/QhrLZTtqlm03c3Qgl9E948aCIV/+ON/+N2zYHbX0w8Hhvhpl5QSK4Nq777RTW1jc2t4rbpZ3dvf2D8uFRS8epYthksYhVJ6AaBZfYNNwI7CQKaRQIbAfj25nffkKleSwfzCRBP6JDyUPOqLFSp3eHwlDS6JcrbtWdg6wSLycVyFHvl796g5ilEUrDBNW667mJ8TOqDGcCp6VeqjGhbEyH2LVU0gi1n83vnZIzqwxIGCtb0pC5+nsio5HWkyiwnRE1I73szcT/vG5qwms/4zJJDUq2WBSmgpiYzJ4nA66QGTGxhDLF7a2EjaiizNiISjYEb/nlVdK6qHpu1WtcVmo3eRxFOIFTOAcPrqAG91CHJjAQ8Ayv8OY8Oi/Ou/OxaC04+cwx/IHz+QNaaY+B</latexit><latexit sha1_base64="klm9NPgBdSUH55vJFmY1NAsBu6k=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GNRDx5bsB/QhrLZTtqlm03c3Qgl9E948aCIV/+ON/+N2zYHbX0w8Hhvhpl5QSK4Nq777RTW1jc2t4rbpZ3dvf2D8uFRS8epYthksYhVJ6AaBZfYNNwI7CQKaRQIbAfj25nffkKleSwfzCRBP6JDyUPOqLFSp3eHwlDS6JcrbtWdg6wSLycVyFHvl796g5ilEUrDBNW667mJ8TOqDGcCp6VeqjGhbEyH2LVU0gi1n83vnZIzqwxIGCtb0pC5+nsio5HWkyiwnRE1I73szcT/vG5qwms/4zJJDUq2WBSmgpiYzJ4nA66QGTGxhDLF7a2EjaiizNiISjYEb/nlVdK6qHpu1WtcVmo3eRxFOIFTOAcPrqAG91CHJjAQ8Ayv8OY8Oi/Ou/OxaC04+cwx/IHz+QNaaY+B</latexit>
Q <latexit sha1_base64="klm9NPgBdSUH55vJFmY1NAsBu6k=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GNRDx5bsB/QhrLZTtqlm03c3Qgl9E948aCIV/+ON/+N2zYHbX0w8Hhvhpl5QSK4Nq777RTW1jc2t4rbpZ3dvf2D8uFRS8epYthksYhVJ6AaBZfYNNwI7CQKaRQIbAfj25nffkKleSwfzCRBP6JDyUPOqLFSp3eHwlDS6JcrbtWdg6wSLycVyFHvl796g5ilEUrDBNW667mJ8TOqDGcCp6VeqjGhbEyH2LVU0gi1n83vnZIzqwxIGCtb0pC5+nsio5HWkyiwnRE1I73szcT/vG5qwms/4zJJDUq2WBSmgpiYzJ4nA66QGTGxhDLF7a2EjaiizNiISjYEb/nlVdK6qHpu1WtcVmo3eRxFOIFTOAcPrqAG91CHJjAQ8Ayv8OY8Oi/Ou/OxaC04+cwx/IHz+QNaaY+B</latexit><latexit sha1_base64="klm9NPgBdSUH55vJFmY1NAsBu6k=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GNRDx5bsB/QhrLZTtqlm03c3Qgl9E948aCIV/+ON/+N2zYHbX0w8Hhvhpl5QSK4Nq777RTW1jc2t4rbpZ3dvf2D8uFRS8epYthksYhVJ6AaBZfYNNwI7CQKaRQIbAfj25nffkKleSwfzCRBP6JDyUPOqLFSp3eHwlDS6JcrbtWdg6wSLycVyFHvl796g5ilEUrDBNW667mJ8TOqDGcCp6VeqjGhbEyH2LVU0gi1n83vnZIzqwxIGCtb0pC5+nsio5HWkyiwnRE1I73szcT/vG5qwms/4zJJDUq2WBSmgpiYzJ4nA66QGTGxhDLF7a2EjaiizNiISjYEb/nlVdK6qHpu1WtcVmo3eRxFOIFTOAcPrqAG91CHJjAQ8Ayv8OY8Oi/Ou/OxaC04+cwx/IHz+QNaaY+B</latexit><latexit sha1_base64="klm9NPgBdSUH55vJFmY1NAsBu6k=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GNRDx5bsB/QhrLZTtqlm03c3Qgl9E948aCIV/+ON/+N2zYHbX0w8Hhvhpl5QSK4Nq777RTW1jc2t4rbpZ3dvf2D8uFRS8epYthksYhVJ6AaBZfYNNwI7CQKaRQIbAfj25nffkKleSwfzCRBP6JDyUPOqLFSp3eHwlDS6JcrbtWdg6wSLycVyFHvl796g5ilEUrDBNW667mJ8TOqDGcCp6VeqjGhbEyH2LVU0gi1n83vnZIzqwxIGCtb0pC5+nsio5HWkyiwnRE1I73szcT/vG5qwms/4zJJDUq2WBSmgpiYzJ4nA66QGTGxhDLF7a2EjaiizNiISjYEb/nlVdK6qHpu1WtcVmo3eRxFOIFTOAcPrqAG91CHJjAQ8Ayv8OY8Oi/Ou/OxaC04+cwx/IHz+QNaaY+B</latexit><latexit sha1_base64="klm9NPgBdSUH55vJFmY1NAsBu6k=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GNRDx5bsB/QhrLZTtqlm03c3Qgl9E948aCIV/+ON/+N2zYHbX0w8Hhvhpl5QSK4Nq777RTW1jc2t4rbpZ3dvf2D8uFRS8epYthksYhVJ6AaBZfYNNwI7CQKaRQIbAfj25nffkKleSwfzCRBP6JDyUPOqLFSp3eHwlDS6JcrbtWdg6wSLycVyFHvl796g5ilEUrDBNW667mJ8TOqDGcCp6VeqjGhbEyH2LVU0gi1n83vnZIzqwxIGCtb0pC5+nsio5HWkyiwnRE1I73szcT/vG5qwms/4zJJDUq2WBSmgpiYzJ4nA66QGTGxhDLF7a2EjaiizNiISjYEb/nlVdK6qHpu1WtcVmo3eRxFOIFTOAcPrqAG91CHJjAQ8Ayv8OY8Oi/Ou/OxaC04+cwx/IHz+QNaaY+B</latexit>
Q <latexit sha1_base64="klm9NPgBdSUH55vJFmY1NAsBu6k=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GNRDx5bsB/QhrLZTtqlm03c3Qgl9E948aCIV/+ON/+N2zYHbX0w8Hhvhpl5QSK4Nq777RTW1jc2t4rbpZ3dvf2D8uFRS8epYthksYhVJ6AaBZfYNNwI7CQKaRQIbAfj25nffkKleSwfzCRBP6JDyUPOqLFSp3eHwlDS6JcrbtWdg6wSLycVyFHvl796g5ilEUrDBNW667mJ8TOqDGcCp6VeqjGhbEyH2LVU0gi1n83vnZIzqwxIGCtb0pC5+nsio5HWkyiwnRE1I73szcT/vG5qwms/4zJJDUq2WBSmgpiYzJ4nA66QGTGxhDLF7a2EjaiizNiISjYEb/nlVdK6qHpu1WtcVmo3eRxFOIFTOAcPrqAG91CHJjAQ8Ayv8OY8Oi/Ou/OxaC04+cwx/IHz+QNaaY+B</latexit><latexit sha1_base64="klm9NPgBdSUH55vJFmY1NAsBu6k=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GNRDx5bsB/QhrLZTtqlm03c3Qgl9E948aCIV/+ON/+N2zYHbX0w8Hhvhpl5QSK4Nq777RTW1jc2t4rbpZ3dvf2D8uFRS8epYthksYhVJ6AaBZfYNNwI7CQKaRQIbAfj25nffkKleSwfzCRBP6JDyUPOqLFSp3eHwlDS6JcrbtWdg6wSLycVyFHvl796g5ilEUrDBNW667mJ8TOqDGcCp6VeqjGhbEyH2LVU0gi1n83vnZIzqwxIGCtb0pC5+nsio5HWkyiwnRE1I73szcT/vG5qwms/4zJJDUq2WBSmgpiYzJ4nA66QGTGxhDLF7a2EjaiizNiISjYEb/nlVdK6qHpu1WtcVmo3eRxFOIFTOAcPrqAG91CHJjAQ8Ayv8OY8Oi/Ou/OxaC04+cwx/IHz+QNaaY+B</latexit><latexit sha1_base64="klm9NPgBdSUH55vJFmY1NAsBu6k=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GNRDx5bsB/QhrLZTtqlm03c3Qgl9E948aCIV/+ON/+N2zYHbX0w8Hhvhpl5QSK4Nq777RTW1jc2t4rbpZ3dvf2D8uFRS8epYthksYhVJ6AaBZfYNNwI7CQKaRQIbAfj25nffkKleSwfzCRBP6JDyUPOqLFSp3eHwlDS6JcrbtWdg6wSLycVyFHvl796g5ilEUrDBNW667mJ8TOqDGcCp6VeqjGhbEyH2LVU0gi1n83vnZIzqwxIGCtb0pC5+nsio5HWkyiwnRE1I73szcT/vG5qwms/4zJJDUq2WBSmgpiYzJ4nA66QGTGxhDLF7a2EjaiizNiISjYEb/nlVdK6qHpu1WtcVmo3eRxFOIFTOAcPrqAG91CHJjAQ8Ayv8OY8Oi/Ou/OxaC04+cwx/IHz+QNaaY+B</latexit><latexit sha1_base64="klm9NPgBdSUH55vJFmY1NAsBu6k=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GNRDx5bsB/QhrLZTtqlm03c3Qgl9E948aCIV/+ON/+N2zYHbX0w8Hhvhpl5QSK4Nq777RTW1jc2t4rbpZ3dvf2D8uFRS8epYthksYhVJ6AaBZfYNNwI7CQKaRQIbAfj25nffkKleSwfzCRBP6JDyUPOqLFSp3eHwlDS6JcrbtWdg6wSLycVyFHvl796g5ilEUrDBNW667mJ8TOqDGcCp6VeqjGhbEyH2LVU0gi1n83vnZIzqwxIGCtb0pC5+nsio5HWkyiwnRE1I73szcT/vG5qwms/4zJJDUq2WBSmgpiYzJ4nA66QGTGxhDLF7a2EjaiizNiISjYEb/nlVdK6qHpu1WtcVmo3eRxFOIFTOAcPrqAG91CHJjAQ8Ayv8OY8Oi/Ou/OxaC04+cwx/IHz+QNaaY+B</latexit>
Q <latexit sha1_base64="klm9NPgBdSUH55vJFmY1NAsBu6k=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GNRDx5bsB/QhrLZTtqlm03c3Qgl9E948aCIV/+ON/+N2zYHbX0w8Hhvhpl5QSK4Nq777RTW1jc2t4rbpZ3dvf2D8uFRS8epYthksYhVJ6AaBZfYNNwI7CQKaRQIbAfj25nffkKleSwfzCRBP6JDyUPOqLFSp3eHwlDS6JcrbtWdg6wSLycVyFHvl796g5ilEUrDBNW667mJ8TOqDGcCp6VeqjGhbEyH2LVU0gi1n83vnZIzqwxIGCtb0pC5+nsio5HWkyiwnRE1I73szcT/vG5qwms/4zJJDUq2WBSmgpiYzJ4nA66QGTGxhDLF7a2EjaiizNiISjYEb/nlVdK6qHpu1WtcVmo3eRxFOIFTOAcPrqAG91CHJjAQ8Ayv8OY8Oi/Ou/OxaC04+cwx/IHz+QNaaY+B</latexit><latexit sha1_base64="klm9NPgBdSUH55vJFmY1NAsBu6k=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GNRDx5bsB/QhrLZTtqlm03c3Qgl9E948aCIV/+ON/+N2zYHbX0w8Hhvhpl5QSK4Nq777RTW1jc2t4rbpZ3dvf2D8uFRS8epYthksYhVJ6AaBZfYNNwI7CQKaRQIbAfj25nffkKleSwfzCRBP6JDyUPOqLFSp3eHwlDS6JcrbtWdg6wSLycVyFHvl796g5ilEUrDBNW667mJ8TOqDGcCp6VeqjGhbEyH2LVU0gi1n83vnZIzqwxIGCtb0pC5+nsio5HWkyiwnRE1I73szcT/vG5qwms/4zJJDUq2WBSmgpiYzJ4nA66QGTGxhDLF7a2EjaiizNiISjYEb/nlVdK6qHpu1WtcVmo3eRxFOIFTOAcPrqAG91CHJjAQ8Ayv8OY8Oi/Ou/OxaC04+cwx/IHz+QNaaY+B</latexit><latexit sha1_base64="klm9NPgBdSUH55vJFmY1NAsBu6k=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GNRDx5bsB/QhrLZTtqlm03c3Qgl9E948aCIV/+ON/+N2zYHbX0w8Hhvhpl5QSK4Nq777RTW1jc2t4rbpZ3dvf2D8uFRS8epYthksYhVJ6AaBZfYNNwI7CQKaRQIbAfj25nffkKleSwfzCRBP6JDyUPOqLFSp3eHwlDS6JcrbtWdg6wSLycVyFHvl796g5ilEUrDBNW667mJ8TOqDGcCp6VeqjGhbEyH2LVU0gi1n83vnZIzqwxIGCtb0pC5+nsio5HWkyiwnRE1I73szcT/vG5qwms/4zJJDUq2WBSmgpiYzJ4nA66QGTGxhDLF7a2EjaiizNiISjYEb/nlVdK6qHpu1WtcVmo3eRxFOIFTOAcPrqAG91CHJjAQ8Ayv8OY8Oi/Ou/OxaC04+cwx/IHz+QNaaY+B</latexit><latexit sha1_base64="klm9NPgBdSUH55vJFmY1NAsBu6k=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GNRDx5bsB/QhrLZTtqlm03c3Qgl9E948aCIV/+ON/+N2zYHbX0w8Hhvhpl5QSK4Nq777RTW1jc2t4rbpZ3dvf2D8uFRS8epYthksYhVJ6AaBZfYNNwI7CQKaRQIbAfj25nffkKleSwfzCRBP6JDyUPOqLFSp3eHwlDS6JcrbtWdg6wSLycVyFHvl796g5ilEUrDBNW667mJ8TOqDGcCp6VeqjGhbEyH2LVU0gi1n83vnZIzqwxIGCtb0pC5+nsio5HWkyiwnRE1I73szcT/vG5qwms/4zJJDUq2WBSmgpiYzJ4nA66QGTGxhDLF7a2EjaiizNiISjYEb/nlVdK6qHpu1WtcVmo3eRxFOIFTOAcPrqAG91CHJjAQ8Ayv8OY8Oi/Ou/OxaC04+cwx/IHz+QNaaY+B</latexit>
Q <latexit sha1_base64="klm9NPgBdSUH55vJFmY1NAsBu6k=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GNRDx5bsB/QhrLZTtqlm03c3Qgl9E948aCIV/+ON/+N2zYHbX0w8Hhvhpl5QSK4Nq777RTW1jc2t4rbpZ3dvf2D8uFRS8epYthksYhVJ6AaBZfYNNwI7CQKaRQIbAfj25nffkKleSwfzCRBP6JDyUPOqLFSp3eHwlDS6JcrbtWdg6wSLycVyFHvl796g5ilEUrDBNW667mJ8TOqDGcCp6VeqjGhbEyH2LVU0gi1n83vnZIzqwxIGCtb0pC5+nsio5HWkyiwnRE1I73szcT/vG5qwms/4zJJDUq2WBSmgpiYzJ4nA66QGTGxhDLF7a2EjaiizNiISjYEb/nlVdK6qHpu1WtcVmo3eRxFOIFTOAcPrqAG91CHJjAQ8Ayv8OY8Oi/Ou/OxaC04+cwx/IHz+QNaaY+B</latexit><latexit sha1_base64="klm9NPgBdSUH55vJFmY1NAsBu6k=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GNRDx5bsB/QhrLZTtqlm03c3Qgl9E948aCIV/+ON/+N2zYHbX0w8Hhvhpl5QSK4Nq777RTW1jc2t4rbpZ3dvf2D8uFRS8epYthksYhVJ6AaBZfYNNwI7CQKaRQIbAfj25nffkKleSwfzCRBP6JDyUPOqLFSp3eHwlDS6JcrbtWdg6wSLycVyFHvl796g5ilEUrDBNW667mJ8TOqDGcCp6VeqjGhbEyH2LVU0gi1n83vnZIzqwxIGCtb0pC5+nsio5HWkyiwnRE1I73szcT/vG5qwms/4zJJDUq2WBSmgpiYzJ4nA66QGTGxhDLF7a2EjaiizNiISjYEb/nlVdK6qHpu1WtcVmo3eRxFOIFTOAcPrqAG91CHJjAQ8Ayv8OY8Oi/Ou/OxaC04+cwx/IHz+QNaaY+B</latexit><latexit sha1_base64="klm9NPgBdSUH55vJFmY1NAsBu6k=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GNRDx5bsB/QhrLZTtqlm03c3Qgl9E948aCIV/+ON/+N2zYHbX0w8Hhvhpl5QSK4Nq777RTW1jc2t4rbpZ3dvf2D8uFRS8epYthksYhVJ6AaBZfYNNwI7CQKaRQIbAfj25nffkKleSwfzCRBP6JDyUPOqLFSp3eHwlDS6JcrbtWdg6wSLycVyFHvl796g5ilEUrDBNW667mJ8TOqDGcCp6VeqjGhbEyH2LVU0gi1n83vnZIzqwxIGCtb0pC5+nsio5HWkyiwnRE1I73szcT/vG5qwms/4zJJDUq2WBSmgpiYzJ4nA66QGTGxhDLF7a2EjaiizNiISjYEb/nlVdK6qHpu1WtcVmo3eRxFOIFTOAcPrqAG91CHJjAQ8Ayv8OY8Oi/Ou/OxaC04+cwx/IHz+QNaaY+B</latexit><latexit sha1_base64="klm9NPgBdSUH55vJFmY1NAsBu6k=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GNRDx5bsB/QhrLZTtqlm03c3Qgl9E948aCIV/+ON/+N2zYHbX0w8Hhvhpl5QSK4Nq777RTW1jc2t4rbpZ3dvf2D8uFRS8epYthksYhVJ6AaBZfYNNwI7CQKaRQIbAfj25nffkKleSwfzCRBP6JDyUPOqLFSp3eHwlDS6JcrbtWdg6wSLycVyFHvl796g5ilEUrDBNW667mJ8TOqDGcCp6VeqjGhbEyH2LVU0gi1n83vnZIzqwxIGCtb0pC5+nsio5HWkyiwnRE1I73szcT/vG5qwms/4zJJDUq2WBSmgpiYzJ4nA66QGTGxhDLF7a2EjaiizNiISjYEb/nlVdK6qHpu1WtcVmo3eRxFOIFTOAcPrqAG91CHJjAQ8Ayv8OY8Oi/Ou/OxaC04+cwx/IHz+QNaaY+B</latexit>
Q <latexit sha1_base64="klm9NPgBdSUH55vJFmY1NAsBu6k=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GNRDx5bsB/QhrLZTtqlm03c3Qgl9E948aCIV/+ON/+N2zYHbX0w8Hhvhpl5QSK4Nq777RTW1jc2t4rbpZ3dvf2D8uFRS8epYthksYhVJ6AaBZfYNNwI7CQKaRQIbAfj25nffkKleSwfzCRBP6JDyUPOqLFSp3eHwlDS6JcrbtWdg6wSLycVyFHvl796g5ilEUrDBNW667mJ8TOqDGcCp6VeqjGhbEyH2LVU0gi1n83vnZIzqwxIGCtb0pC5+nsio5HWkyiwnRE1I73szcT/vG5qwms/4zJJDUq2WBSmgpiYzJ4nA66QGTGxhDLF7a2EjaiizNiISjYEb/nlVdK6qHpu1WtcVmo3eRxFOIFTOAcPrqAG91CHJjAQ8Ayv8OY8Oi/Ou/OxaC04+cwx/IHz+QNaaY+B</latexit><latexit sha1_base64="klm9NPgBdSUH55vJFmY1NAsBu6k=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GNRDx5bsB/QhrLZTtqlm03c3Qgl9E948aCIV/+ON/+N2zYHbX0w8Hhvhpl5QSK4Nq777RTW1jc2t4rbpZ3dvf2D8uFRS8epYthksYhVJ6AaBZfYNNwI7CQKaRQIbAfj25nffkKleSwfzCRBP6JDyUPOqLFSp3eHwlDS6JcrbtWdg6wSLycVyFHvl796g5ilEUrDBNW667mJ8TOqDGcCp6VeqjGhbEyH2LVU0gi1n83vnZIzqwxIGCtb0pC5+nsio5HWkyiwnRE1I73szcT/vG5qwms/4zJJDUq2WBSmgpiYzJ4nA66QGTGxhDLF7a2EjaiizNiISjYEb/nlVdK6qHpu1WtcVmo3eRxFOIFTOAcPrqAG91CHJjAQ8Ayv8OY8Oi/Ou/OxaC04+cwx/IHz+QNaaY+B</latexit><latexit sha1_base64="klm9NPgBdSUH55vJFmY1NAsBu6k=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GNRDx5bsB/QhrLZTtqlm03c3Qgl9E948aCIV/+ON/+N2zYHbX0w8Hhvhpl5QSK4Nq777RTW1jc2t4rbpZ3dvf2D8uFRS8epYthksYhVJ6AaBZfYNNwI7CQKaRQIbAfj25nffkKleSwfzCRBP6JDyUPOqLFSp3eHwlDS6JcrbtWdg6wSLycVyFHvl796g5ilEUrDBNW667mJ8TOqDGcCp6VeqjGhbEyH2LVU0gi1n83vnZIzqwxIGCtb0pC5+nsio5HWkyiwnRE1I73szcT/vG5qwms/4zJJDUq2WBSmgpiYzJ4nA66QGTGxhDLF7a2EjaiizNiISjYEb/nlVdK6qHpu1WtcVmo3eRxFOIFTOAcPrqAG91CHJjAQ8Ayv8OY8Oi/Ou/OxaC04+cwx/IHz+QNaaY+B</latexit><latexit sha1_base64="klm9NPgBdSUH55vJFmY1NAsBu6k=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GNRDx5bsB/QhrLZTtqlm03c3Qgl9E948aCIV/+ON/+N2zYHbX0w8Hhvhpl5QSK4Nq777RTW1jc2t4rbpZ3dvf2D8uFRS8epYthksYhVJ6AaBZfYNNwI7CQKaRQIbAfj25nffkKleSwfzCRBP6JDyUPOqLFSp3eHwlDS6JcrbtWdg6wSLycVyFHvl796g5ilEUrDBNW667mJ8TOqDGcCp6VeqjGhbEyH2LVU0gi1n83vnZIzqwxIGCtb0pC5+nsio5HWkyiwnRE1I73szcT/vG5qwms/4zJJDUq2WBSmgpiYzJ4nA66QGTGxhDLF7a2EjaiizNiISjYEb/nlVdK6qHpu1WtcVmo3eRxFOIFTOAcPrqAG91CHJjAQ8Ayv8OY8Oi/Ou/OxaC04+cwx/IHz+QNaaY+B</latexit>
Figure 20: Average absolute difference ∆Q between Q-values before and after a gradient step. The Q-value changes are generally smaller and more stable over time for the both pnorm agent (which uses penultimate normalization for the critic). This likely stabilizes the Q-values as per Figure 19.

heading: D LEARNING RATE STABILITY
text: To verify that our fixes increase the stability we now consider changing the learning rate. A high learning rate can cause divergence (Li et al., 2019), and learning rate stability can thus serve as a proxy for learning stability. We consider learning rates in the set {1e−4, 3e−4, 1e−3, 3e−3}, where 1e−4 is the default learning rate used by Yarats et al. (2021b). Results for the walker task are shown in Table 8. We see that the baseline algorithm fails for large learning rates, the average rewards drop quickly while variance decreases when no effective policy is learned. We show the same experiment for additional tasks, but only one learning rate, in Table 9. Again, we see that the agent can tolerate much higher learning rates when using our modifications.

heading: E EVALUATION VARIANCE
text: We now consider the variance arising from evaluating RL agents on a finite set of episodes. Similar to the bias-variance decomposition, we will see that the variance here can be divided into one part that depends upon the randomness of the algorithm, and one term that depends on the randomness from the finite set of episodes we evaluate on. We let R be a set of training runs and S be a set of evaluation episodes sampled. The performance is measured by
perf = 1 |R||S| ∑ i∈R ∑ j∈S rij
Let us calculate the variance of this quantity. We will assume that rewards from different runs are independent, and that rewards from the same training run are independent conditioned on that run. We now have
Var ( 1 |R||S| ∑ i∈R ∑ j∈S rij )
The variance of a sum of independent variables is simply the sum of their variances. Thus we have
Var ( 1 |R||S| ∑ i∈R ∑ j∈S rij ) = 1 |R|Var ( 1 |S| ∑ j∈S rij )
Let µ = E[rij ]. We then have
Var ( 1 |S| ∑ j∈S rij ) = Ealg,samples [( 1 |S| ∑ j∈S rij − µ )2]
Here we explicitly calculate the expectation over random training runs (alg) and the random episodes sampled (samples). We can condition the expectation upon the random training run and then integrate over this expectation. Furthermore, let µi = E[rij |i], i.e. the expected value an episode sampled from training run i. We then have
Var ( 1 |S| ∑ j∈S rij ) = Ealg [ Esamples [( 1 |S| ∑ j∈S rij − µ )2∣∣∣∣i]]
= Ealg [ Esamples [( 1 |S| ∑ j∈S rij − µi + µi − µ )2∣∣∣∣i]]
= Ealg [ Esamples [( 1 |S| ∑ j∈S rij − µi )2∣∣∣∣i]
+2Esamples [( 1 |S| ∑ j∈S rij − µi )( µi − µ )∣∣∣∣i]
+Esamples [( µi − µ )2∣∣∣∣i]] The middle part vanishes since µi = E[rij |i]. For the first term, given a fixed training run the terms rij are independent. This gives us
j∈S j∈S
= 1 |S|2Ealg [ Esamples [∑ j∈S ( rij − µi )2∣∣∣∣i] = 1|S|Ealg [ Esamples [( rij − µi )2∣∣∣∣i] We then have
Var(perf) = 1 |R| Ealg[(µalg − µ) 2]︸ ︷︷ ︸
algorithm variance
+ 1 |R||S| Ealg [ Esamples[(r − µalg)2|i] ]︸ ︷︷ ︸ sample variance
(3)
Here µalg = µi. The first term is what we are interested in measuring. The second term can be large if the environment is noisy and we sample too few episodes. The second term is straightforward to estimate: run the algorithm N times and measure the performance over M samples. For each run, calculate the variance over the samples. Then average this variance over all runs.
In Table 10 we estimate these quantities over ten training runs and ten episodes for the baseline algorithm. As we can see, the environments have very different characteristics. Some have high sample variance and some have very low sample variance. It is common to evaluate with ten training seeds and ten episodes. Can we improve the variance by evaluating on more episodes, say 100? Even for the task with the highest sample noise (reacher), the relative improvement in the standard deviation of the sample variance, i.e.,
√( σ2alg + 1
10 σ2sample
) / ( σ2alg + 1
100 σ2sample
) ≈ 1.195 (4)
is no larger than 20 %. Thus, we conclude that only minor gains can come from evaluating on more episodes. Other environments outside the five we consider, however, might benefit more from using more samples.

heading: F EXTENDED RELATED WORK
text: There are plenty of important applications of RL. Examples include RL for medical testing (Bastani et al., 2021), autonomous vehicles (Bellemare et al., 2020), chemistry (Zhou et al., 2019; Liu et al., 2019), robotics (Kober et al., 2013) and communications infrastructure (Liu et al., 2020). Many more applications remain. Many of these applications require a low variance, but low-power inference (Bjorck et al., 2021a) can also be important for e.g. autonomous robots. Below we discuss some papers closely related to ours in detail.
Engstrom et al. (2019) studies how implementation level details affects performance from policy gradient methods. They specifically find that code-level optimizations can have a significant impact on performance, often rivaling that of algorithmic innovations. This insight agrees with our experimental observations that small modifications can significantly influence performance.
Henderson et al. (2018) presents among the first studies of variability and reproducibility in deep reinforcement learning. They demonstrate that many minor details can have an outsized effect on the outcomes – seeds, environment randomness, and even hardware non-determinism. In our work, we have demonstrated how improved implementations can significantly decrease variance – and our improved agent suffers from much smaller variability between seeds.
Islam et al. (2017) studies variance and reproducibility for continuous control with policy gradient methods. They discuss what metrics are suitable for comparing algorithms and highlight how hyperparameter tuning significantly affects outcomes and thus fairness of evaluations. Our results show that variance in continuous control can be significantly decreased with minor fixes, but we do not study variance from hyperparameter tuning which is an important direction.
Mania et al. (2018) demonstrates that random search is in fact a competitive method for finding policies for continuous control tasks. This strategy is computationally cheap, and the authors present a large study of over 100 seeds, demonstrating that the variance is high. RL still suffers from high variance, but it is not clear if random search still is competitive compared to RL agents.
Clary et al. (2018) studies variability for the Atari benchmark and finds large variability. In light of this, it is proposed that post-training performance should be reported as a distribution rather than as a point estimate. Our paper does not study the Atari suite, but this is an important benchmark that can model discrete tasks rather than continuous robot control problems.
Lynnerup et al. (2020) presents a study over variance for evaluating RL agents on real-world robots. It is concluded that documenting hyperparameters and performing rigorous statistical analysis of the results is imperative. We only consider simulated tasks which are easy to reproduce as long as source code is available, reproducibility becomes much harder in the real world.
Colas et al. (2019) discusses hypotheses testing in the context of RL experiments. They make recommendations regarding significance testing and discuss violations of statistical assumptions. Statistical testing has been criticized in later research (Agarwal et al., 2021), and we refrain from using it.
Chan et al. (2020) proposes multiple metrics to measure notations of reliability in reinforcement learning, and presents an open-sources library to help with this. They discuss the distinction between variability during or after training and the use of significance testing. We have refrained from using hypothesis testing as it suffers from some issues (Agarwal et al., 2021).
Jordan et al. (2020) shows that hyperparameters significantly affect the outcome of RL experiments, and proposes an evaluation strategy that takes this into account. To limit the computational footprint, we have used fixed hyperparameters. While we do not study it, hyperparameter stability is very important for practical applications, and developing RL agents with such stability is an outstanding challenge.
Agarwal et al. (2021) studies variance and statistical significance for RL, focusing especially on Atari tasks where they provide experiments over 100 seeds which shows outlier runs. Among other things, they propose using inter-quartile-means and performance profiles for measuring performance. We adopt the use of performance profiles, but since we are especially interested in outliers inter-quartile-mean is not suitable for our purposes.
Fujimoto et al. (2018) discusses how variance can be harmful to Q-learning, and propose several fixes to it, culminating in the TD3 agent. Some of these fixes have become very popular, especially double clipped Q-values. In Appendix B we revisit double clipped Q-values and find that it can hurt performance for sparse tasks.
Anschel et al. (2017) shows that averaging Q-values can improve the stability and improve performance for a DQN agent. This strategy increases the computational footprint, and such ensembling is known to often work well for supervised learning. The paper mainly focuses on Atari tasks, in contrast, we consider continuous control tasks which arguably are closer to real-world applications.
Nikishin et al. (2018), in a similar spirit, demonstrates that averaging weights can improve the stability of learning. The authors demonstrate that this can improve performance on Atari tasks, although only a handful of seeds appears to be used. To the best of our knowledge, this method is not widely adopted in RL, and we do not experimentally study it.
Jia et al. (2020) considers using classical variance reduction methods such as SVRG (Johnson & Zhang, 2013) to improve deep Q-learning. The authors introduce a recursive framework for computing the stochastic gradient estimate and demonstrate that it can improve the performance of DQN. Again, this method is not widely adopted in RL to the best of our knowledge, and we do not empirically study it.
Chen et al. (2018) considers RL as applied to a dynamic environment, motivated by applications to online recommendation. They propose stratified sampling replay, which relies on a prior customer distribution to decrease variance and the use of approximate regret. These ideas improve performance for DQN agents for a tip recommendation system, while this is an important setting, it is far from continuous control.
Greensmith et al. (2004) studies the use of variance reduction methods for policy gradient methods. Specifically, they study the use of baseline methods, which can significantly decrease the variance and often is adopted in deep reinforcement learning. The use of baselines for policy gradient methods is distinct from the setup we consider.
Parisotto et al. (2020) considers the use of transformers for reinforcement learning and finds that naively using them leads to poor performance. To address this, architectural modifications are proposed that lead to more stable training and improve performance. Studying transformer architectures is important for RL, but in this study, we only consider classical CNN and MLP networks.
Kumar et al. (2019) considers stability for off-policy reinforcement learning, focusing on a setting where the off-policy data is fixed and there is no further interaction with the environment. In this setting, it is demonstrated that bootstrapping error is a key driver of instability, a method to address this issue is proposed. Off-policy reinforcement learning is very important for practical applications, but not the focus of our work.
Mao et al. (2019) considers RL where a stochastic exogenous process influences the environment. In such a setting, a gradient policy method with a standard state-dependent baseline suffers from high variance, and the authors consider fixes to this. The setting considered in this work is distinct from the continuous control tasks we use.
Gogianu et al. (2021) studies the use of spectral normalization (Miyato et al., 2018) for improving the performance of DQN for Atari tasks. They find that this method can stabilize learning when applied to the penultimate layer. Due to the similarity with the normalization methods we consider, we consider this strategy in Table 2.
Wang et al. (2020) studies the behavior of a SAC agent when the entropy bonus is removed. They find that the tanh activations saturate which effectively eliminates exploration since SAC adds exploration noise inside the tanh non-linearity. This argument does not apply to DDPG which adds the exploration noise outside the non-linearity. We study the proposed method Table 2 and find that it underperforms, especially for the acrobot task.
Salimans & Kingma (2016) proposes weight normalization, which reparametrizes the weights by decoupling magnitude and direction. They demonstrate that this idea improves performance in DQN and other settings. Despite the paper being well-known, this strategy has not become widely adopted in RL to the best of our knowledge. Similarly, to the best of our knowledge, batch normalization (Ioffe & Szegedy, 2015; Bjorck et al., 2018) is typically preferred in supervised learning.

Review summaries:
This paper gives a systematic evaluation on different factors that may cause the high-variance of actor-critic algorithm on various robotic control problems. They found that the main source of variance is from the numerical instability. The authors then propose four training techniques to mitigate the instability, which significantly reduced the variance.

This paper is an empirical study of the issue of high variance in RL. Learning continuous control from pixels is adopted as the setup. The authors demonstrated that the failure of a few runs is responsible for much of the variance. To look for the source of the variance, they separately studied the impact of the initial and training-time randomization as well as variation in representation learning between runs, none of which were found to be majorly responsible for the variance. The authors hypothesized and isolated the exploding activation of the actions to be the main source of the variance, which cause some runs to saturate beyond salvation and fail at performing. The hypothesis is tested by employing several measures to counter this phenomenon. Employing normalization of the penultimate features turned out to be particularly helpful. 

This paper investigates sources of poor outlier runs in a DDPG-based agent on 5 image-based DM control tasks. The empirical analysis suggests that such outlier runs result from saturating tanh nonlinearities due to large-valued activations in the policy network. To prevent saturation of tanh, the paper proposes and evaluates fixes and shows that combining a bunch of them results in better performance with lower variance.

The paper studies the variance of the DDPG algorithm on continuous control tasks. They identify that the variance in rewards arises in part due to the failure of some runs to learn. They argue that the failure to learn is due to saturating tanh nonlinearities, and suggest two methods to address this problem.