PID-Inspired Inductive Biases for Deep Reinforcement Learning in Partially Observable Control Tasks
Authors: Ian Char
Jeff Schneider
Venue: NeurIPS 2023 poster (2023)
Keywords: Reinforcement Learning
Control
POMDP
TL;DR: We propose a new architecture for encoding history inspired by PID and find that it results in higher-performing, more robust policies.

Abstract: Deep reinforcement learning (RL) has shown immense potential for learning to control systems through data alone. However, one challenge deep RL faces is that the full state of the system is often not observable. When this is the case, the policy needs to leverage the history of observations to infer the current state. At the same time, differences between the training and testing environments makes it critical for the policy not to overfit to the sequence of observations it sees at training time. As such, there is an important balancing act between having the history encoder be flexible enough to extract relevant information, yet be robust to changes in the environment. To strike this balance, we look to the PID controller for inspiration. We assert the PID controller's success shows that only summing and differencing are needed to accumulate information over time for many control tasks. Following this principle, we propose two architectures for encoding history: one that directly uses PID features and another that extends these core ideas and can be used in arbitrary control tasks. When compared with prior approaches, our encoders produce policies that are often more robust and achieve better performance on a variety of tracking tasks. Going beyond tracking tasks, our policies achieve 1.7x better performance on average over previous state-of-the-art methods on a suite of locomotion control tasks.

PID-Inspired Inductive Biases for Deep Reinforcement Learning in Partially Observable Control Tasks
Ian Char
Jeff Schneider
Machine Learning Department Carnegie Mellon University Pittsburgh, PA 15213 ichar0cs. cmu. edu
Machine Learning Department, Robotics Institute Carnegie Mellon University
Pittsburgh, PA 15213 schneide@cs. cmu, edu
Abstract
Deep reinforcement learning (RL) has shown immense potential for learning to control systems through data alone. However, one challenge deep RL faces that the full state of the system is often not observable. When this is the case, the policy needs leverage the history of observations infer the current state. At the same time, differences between the training and testing environments makes it critical for the policy not overfit the sequence of observations it sees training time. As such, there is an important balancing act between having the history encoder be flexible enough to extract relevant information, yet be robust to changes in the environment. To strike this balance, we look to the PID controller for inspiration. We assert the PID controller's success shows that only summing and differencing are needed accumulate information over time for many control tasks. Following this principle, we propose two architectures for encoding history: one that directly uses PID features and another that extends these ideas and can be used in arbitrary control tasks. When compared with prior approaches, our encoders produce policies that often more robust and achieve better performance variety of tracking tasks. Going beyond tracking tasks, our policies achieve 1.7x better performance average previous state-of-the-art methods suite of locomotion control tasks.
Introduction
Deep reinforcement learning (RL) holds great potential for solving complex tasks through data alone, and there have already been exciting applications of RL in video game playing [70], language model tuning [53], and robotic control (]. Despite these successes, there still remain significant challenges in controlling real-world systems that stand in the way of realizing RL's full potential (20 One major hurdle is the issue of partial observability, resulting in a Partially Observable Markov Decision Process (POMDP). In this case, the true state of the system unknown and the policy must leverage its history of observations. Another hurdle stems from the fact that policies are often trained in an imperfect simulator, which likely different from the true environment. Combining these two challenges necessitates striking balance between extracting useful information from the history and avoiding overfitting modelling error. Therefore, introducing the right inductive biases the training procedure is crucial.
The of recurrent network architectures in deep RL for POMDPs
of the initial proposed solutions (29) and remains a prominent approach for control tasks [4] 75, (5)] Theses architectures certainly flexible; however, unclear whether they the best choice for control tasks,
I 'Code available httppp:://gttaabb...///aaaaaaaat
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

especially since they were originally designed with other applications in mind such as natural
language processing.
In contrast with deep RI. methods, the Proporional--ntegr---eeeee Derivvatie (PID) controller remains a cornerstone of modern control systems despite its simplicity and the fact it is over 100 years old [5] PID controllers are single-input single-output (SISO) feedback controllers designed for tracking problems, where the goal is to maintain a signal at a given reference value. The controller adjusts a single actuator based on the weighted sum of three terms: the current error between the signal and its reference, the integral of this error over time, and the temporal derivative of this error. PID controllers are far simpler than recurrent architectures and yet are still able to perform well in SISO tracking problems despite having no model for the system's 's dynamics. We assert that PID's success teaches us that in many cases only two operations are needed for successful control: summing and differencing.
To investigate this assertion, we conduct experiments on a variety of SISO and multi-input multi- output (MIMO) tracking problems using the same featurizations PID controller to encode history. We find that this encoding often achieves superior performance and is significantly more resilient to changes in the dynamics during test time. The biggest shortcoming with this method, however, is that it can only be used for tracking problems. As such, we propose an architecture that is built on the same principles as the PID controller, but is general enough to be applied to arbitrary control problems. Not only does this architecture exhibit similar robustness benefits, but policies trained with achieve an average of 1.7x better performance than previous oftt--of-the-ar methods on suite of locomotion control tasks.
2
Preliminaries
The MDP and POMDP We define the discrete time, infinite horizon Markov Decision Process (MDP) to be the tuple A,, r, T, 7): where S is the state space, A is the action space, r S S A x S - IR is the reward function, T 8 A A(S) is the transition function, To (S8) is the initial state distribution, and y is the discount factor. We use A(S) to denote the space of distributions over S. Importantly, the Markov property holds for the transition function, i.c. the distribution over a next state s' depends only on the current state, s, and current action, a. Knowing previous states and actions does not provide any more information. The objective is to learn a policy : S + (AA) that maximizes the objective (TT) = E |E t=
Dc 7'r(st. az, St+1)], 1, where Su To.
a (st), and 81+1 ~ T(s ot). When learning a policy, it often key to learn a corresponding value function, Q" S x A + R, which outputs the expected discounted returns after playing action a at state and then following v afterwards.
In a Partially Observable Markov Decision Process (POMDP), the observations that the policy receives are not the true states of the process. In control this may happen for variety of reasons such as noisy observations made by sensors, but in this work we specifically focus on the case where aspects of the state space remain unmeasured. In any case, the POMDP is defined as the tuple (S, A, T, To, S2 7), where Ω is the space of possible observations, O: S x A + A(?) is the conditional distribution of seeing an observation, and the rest of the elements of the tuple remain the same as before. The objective remains the same as the MDP, but now the policy and value functions are not allowed access to the state.
Crucially, the Markov property does not hold for observations in the POMDP. That is, where 01 := 01, 02,-.., 0t+1 observations seen times 1 through f + 1, 01:1-1 / o0+1 az A naive solution to this problem is to instead have the policy take in the history of the episode so far. Of course, it is usually infeasible learn a policy that takes in the entire history for long episodes since the space of possible histories grows exponentially with the length of the episode. Instead, one can encode the information into a more compact representation. In particular, one can use an encoder ø which outputs an encoding 2 = 0(01: 01:--1, r1:--1) (note that encoders need not always take in the actions and rewards). Then, the policy and Q-value functions are augmented to take (o =t) and (o,, a, 22), respectively.
Tracking Problems and PID Controllers. We first focus on the tracking problem, in which there are a set of signals that we wish to maintain at given reference values. For example, in espresso machines the temperature of the boiler (i.e. the signal) must be maintained at a constant reference temperature, and controller is used to vary the boiler's on-off time the temperature
2

---

is maintained at that value [4]. Casting tracking problems as discrete time POMDPs, we let (c) are the 1th
Or = a () 2...,, (.1) ((4)) o ......, o.)
(M)) be the observation at time t, where x1(") and o₂
signal and corresponding reference value, respectively. The reward time t is simply the negative error summed across dimensions, i.e.
- MM
(m)
ot [m))
m==1
When dealing with a single-input single-output (SISO) system (with one signal and one actuator that influences the signal), one often uses a Proportional -npportioall-nterra----a -eriivvt-eevv (PID) controller: feedback controller that is often paired with feedforward control. This controller requires no knowledge of the dynamics, and simply sets the action via linear combination of three terms: the error (P), the integral of the error (I), and the derivative of the error (D). When comparing other architectures the PID controller, we will use orange colored text and blue colored text to highlight similarities between the and D terms, respectively. Concretely, the policy corresponding to a discrete-ting PID controller is defined as
(
-ot - (22t))
(1) PID (o.) = -Kpla ((()) + K₁ (1) of (d) + Kn
dt
(1)
where Kp, K,, and KD are scalar values known as gains that must be tuned. PID controllers are designed for SISO control problems, but many real-world systems are multi-input multi-output (MIMO). In the case of MIMO tracking problems, where there are M signals with M corresponding actuators, one can control the system with M separate PID controllers. However, this assumes there is clear breakdown of which actuator influences which signal. Additionally, there are often interactions between the different signals, which the PID controllers do not account for. Beyond tracking problems, it is less clear how use PID controllers without substantial engineering efforts.
3 Methodology
To motivate the following, consider the task of controlling a tokamak: a toroidal device that magneti- cally confines plasma and used for nuclear fusion. Nuclear fusion holds the promise of providing an energy source with few drawbacks and an abundant fuel source. As such, there has recently been a
surge of interest in applying machine learning [3], and especially RL [4 45 71
64 [4] 65
for tokamak control. However, applying deep RL has the same problems as mentioned earlier; the state partially observable since there are aspects of the plasma's 's state that cannot be measured in real time, and the policy must be trained hefore-hand on an imperfect simulator since operation of the actual device is extremely expensive.
How should one choose historical encoder with these challenges in mind? Previous works 50 46
suggest using Long Short Term Memory (LSTM) BII. Gated Recurrent Units 15), or transformers [69]. These architectures have been shown to be powerful tools in natural language processing, where there exist complicated relationships between words and how they are positioned with respect each other. However, do the same complex temporal relationships exist in something like tokamak control? The fact that PID controllers have been successfully applied for feedback control on tokamaks suggests this may not be the case (72 27 In reality, the extra flexibility of these architectures may become a hindrance when deployed on the physical device if they overfit to quirks in the simulator.
In this section, we present two historical encoders that we believe have good inductive biases for control. They are inspired by the PID controller in that they only sum and difference in order to combine information throughout time. Following this, in Section 5] we empirically show the benefits of these encoders on number of control tasks including tokamak control.
The PID Encoder. Under the framework of a policy that uses history encoder, the standard PID controller ① is simply linear policy with an encoder that outputs the tracking error, the integral of the tracking error, and the derivative of the tracking error. This notion can be extended to MIMO problems and arbitrary policy classes, resulting in the PID-Encoder (PIDE). Given input 01 this
encoder outputs 3M dimensional vector consisting of (1/(m)
(m) (m) dtt, and (m) (m))) (i o
Vm = 1,..., M. For SISO problems, policies with this encoder have access To the same information PID controller. However, for MIMO problems the policy has
di
3

---

access to all the information that each PID controller, acting in isolation, would have. Ideally sophisticated policy would coordinate each actuator setting well.
Figure 1: Architecture for GPIDE. The diagram shows how one encoding, 2t. formed. Each the gray, rounded boxes corresponds to one of the heads that makes up GPIDE. Each green box shows function be learned from data, and the orange box shows the weighted summation ous vectors, oi We write the difference in observations in blue text to highlight the part of GPIDE that relates to PID controller's term. Note that qf: and ki only play role this process if head h uses attention; as such, we write these terms in gray text.
Ot
4/-2
f8
76-2
previ-
df
04-1
04-1
8"
g6
7t-1
The Generalized PID Encoder. A shortcoming of PIDE is that it is only applicable tracking problems since operates over tracking error explicitly. more general encoder should instead accumulate information over arbitrary features of each observation. With this in mind, introduce the Generalized- Peiecd--PD----ddd- (GPIDE).
GPIDE consists of number of "heads", each accumulating information about the history in different manner. When there are heads, GPIDE forms history encoding, 22. through the following:
2 f (concatenate(0--1 04- oi-1))
Vi (1, ................................ t) f1 [1..., H) wh E fl..., H
w L (vt.e)
2, = go (concatenate(wr, wf.................. w!
Here, GPIDE is parameterized by 9. For head h, få is linear projection of the previous observation, action, reward, and difference between the current and previous observation to RD,, and thh is weighted summation of these projections. go is a decoder which combines all of the information from the heads. A diagram this process is shown in Figure [ Note that 0 trained along with the policy and Q networks end-to-end with gradient based optimizer.
Notice that the key aspects of the PID controller are present here. The difference in observations is explicitly taken before the linear projection f/. We found that this simple method works best for representing differences when the observations are scalar descriptions of the state (e.g. joint positions). Although we do not consider image observations in this work, we imagine similar technique could be done by taking the differences image encodings. Like the integral term of the PID, also accumulates information over time. In the following, we consider several possibilities for gh and we will refer to these different choices as "head types" throughout this work. We omit the superscript h below for notational convenience.
Summation. Most in line with PID, the projections can be summed, i.e. ((VIx) = - Ui.
Exponential Smoothing. In order to weight recent observations more heavily, exponential smoothing can be used. That is, ((V1.) (1 - + Ei= a(1 a where 0 < 1 is the smoothing parameter. Unlike summation, this head type cannot accumulate information in the same way because it is convex combination.
Attention. Instead of hard-coding weighted summation of the projections, this weighting can be learned through attention 66). Attention is one of the key components of transformers because of its ability to learn relationships between tokens. To implement this, two additional linear functions should be learned that project concatenate(o;-- di-1, O; 0 to DRD. These new projections are referred to as they key and query vectors, denoted as k; and ds respectively. The softmax between their inner products is then used to form the weighting scheme for f1.- We can rewrite the first two

---

steps of GPIDE for a head that uses attention
Vi, kxs qi = fo (cconcatenate(o--111 a Ti-1, 1i-,, - 0i-1)) Vi E (l,...,
qiacki
Wid = E((i:e ki,,, V1+)) = softmax
VD
Clat
Here, Q1:t, k1sts and V1zt are treated as t D dimensional matrices. Since results in convex combination, attention has the capacity to reproduce exponential smoothing but not summation.
To anchor the GPIDE architecture back the PID controller, note that the P, and D terms can be formed exactly. At a high level, this is achieved when f3 simply subtracts the target from the state measurement and when using summation and exponential smoothing heads with a 1. We write down the specific instance of GPIDE that results in these terms in Appendix A. While is trivial for GPIDE reconstruct the P. and D terms, it is less clear how an LSTT GRU would achieve this, especially because of the I term. At the same time. GPIDE is much more flexible than the #iD-representation since altering fo results in different representations each time step and altering the type of head results in different temporal relationships.
4 Related Work
A control task may be partially observable for myriad of reasons including unmeasured state variables [28] (5 29), sensor noise[47], and unmeasured system parameters [76] 54]. When there are unmeasured system parameters, this is usually framed as a meta-reinforremmet learning (MetaRL) 73 problem. This is a specific subelass of POMDPs where there is a collection of MDPs, and each episode, MDP is sampled from this collection. Although these works do consider system parameters varying between episodes, the primary focus the experiments usually tends to be on the multi-task setting (i.e. different reward functions instead of transition functions) [77] [8, 60). We consider not only differing system parameters but also the presence of unmeasured state variables; therefore, the class of POMDPs considered in this paper is broader than the one studied in MetaRL.
Using recurrent networks has long been an approach for tackling POMDPs [29]. and is still a common way do in wide variety of settings [3 70 (50 47 75 11] Moreover implementations
are publicly available both for on-policy 41 30 and off-policy (50
75
I| algorithms, making
recurrent networks to estimate the belief state (37), which is a distribution over the agent's ' true state. However, Ni et al. (50) recently showed that well-implementeed recurrent versions of SAC 26) and TD3 (3] perform competitively with many of these specialized algorithms. In either case, we believe works that estimate the belief state not in conflict with our own since their architectures can be modified to use GPIDE instead of a recurrent unit.
it an easy pick for those wanting quick solution. Some works
32
77
28
18
3
Beyond recurrent networks, there has been surge of interest in applying transformers to reinforce- ment learning [40). However, we were unable to find many instances of transformers being used as history encoders in the online setting, perhaps because of their difficulty to train. Parisotto et al. [55] introduced a new architecture to remedy these difficulties; however, Melo [46] applied transformers to MetaRL and asserted that careful weight initialization is the only thing needed for stability in training. We note that GPIDE with only attention heads is similar to a single multi-heade self-attention block that appears in many transformer architectures; however, show that attention is the least important type of head in GPIDE and often hurts performance (see Section 5.3)
Perhaps closest to our proposed architecture is PEARL 660 which does a multiplicative combination of Gaussian distributions corresponding to each state-action-reward tuplc. However, their algorithm is designed for the MetaRL setting specifically. Additionally, we note that the idea of summations and averaging has been shown to be powerful in prior works. Specifically, Oliva et al. [52) introduced the Statistical Recurrent Unit, alternative architecture to LSTMs and GRUs that leverages moving averages and performs competitively across several supervised learning tasks.
There are many facets of RL where improvements can be made to robustness, and many works focus on altering the training procedure. They use techniques such as optimizing the policy's worst-case performance [59 36 or using variational information bottlenecking (VIB) 4 to limit the information used by the policy 42 33 21 In contrast, our work specifically focuses on how architecture choices of history encoders affect robustness, but we note our developments can be used in conjunctions with
5

---

these other directions, possibly resulting in improved robustness. We perform additional experiments that consider VIB in Appendix [.I]]
Lastly, we note that there is plethora of work interested in the intersection of reinforcement learning
and PID control 35 25
74 12]. These works focus using reinforcement learning tune the coefficients of PID controllers (often in MIMO settings). We view these as important works on how to improve PID control using reinforcement learning; however, we view our own work how to improve deep reinforcement learning by leveraging ideas from PID control.
39
22
5 Experiments
In this section, we experimentally compare PIDE and GPIDE against recurrent and transformer encoders. In particular, we explore the following questions:
How does the performance of policy using PIDE or GPIDE do on tracking problems? In addition, how well can policies adapt to different system parameters and how robust to modelling error are they on these problems? (Section (51)
Going beyond tracking problems, how well does GPIDE perform on higher dimensional locomotion control tasks (Section 5.2
How important is each type of head in GPIDE? (Section 5.3
For the following tracking problems we use the Soft Actor Critic (SAC) 26 algorithm with each of the different methods for encoding observation history. Following Ni et al. 50, we make two separate instantiations of the encoders for the policy and value networks, respectively. Since the tracking problems are relatively simple, we use a small policy network consisting of hidden layer with 24 units; however, we found that we still needed to use a relatively large Q network consisting of 2 hidden layers with 256 units each to solve the problems, All hyperparameters remain fixed across baselines and tracking tasks; only the history encoders change.
For the recurrent encoder, we use a GRU and follow the implementation of Ni et (50 closely. Our transformer encoder closely resembles the GPT2 architecture (58) and it also includes positional encodings for the observation history. For GPIDE, we use H = heads: one summation head, two attention heads, and three exponential smoothing heads (with not optimized, but rather was picked so that all types of heads were included and so that GPIDE has roughly the same amount of parameters methods, we also evaluate the performance of a tuned PID controller. Not only do PID controllers have an incredibly small number of parameters compared to the other RL-based controllers, but the training procedure is also much more straightforward since it can be posed as a black-box optimization over the returns. While there exists many sophisticated extensions of the PID controller (especially in MIMO systems {8), we only consider the vanilla PID controller since we believe serves good reference point. All methods built top of the rlkit library (57). More details about implementations, hyperparameters, and computation can be found in Appendices B C and [
0.25, 0.5, 1.0). This choice was
our GRU baseline. As reference point for these RL
respectively. We also include additional experiments regarding variational information bottlenecking
(VIB) and lookback size ablations in Appendices F.I and F.2 Implementations can be found at https:/ gi thub. www.huu..oommaaaaaaaaaeeeeeeaaae wwwm/maaChaa//PIID
5.1 Tracking Problems
In this subsection we consider a number of tracking problems. For each environment, the observation consists of the current signals, the reference values, and additional information about the last action made. Unless stated otherwise, the reward is as described Section 2 More information about environments can be found in Appendix E To make a fair comparison against PID controls, we choose to only encode the history of observations. For evaluation, we use 100 fixed settings of the environment (each setting consists of targets and system parameters). To avoid overfitting to these 100 settings, we used a separate set of 100 settings and averaged over 3 seeds when developing our methods. We evaluate policies throughout training. but report the average over the last 10% of evaluations as the final returns. We allow each policy to collect one million environment transitions, and all scores are averaged over 5 seeds. Lastly, each table shows scores formed by scaling the returns by the best and worst average returns across all methods in a particular variant of the environment, where scores of 0 and 100 correspond to the worst and best returns respectively.

---

Environment (Train/Test)
PID Controller
GRU
Transformer
PIDE
GPIDE
MSD Fixed/Fixed
0.00 + 3.96
$3.7313.4
85.79 + 1.98
100.00 + 0.66
83.72 : 2.86
MSD Small/Small
0.00 = 5.58
100.00 + 1.59
73.27 + 4.98
75.51 = 1.31
80.21 + 8.39
MSD Fixed/Large
36.58 2.86
0.00 1 3.42
53.70 + 1.71
34.92 1 0.93
29.55 1 2.32
MSD Small/Large
43.52 2.82
87.63 + 2.28
81.44 +: 0.82
53.21 + 1.31
68.03 1: 4.43
MSD Large/Large
45.60 1.71
100.00 + 0.61
92.60 1.49
69.88
+ 0.69
93.03 + 1.27
Average
25.14
74.27
77.36
66.70
70.91
DMSD Fixed/Fixed
24.33 3.97
0.00 8.69
22.05 + 3.58
100.00 1.08
76.23 6.26
DMSD Small/Small
16.17 3.09
0.00 + 7.79
43.74 + 3.70
100.00 + 0.94
86.74 + 3.94
DMSD Fixed/Large
63.59
2.91
0.00 = 2.28
69.84 + 1.13
78.77 + 1.16
63.89 + 2.16
DMSD Small/Large
70.35 1.44
39.26 + 2.37
73.81 + 1.60
88.52 + 0.83
89.66 = 1.33
DMSD Large/T arrg
78.77 1.97
52.01 2.01
84.45 + 1.41
86.90 + 0.18
100.00 - 0.91
Average
50.64
18.25
56.78
90.84
83.30
Total Average
37.89
46.26
67.07
78.77
77.11
Table I: Mass Spring Damper Task Results. The scores presented are averaged over five seeds and we show the standard error for cach score.
Mass Spring Damper Tracking The first tracking task is the control of a classic ID toy physics system in which there is mass attached a wall by spring and damper. The goal is then to apply a force to the mass in order to move it to given reference location. There are three system parameters to consider here: the mass, spring constant, and damping factor. We also consider the substantially more difficult problem in which there are two masses sandwiched between two walls, and the masses are connected the walls and each other by springs and dampers (see Appendix [..T for a diagram of this). Overall there are eight system parameters (three spring constants, three damping factors, and two masses) and two actuators (a force applied to each mass). We refer to the first problem as Mass-Spring- Damper (MSD) and the second problem as Double- -ammeer (DMSD).
Additionally, we test how adaptive these policies are by changing system parameters in MetaRL- type fashion (i.e. for each episode we randomly select system parameters and then fix them for the rest of the episode). Similar to Packer et al. [54]. we train the policies on three versions of the environment: one with no variation in system parameters, one with small amount of variation, and one with a large amount of variation. We evaluate all policies on the version of the environment with large system parameter variation to test generalization capabilities.
Table Table[]show shows the scores achieved for each of the settings. While GRU and transformers seem to do a good job at encoding history for the MSD environment, both are significantly worse on the more complex DMSD task when compared our proposed encoders. This is true especially for GRU, which performs worse than two independent PID controllers for every configuration. Additionally. while seems that GRU can generalize to large amounts of variation in system parameters when a small amount is present, it fails horribly when trained on fixed system parameters. On the other hand, transformers are able to generalize surprisingly well when trained on both fixed system parameters and with small variation. We hypothesize the autoregressive nature of GRU may make it particularly susceptible to overfitting. Comparing PIDE and GPIDE, we see that PIDE tends to shine in the straightforward cases where there is little change in system parameters, whereas GPIDE is able to adapt when there is a large variation in parameters since it has additional capacity.
Navigation Environment To emulate the setting where the policy is trained on an imperfect simulator, we consider an environment in which the agent tasked with moving itself across a surface to a specified 2D target as quickly and efficiently as possible. At every point in time, the agent can apply some force to move itself, but a penalty term proportional to the magnitude of the force is subtracted from the reward. Suppose that we have access to simulator of the environment that is perfect except for the fact that does not model friction between the agent and the surface. We refer to this simulator and the real environment the "No Friction" and "Friction" environment, respectively. In both environments, the mass of the agent is treated as a system parameter that is sampled for each episode; however, the Friction environment has larger range of masses and also randomly samples the coefficient of friction each episode.
Figure 2 shows the average returns recorded during training for both navigation environments and when the policies trained in No Friction are evaluated in Friction. A table of final scores can be found in Appendix G.3) One can see that GPIDE not only achieves the best returns in the environments it was trained in, but is also robust when going from the frictionless environment to the one with friction. On the other hand, PIDE has less capacity and therefore cannot achieve the same results; however, it
7

---

GRU
Transtomer
GPIDE
No Friction - No Friction
Frictin - Friction
Friction Friction
-13.8
-15.0
-22
-13.7
Retumss
-1388
16.2
-26
-164
-13.9
-28
-15.6
-140
-1.8
2
Envirorment Steps
1e5
Environment Steps
1e6
Environment Steps
108
Figure 2: Average Returns for Navigation Environments. The curves show the average over five seeds and the shaded region shows the standard error. For this plot, we allowed for 5x the normal amount budget to allow all methods to converge. We omit the PID controllers from this plot since it gets substantially worse returns.
immediately more robust than the other methods, although it begins to overfit over time. It is also clear that using GRU less sample efficient and less robust to changes in the test environment.
Tokamak Control For last tracking experiment return to tokamak control. In particular, we focus on the DIIIDD tokamak, device operated by General Atomics in San Diego, California. We aim to control two quantities: BN, the normalized ratio between plasma and magnetic pressure, and rotation, i.e. how fast the plasma is spinning around the toroid. These are important quantities track because BN serves as an approximate economic indicator and rotation control of the plasma has been suggested
be key for stability (7 677 [0 61 [6]. The policy has control over the eight neutral beams (24], which are able to inject power and torque by blasting neutrally charged particles into the plasma. Importantly, two of the eight beams can be oriented in the opposite direction from the others, which decouples the total combined power and torque to some extent (see Figure
Illustration of DIIIDD from Above. [14) Each beamline the figure con- tains two independent beams (yellow boxes). The plasma is rolating counter-clockwise and the two beams in the bottom left of the figure are oriented in counter-curee cuurent direction, allowing power and torque to be decoupled. This figure gives a rough idea of beam posi- tioning but is not physically accurate.
Figure 3:
To emulate the sim-to-real training experience, we create simulator based on the equations described in Boyer et al. [9] and Scoville et [63]. This simulator has two major shortcomings: it assumes that certain states of the plasma (c.g. its shape) are fixed for entire episodes, and it assumes that there are no events that cause loss of confinement of the plasma. We make up for part of the former by randomly sampling plasma states each episode. The approximate
Environment (Train/Test)
PID Controller
GRU
Transformer
PIDE
GPIDE
8y-Track Sim/Sim
40.69 + 0.32
100.00
0.20
97.56
0.19
0.00 1.05
98.33 + 0.41
Ty-Track Sim/Real
89.15 + 0.99
40.96 + 5.45
11.91 40.05
0.00 21.04
55.21 = 4.41
8y-Track Real/Real
98.45 1 0.77
98.24 1 0.38
98.74 1 0.29
100.00 + 0.23
99.30 1 0.64
Average
76.10
79.73
78.79
33.33
84.28
By--ot-Trrck Sim/Sim
0.00 + 0.83
99.06 + 01.2
96.22 0.94
67.98 0.50
100.00 0.29
8y Sy-Rot-Tracc Sim/Rcal
83.71 + 2.64
39.76 + 5.84
33.31 + 0.69
0.00 + 8.89
51.00 + 1.92
8y-Rot So--Ro-Tracc Real/Real
92.02 1 0.84
98.34 1 0.52
96.32
0.33
98.21 1. 0.23
100.00 + 0.46
Average
58.58
79.05
75.28
55.40
83.67
Total Average
67.34
79.39
77.03
44.36
83,97
Table 2: Tokamak Control Task Results. The scores presented are averaged over five seeds and we show the standard error for each score.
8

---

Enviromment
PPO-GRU
TD3-GRU
VRM
SAC-Translormer
SAC-GPIDE
HalfCheetah-PP
27.09 + 7.85
85.80 + 5.15
-107.00 1.39
37.00 9.97
82.63 + 3.46
Hopper-P P
19.00 + 5.22
81633 + 8.33
3.5 1.63
59.54 19.64
93.27 13.56
W
1.67 + 4.39
29.08 9.67
-3.89
24.89 = 14.80
96.61 1.60
Ant-P
39.48 +: 3.74
-36,36 1: 3.35
-36.39 0.17
-10.57 + 2.34
66.66 2.94
HalfCheetah-V
19.68 + 11.71
59.03 2.88
90044++2.99
-41.31 26.15
20.39 + 29.60
Hopper-V V
13.86 + 4.80
57.43 8.66
10.08 = 3.51
0.28 8.49
90.98 + 4.28
W
8.12 + 5.43
-4.63 + 1.30
-1.80
-8,21 + 1.31
36.90 + 16.59
Ant-V
1.43 3.26
17.03 6.55
13.41
0.81 + 1.31
18.03 + 5.10
Average
20.04
36.50
-28.67
7.80
63.18
Table 3: PyBullet Task Results. Each score is averaged over four seeds and we report the standard errors. Unlike before, we scale the returns by the returns of an oracle policy (i.e. one which sees position and velocity) and a policy which does not encode any history. For the environment names, "P" and "V" denote only position only velocity the observation, resepctively.
"real" environment addresses these shortcomings by using a data-driven simulator. This approach to adapted version of
simulating has been shown to be relatively accurate [14, 65 [1] and the simulator appearing in Char et al. [14] for our work. This simulator accounts for a greater set of the plasma's state, and the additional information is rich enough that loss of confinement events play a role in the dynamics.
We consider two versions of this task: the first is SISO task where total power is controlled to achieve a 3y target, and the second is a MIMO task where total power and torque is controlled to achieve AN and rotation targets. The results for both of these tasks are shown in Table? 2 Most of the RL techniques are able to do well if tested in the same environment they were trained in; the exception of this is PIDE, which curiously is unable to perform well in the simulator environment. While no reinforcement learning method matches the robustness of PID controller, policies trained with GPIDE fare significantly better.
5.2 PyBullet Locomotion Tasks
Moving past tracking problems, we evaluate GPIDE on the PyBullet [16] benchmark proposed by Han et al. [28] and adapted in Ni et al. (50) The benchmark has four robots: halfcheetah, hopper, walker, and ant. For each of these, either the current position information or velocity information is hidden from the agent. Except for GPIDE and transformer encoder, we use all of the performance
traces given by Ni al. [50]. In addition SAC, they also train using PPO 622 A2C 49], TD3 233. and VRM [28] a variational method that uses recurrent units to estimate the belief state. We reproduce as much of the training and evaluation procedure as possible, including using the same hyperparameters in the SAC algorithm and giving the history encoders access to actions and rewards. For more information see Appendix C.2 3 the performance of GPIDE along with subset of best performing methods (more results can be found in Appendix G.5) These results make clear that GPIDE is powerful in arbitrary control tasks besides tracking since the average score achieved across all tasks is a 73% improvement over TD3-GRU, which we believe is the previous sate--of-the-ar for this benchmark at the time of this work.
Moreover, GPIDE dominates performance for every robot except HalfCheetah. The only setting where GPIDE achieves significantly worse performance is HalfCheetah-V, This setting seemed to cause stability issues in some seeds, lowering the average score. We believe that these stability issues stem from the attention heads, and we found that removing these heads fixed stability issues and resulted in competitive average score (see Appendix G.5).
5.3 GPIDE Ablations
To investigate the role of cach type of head, reran all experiments with three variants of GPIDE: one with six exponential smoothing heads (ES), one with five exponential smoothing heads and
MSD
DMSD
Navigation
By Track
8y-Rot Track PyBullet
ES
+2.69%
-11.14%
-0.11%
+2.57%
+0.29%
+5.81 %
ES Sum
-8.33%
+5.49%
-1.65%
+4.22%
+0.76%
+11 +1100%
Attention
-0.36%
-54.95%
-3.91%
-8.85%
-7.55%
-39.44%
Table GPIDE Ablation Percent Difference for Average Scores. All final scores can be found in Appendix G

---

one summation head (ES + Sum), and one with six attention heads (see Appendix C for details).
We choose these three configurations specifically better understand the roles that attention and summation play.
Table 4 shows the differences in the average scores for each environment. The first notable takeaway that having summation is often im- portant in some of the more complex environ- ments. The other takeaway is that much of the heavy lifting is being done by the exponential smoothing. GPIDE fares far worse when only having attention heads, especially in DMSD and the PyBullet environments.
MSD-Small
Ha HalfCheetah-P
Sequence Length
20
40-
60
80
50
50
Time Step
Time Step
We visualize some of the attention schemes learned by GPIDE for MSD with small vari- ation and HalfCheetah (Figure 4). While the attention scheme learned for MSD could poten- tially be useful since it recalls information from near the beginning of the episode when the most movement is happening. it appears that the at- tention scheme for HalfCheetah is simply a poor reproduction of exponential smoothing, making it redundant and suboptimal. In fact, we found this phenomenon to be true across all attention heads and PyBullet tasks. We believe that the periodicity that appears here is due to the oscillatory nature of the problem and lack of positional encoding (although we found including positional encoding degrades performance).
Figure 4: Averaged Attention Schemes for MSD- Small and HalfCheetah-P Each y-position on the grid corresponds to an amount of history being recorded, and each x-position corresponds a time point in that his- tory. As such, each of the left-most points are the oldest observation in the history, and the diagonals correspond to the most recent observation. The darker the blue, the greater the weight that is assigned to that time point.
6 Discussion
In this work, we introduced the PIDE and GPIDE history encoders to be used for reinforcement learning in partially observable control tasks. Although both are far simpler than prior methods of encoding, they often result in powerful yet robust controllers. We hope that this work inspires the research community to think about how pre-existing control methods can inform architecture choices.
Limitations There are many different ways control task may be partially observable, and we do not believe that our proposed methods are solutions to all of them. For example, we do not think GPIDE is necessarily suited for tasks where the agent needs remember events (e.g. picking up a key to unlock door).
As with any bias, the PID-inspired biases that we propose in this work come at the cost of flexibility. For the experiments considered in this work, this trade off is beneficial and results in better policies. However, it is unclear whether this trade off is always worth making. It is possible that in higher dimensional environments or environments with more complex dynamics that having more flexibility preferable our proposed architecture.
Lastly, some tasks may require the policy to act on images as observations. We are optimistic that PIDE and GPIDE are still useful architectures in this setting, but speculate that this is contingent on training an image encoder that is well-suited for these architectures, and we leave this research direction for future work.
7 Acknowledgements
We would like to thank Conor Igoe for his helpful discussions and advice on this work. We would also like to thank the NeurlPS 2023 reviewers assigned to our paper for their discussion and feedback.
This work was supported in part by US Department of Energy grants under contract numbers DE-SC0021414 and DE-AC02-09CHI 146. This work also supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. DGE1745016 6 and DGE21 Any opinions, findings, and conclusions orrecommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
10

---

References
[1] Joseph Abbate, R Conlin, and Kolemen. Data-driven profile prediction for diiid. Nuclear Fusion, (1(4:04402 2021.
[2] Ananye Agarwal, Ashish Kumar, Jitendra Malik, and Deepak Pathak. Legged locomotion in challenging terrains using egocentric vision. In Conference on Robot Learning, pages 403-415. PMLR, 2023.
[3] Kei Akuzawa, Yusuke Iwasawa, and Yutaka Matsuo. Estimating disentangled belief about hidden state and hidden task for meta-reinforcemmnt learning. In Learning for Dynamics and Control, pages 73-86. PMLR, 2021.
[4] Alexander Alemi, lan Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXivv/6/2.004110 2016.
[5] Lucas Alpi Pid control: Breaking the time barrier, November 2019.
URL
https://wwrz
:tex:=II/
[6] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization, arXiv preprint 1:0iv:667.0644: 2016.
[7] Laszlo Bardoczi, NC Logan, and EJ Strait. Neoclassical tearing mode sceding by nonlinear whre interactions tokamaks. Physical Review Letters, 127(5):05500,, 2021.
[8] Stephen Boyd, Martin Hast, and Karl Johan Åström. Mimo pid tuning via iterated Imi restriction. International Journal of Robust and Nonlinear Control, 26(8):1718-1733 2016.
[9] MD Boyer, KG Erickson, BA Grierson, DC Pace, JT Scoville, J Rauch, BJ Crowley, JR Ferron, SR Haskey, DA Humphreys, al. Feedback control of stored energy and rotation with variable beam energy and perveance on diii-d. Nuclear Fusion, 59(7):076004, 2019.
[10] RJ Buttery, RJ La Haye, P Gohil, GL Jackson, H Reimerdes, EJ Strait, and DIII-D Team. The influence of rotation on the 3 threshold for the neoclassical tearing mode in dii-d. Physics of Plasmas, 15(5):056115, 2008.
[11] Massimo Caccia, Jonas Mueller, Taesup Kim, Laurent Charlin, and Rasool Fakoor. Task- agnostic continual reinforcement learning: In praise of simple baseline. arXiv preprint arXiv:2205. 14495, 2022.
[12] Ignacio Carlucho, Mariano De Paula, and Gerardo G Acosta. An adaptive deep reinforcement learning approach for mimo pid control mobile robots. ISA transactions, 102:280-281 294, 2020.
[13] Ian Char, Youngscog Chung, Willie Neiswanger, Kirthevasan Kandasamy, Andrew O Nelson, Mark Boyer, Egemen Kolemen, and Jeff Schneider. Offline contextual bayesian optimization. Advances in Neural Information Processing Systems, 32, 2019.
[14] Ian Char, Joseph Abbate, László Bardóezi, Mark Boyer, Youngseog Chung, Rory Conlin, Keith Erickson, Viraj Mehta, Nathan Richner, Egemen Kolemen, et Offline model-based reinforcement learning for tokamak control. In Learning for Dynamics and Control Conference, pages 1357-1372. PMLR, 2023.
[15] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcchre. Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using encoder- decoder for statistical machine translation. arXiv preprint arXiv: 1078 2014.
[16] Erwin Coumans and Yunfei Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. hattp:/pyybulet..om org 2016-2021.
[17] Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897): 414-419 2022.

---

[18] Ron Dorfman, Idan Shenfeld, and Aviv Tamar. Offline meta learning of exploration. arXiv preprint arXiv:2008.0233 025988, 2020.
[19] Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, llya Sutskever, and Pieter Abbeel. RL.: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv: 1.02.997 2016.
[20] Gabriel Dulac-Amold, Nir Levine, Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning, 11009)2449-2444 2468, 2021.
[21] Ben Eysenbach, Russ R Salakhutdinov, and Sergey Levine. Robust predictable control. Ad- vances in Neural Information Processing Systems, 34:27813-22 17-88111--72,,,- 2021.
[22] Fumitake Fujii, Akinori Kaneishi, Takafumi Nii, Ryu' ichiro Maenishi, and Soma Tanaka. Self- tuning two degree-of- degree-of-freedom freedomm proportional-intege control system based on reinforcement learning for multiple- input multiple-outpi industrial process that suffers from spatial input coupling, Processes, 9(3):487, 2021.
[23] Scott Fujimoto, Herke Hoof, and David Meger. actor-critic methods. International conference on machine learning, pages 1587-1596. PMLR, 2018.
Addressing function approximation error
[24] BA Grierson, MA Van Zeeland, JT Scoville, B Crowley, Bykov, JM Park, WW Heidbrink, A Nagy, SR Haskey, and D Liu. Testing the dii-d colcounter off-axis neutral beam injected power and ability to balance injected torque. Nuclear Fusion,
2021.
[25] Zhe Guan and Toru Yamamoto. Design of reinforcement learning pid controller. IEEJ Transactions Electrical and Electronic Engineering, 16(10):1354 1360, 2021.
[26] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018
[27] Sang-hee Hahn, YJ Kim, BG Penaflor, JG Bak, H Han, JS Hong, YM Jeon, JH Jeong, M Joung, JW Juhn, et al. Progress and plan of kstar plasma control system upgrade. Fusion Engineering and Design, 1 112:687-691, 2016.
[28] Dongqi Han, Kenji Doya, and Jun Tani. Variational recurrent models for solving partially observable control tasks. arXiv preprint arXiv: 10703, 2019.
[29] Nicolas Heess, Jonathan J Hunt, Timothy P Lillicrap, and David Silver. Memory-based control with recurrent neural networks. arXiv preprint 15Xiv:1512.04455, 2015.
[30] Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines. https 2018.
//github.com/hill cww.hillmohhhceeeccccce.... stssstabbeebbellll
[31] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9()): 1735-1780, 1997.
[32] Maximilian Igl, Luisa Zintgraf, Tuan Anh Le, Frank Wood, and Shimon Whiteson. Deep varia- tional reinforcement learning for pomdps. In International Conference on Machine Learning. pages 2117-2126. PMLR, 2018
[33] Maximilian Igl, Kamil Ciosek, Yingzhen Li, Sebastian Tschiatschek, Cheng Zhang, Sam Devlin, and Katja Hofmann. Generalization in reinforcement learning with selective noise injection and information bottleneck. Advances in newral information processing systems, 32, 2019.
[34] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift, International conference on machine learning, pages 448-456. pmlr, 2015.
12

---

[35] Hozefa Jesawada, Amol Yerudkar, Carmen Del Vecchio, and Navdeep Singh. A model-basel based reinforcement learning approach for pid design. arXiv preprint arXiv:2206.03567 2022.
[36] Yuankun Jiang, Chenglin Li, Wenrui Dai, Junni Zou, and Hongkai Xiong. Monotonic robust policy optimization with model discrepancy. In International Conference on Machine Learning, pages 4951-4960 PMLR, 2021.
[37] Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2)99--134 1998.
[38] Andrej Karpathy, nanogpt, 2022-. URL htapahpps/ggtabb..cc///aaaaaaaaa
[39] Nathan Lawrence, Gregory Stewart, Philip Loewen, Michael G Forbes, Johan U Back- strom, and R Bhushan Gopaluni. Reinforcement learning based design of linear fixed structure controllers. PTAA-,, 53(2):230-235, 2020.
[40] Wenzhe Li, Hao Luo, Zichuan Lin, Chongjie Zhang, Zongqing Lu, and Deheng Ye. A survey on transformers in reinforcement learning. arXiv preprint (.X.xx:3311.330444 2023.
[41] Eric Liang, Richard Liaw, Philipp Moritz, Robert Nishihara, Roy Fox, Ken Goldberg, Joseph Gonzalez, Michael Jordan, and Ion Stoica. Rllib: Abstractions for distributed reinforcement learning. arxiv e-prints, page. arXiv preprint arXiv: 2017.
[42] Xingyu Lu, Kimin Lee, Pieter Abbeel, and Stas Tiomkin. Dynamics generalization via informa- tion bottleneck in deep reinforcement learning. arXiv preprint arXiv:2008.006114 2020.
[43] La Marzocco. A brief history of the pid, October 2015, URL URUJsttee:/hooeeeLaaaaooooo com com/history-of tet-tteeddeee pid/
[44] Viraj Mehta, Biswajit Paria, Jeff Schneider, Stefano Ermon, and Willie Neiswanger. An experimental design perspective on model-based reinforcement learning. arXiv preprint arXi::2112.05244, 2021.
[45] Viraj Mehta, Ian Char, Joseph Abbate, Rory Conlin, Mark D Boyer, Stefano Ermon, Jeff Schneider, and Willie Neiswanger. Exploration via planning for information about the optimal trajectory. arXiv preprint arXiv:2210.0445 2022.
[46] Luckeciano C Melo. Transformers are meta-reinnfrrcemet learners. In International Conference on Machine Learning, pages 15340-15359, PMLR, 2022.
[47] Lingheng Meng, Rob Gorbet, and Dana Kulić. Memory-basee deep reinforcement learning for pomdps. In 202/ IEEE/RS/ International Conference Intelligent Robots and Systems (IROS), pages 5619-5626. IEEE, 2021.
[48] Nicolas Minorsky, Directional stability of automatically steered bodies. Journal of the American Society for Naval Engineers, 34(2):280-309 1922.
[49] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce- ment learning. In International conference on machine learning, pages 1928-1933 PMLR, 2016.
[50] Tianwei Ni, Benjamin Eysenbach, and Ruslan Salakhutdinov. Recurrent model-free RL can be a strong baseline for many POMDPs. In International Conference on Machine Learning, pages 1 16691-16723. PMLR, 2022.
[51] Fernando Nogucira. Bayesian Optimization: Open source constrained global optimization tool for Python, 2014- URL
[52] Junier B Oliva, Bamabás Póezos, and Jeff Schneider. The statistical recurrent unit. In Interna- tional Conference on Machine Learning, pages 2671-2680. PMLR, 2017.
[53] OpenAl. Gpt-4 technical report. arXiv, 2023.
13

---

[54] Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song. Assessing generalization in deep reinforcement learning. arXiv preprint 1810.1222.222. 2018.
[55] Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayaku- mar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, et al. Stabilizing transformers for reinforcement learning. In International conference on machine learning, pages 7487-7498. PMLR, 2020.
[56] PA Politzer, CC Petty, RJ Jayakumar, TC Luce, MR Wade, JC DeBoo, JR Ferron, P Gohil, CT Holcomb, AW Hyatt, et al. Influence of toroidal rotation on transport and stability in hybrid scenario plasmas in diii-. Nuclear Fusion, 48(7):07501,, 2008
[57] Vitchyr Pong and Ashvin Nair. rikit. http:////gccubb..comaaaaaleeeeee
2018-
[58] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
[59] Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine. Epopt: Learn- ing robust neural network policies using model ensembles. arXiv preprint arXiv:6610001283, 2016.
[60] Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In International conference on muchine learning, pages 5331-5340. PMLR, 2019.
[61] H Reimerdes, AM Garofalo, GL Jackson, M Okabayashi, EJ Strait, Ming-Sheng Chu, Y In, RJ La Haye, MJ Lanetot, YQ Liu, et al. Reduced critical rotation for resistive-wall mode stabilization in a near-axisymmetric configuration. Physical review letters, 98(5):05001,, 2007.
[62] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1777v:7777.06347, 2017.
[63] JT Scoville, DA Humphreys, JR Ferron, and P Gohil. Simultaneous feedhack control of plasma rotation and stored energy on the diii-d tokamak. Fusion engineering and design, 82(5-14): 1045-1050, 2007.
[64] J Seo, Y-S Na, B Kim, CY Lee, MS Park. SJ Park, and YH Lee. Development of an operation trajectory design algorithm for control of multiple Od parameters using deep reinforcement learning in kstar. Nuclear Fusion, 62(8) (2880406 2022.
[65] Jaemin Seo, Y-S Na, B Kim, CY Lee, MS Park, SJ Park, and YH Lee. Feedforward beta control in the kstar tokamak by deep reinforcement learning. Nuclear Fusion, (10)) 2021.
[66] Open Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michael Mathieu, et al. Open- ended learning leads to generally capable agents. arXiv preprint arXiv:2107.1288 2021.
[67] B Tobias, M Chen, IGJ Classen, CW Domier, R Fitzpatrick, BA Grierson, NC Luhmann Jr, CM Muscatello, M Okabayashi, KEJ Olofsson, et al. Rotation profile flattening and toroidal flow shear reversal due to the coupling of magnetic islands in tokamaks. Physics of Plasmas, 23 (5):056107, 2016.
[68] ITER Physics Expert Group on Confinement Transport, ITER Physics Expert Group on Confinement Modelling Database, and ITER Physics Basis Editors. Chapter 2: Plasma confinement and transport. Nuclear Fusion, 39(12)2175-2249, 1999.
[69] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[70] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Jun- young Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782)3350-333 2019.
14

---

[71] T Wakatsuki, T Suzuki, N Oyama, and N Hayashi. Ion temperature gradient control using reinforcement learning technique. Nuclear Fusion. 61(4):04603,, 2021.
[72] ML Walker, DA Humphreys, JA Leuer, JR Ferron, and BG Penaflor. Implementation of model-based multivariable control on diiid. GA-A23468, 2000.
[73] Jane X Wang. Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z. Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. 2016.
arXiv preprint
[74] Junhang Wang, Jianing Zhu, Chao Zou, Linlin Ou, and Xinyi Yu. Robust adaptive pid control based on reinforcement learning for mimo nonlinear six-joint manipulator. In 2021 China Automation Congress (CAC), pages 2633-2638. IEEE, 2021.
[75] Zhihan Yang and Hai Huu Nguyen. Recurrent off-policy baselines for memory continuous control. In Deep RI. Workshop NeurlPS 2021, 2021.
[76] Wenhao Yu, Jie Tan, C Karen Liu, and Greg Turk. Preparing for the unknown: Learning universal policy with online system identification. arXiv preprint arXiv:1702.02453, 2017.
[77] Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta- learning. arXiv preprint 19Xiv:1910.0830 2019.
15

---

Appendices
Contents
Introduction
2
Preliminaries
3
Methodology
4
Related Work
5
Experiments
5.1
Tracking Problems
6
5.2
PyBullet Locomotion Tasks
5.3
GPIDE Ablations
6
Discussion
10
7
Acknowledgements
10
Appendices
16
A Additional GPIDE Details
17
A.I Forming the P. I, and D Features with GPIDE
17
Implementation Details
17
B.1 GPIDE Implementation Details
18
2 Recurrent and Transformer Bascline Detail
18
PID Baseline
18
Hyperparameters
19
Hyperparamters for Tracking Tasks
19
Hyperparameters for PyBullet Task
19
Hyperparameters for Ablations
19
Computation Details
20
Environment Descriptions
20
Mass Spring Damper
20
Navigation Environment
21
Tokamak Control Environment
21
Additional Experiments
23
Experiments Using VIB GRU
23
16

---

F.2
Lookback Size Ablations
24
Further Results
24
Evaluation Procedure
25
G.2
MSD and DMSD Results
25
G.3
Navigation Results
28
4
Tokamak Control Results
30
G.5
PyBullet Results
33
G.6
Attention Scheme Visualizations
36
A
Additional GPIDE Details
A.1 Forming the P, and Features with GPIDE
As mentioned in Section the P. I, and D features associated with a PID controller can be easily formed with GPIDE, and in this Appendix we show this concretely, Assume that the observations take the form or = (2(11) ...... (M)
(M1) and assume that the actions and rewards are not given to GPIDE for simplicity. Then the linear projection for each head, f takes in (M)
(1)
o ....,0
T (1)
t--..... 2 (M) -........................... o--1 each time step t. We use a GPIDE architecture with three heads and where go is the identity function. The linear projections for each head is as follows:
(1)
(x (MM)
..(21 2 (o/ (((1)) (a, a (r-1)..... (o(Ad)
(M)
(xt (1) - 2 (-()) + 1 - (0, ( + +0,-1
fd (00-1, - 01-1) = - (0 --12)=
0000-1,0-00-000
(I( (M)) (M) (a/ (M) (M)
(x( x(_)) - (ot
(1) -0
f (0 o--1)
(M) (MM) (M) (M) [x
Note that fJ and f3 form the current eтror time & (i.e. the P term), and f forms the change in error (i.e. the term). For accumulation strategies, using exponential smoothing with a for the first and third heads and summation head for the second head will recover the P. and D terms for heads 1, 2, and respectively. Note that the above assumes dt 1, but the linear projections can be adjusted to take different di values into account.
B Implementation Details
Code Release All code for implementations are provided in the supplemental material along with instructions for how run experiments. The code can also be found //ttps://gitae com/ TanChar/GPIDE The only experiment that cannot be run are the "real" cases for tokamak control.
Architecture We use the same general architecture for each of the RL methods in this paper (see Figure 5). Each input the history encoders, policy functions, and Q-value functions have corresponding encoders. This setup closely follows what was done in Ni et al. [50]. The encoders are simply linear projections; however, the case of our GRU history encoder we linear projections followed by ReL.U activation (as done in Ni et al. needs to take in history encoding, 21, since int includes the current observation, we found it essential for the current observation to be passed in independently and have its own encoder.
(0 Although hypothetically the policy only
17

---

π
2t
GPIDE Transformer
GPIDE/ Transformer
Observation
Osservtion Encoder
Reward Encodar
Trer
Enooder
Encoder
Ensoder
Encoder
Encoder
Encoder
Encoder
Encocer
01-1
d4-1
F/-1
0
041
(Oty at)
O-11
at11
P--1
04
04-1
l...,
Figure 5: General Policy and Function Architectures. This architecture heavily inspired by Ni al. [50]. The gray box shows history encoder modules, and this the only thing that changes between baselinc methods in the tracking problems. Note that there are two encoders: one for the policy function and one for the Q value function. The purple boxes show the input encoders, and hyperaparmeters for these can be found in Table We found the shortcut encoders to be essential to good performance, The architecture when using GRU nearly identical; however, there is "Transition Encoder" since Ni al. [0 encodes (o,, 1i-1, for each time step instead.
B.1 GPIDE Implementation Details
In addition to what is mentioned in Section g found that there several choices that helped with training. First, there may be some scaling issues because o₂ O--1 may be small or the result of summation type heads may result in large encodings. To account for this, use batch normalization layers 34 before each input encoding and after each gh.
There are very few nonlinear components of GPIDE. The only one that remains constant across all experiments that a tanh activation used for the final output of the encoder. For tracking tasks, the decoder ge has I hidden layer with 64 units and uses ReLU activation function. For PyBullet tasks, go is a linear function.
B.2 Recurrent and Transformer Baseline Details
Recurrent Encoder. For the recurrent encoder, we tried to match as many details Ni et al. (50 as possible. We double checked our implementation against theirs and confirmed that achieves similar performance.
Transformer Encoder. We follow the GPT2 architecture (58 for inspiration, and particularly the code provided in Karpathy (38]. In particular, we use number of multi-headed self-attention blocks sequence with residual connections. We use layer normalization (6) before multi-headed attention and out projections; however, we do not use dropout. The out projection for each self-attention block has one hidden layer with four times the number of units as the embedding dimension. Although Melo [46] suggests using T-Fixup weight initialization, we found that more reliably high performance was achieved with the weight initialization of Radford et [58] Lastly, we used the same representation for the history as GPIDE, i.e. 40--1,4-1,,2 Oz ot-1), since it results in better performance.
B.3 PID Baseline
To tune our PID baseline, we used Bayesian Optimization over the three (for SISO) or six (for MIMO) dimensional space. Specifically we use the library provided by Nogueira [11] The output of the blackbox optimization is the average over 100 different settings (independent from the 100 settings used testing). We allow the optimization procedure to collect as many samples as the RI. methods. The final performance reported uses the PID controller with the best gains found during the
18

---

optimization procedure. The bounds for each of the tracking tasks were eyeballed to be appropriate, which potentially preferably skews performance.
C Hyperparameters
Because of resource restrictions, we were unable to do full hyperparameter tuning for each benchmark presented in this paper. Instead, we focused on ensuring that all history encoding methods were roughly comparable, e.g. dimension of encoding, number of parameters, etc. Tables 5 show selected hypperaarametes, and the following subsections describe how important subset these hyperparameters were picked. Any tuning that was done was over three seeds using 100 fixed settings (different from the 100 settings used for testing).
Task Type
Learning Rate
Batch Size
Discount Factor
Policy Network
Q Network
Path Length Encoding
Tracking
3e-d
32 (256 for PIDB)
0.95
[24]
[256, 256]
100
PyBullet
3e-1
32 (256 for PIDE)
0.99
[256. 256]
[256. 256]
64
Table SAC Hyperparameters. The "Path Length Encoding" is the amount of history each encoder gets to
observe besides PIDE which, because of the nature of it, uses the entire episode.
Observation
Action
Reward
Transition
Policy Shorteut
Shorteut
History Encoding
GPIDE (Tracking)
N/A
N/A
8
64
64
GRU (Trucking)
N/A
N/A
N/A
8
64
64
Transformer (Tracking)
N/A
N/A
16
64
GPIDE (PyHullet) Transformer (PyBullet)
16
64
128
48
16
16
48
64
128
Table 6: Dimension for the Input Encoders and Final History Encoding, The input encoders correspond to
the output dimensions of the purple boxes in Figure By "History Encoding" size mean the dimension of
Task Type
D
9o Hidden Size
Tracking
16
[64]
PyBullet 32
Table 7: GPIDE Specifie Hyperparamters. Recall that 1) corresponds the output dimension of fo. Empty brackets hidden size means that go a linear function.
C.1 Hyperparamters for Tracking Tasks
For tracking tasks, we tried using a history encoding size of 32 and 64 for GRU, and we found that performance better with 64. This is surprising since PIDE can perform well in these environments even though its history encoding is much smaller (3 or dimensional). To make it fair comparison, we set the history encoding dimension for GPIDE and transformer to be 64 as well. We use one layer for GRU. For the transformer-specifie hyperparameters we choose half of what appears in the PyBullet tasks.
C.2 Hyperparameters for PyBullet Task
For the PyBullet tasks, we simply tried to emulate most the hyperparameters found in Ni et al. [50]. For the transformer, However, we found that, unlike the tracking tasks, positional encoding hurts performance. As such, do not include it for PyBullet experiments.
choose to similar hyperparameters to those found in Melo [46].
C.3 Hyperparameters for Ablations
For the ablations of GPIDE, we use a = 0.01, 0.1, 0.25, 0.5, 0.9, 1.0 for the smoothing parameters when only exponential smoothing is used. When using exponential smoothing and summation, the 0.01 head is replaced with summation head. The attention version of GPIDE replaces all six of these heads with attention.
19

---

Task Type
Number of Layers
Number of Heads
Embedding Size per Head
Tracking PyBullet
4
8
4
8
16
Table 8: Transformer Specific Hyperparamters
Encoder
SISO Tracking
MIMO Tracking (2D)
PyBullet
Transformer
25,542
25,644
793,868-795.022
GRU
14,240
14,264
74,816-75,248
GPIDE
13,228
13,288
75,296-76,486
GPIDE-ES
12,204
12,264
50,720-51,911
GPIDE-ESS
12,204
12,264
50,720-51.00
51,910
GPIDE- Attentionn
15,276
15,336
19,872-101,062
Table 9: Number of Parameters in History Encoder Modules. The number of parameters corresponds
the gray boxes Figure 5 The difference in SISO vs MIMO and the PyBullet tasks due to the different observation and action space dimensionalities.
D Computation Details
We used an internal cluster of machines to run these experiments. We mostly leveraged Nvidia Titan X GPUs for this, but also used a few Nvidia GTX 1080s. It is difficult to get an accurate estimate of run time since job loads vary drastically on our cluster from other users. However, train single policy on DMSD to completion (I million transitions collected, 1,000 epochs) using PIDE takes roughly 4.5 hours, using GPIDE takes roughly 17.25 hours, using a GRU takes roughly 14.5 hours, and using transformer takes roughly 21 hours. This is similar for other tracking tasks. For PyBullet tasks, using GPIDE took roughly 43.2 hours and using a transformer took roughly 64.2 hours. We note that implementation of GPIDE is somewhat naive and could be vastly improved. In particular, for exponential smoothing and summation heads, Wy can be cached to save on compute, which is not being done currently. This is big advantage in efficiency that GPIDE (especially one without attention heads) has over transformers.
E
Environment Descriptions
E.1
Mass Spring Damper
00
m
(9)
MSD
DMSD
Figure 6: Diagram of the Mass Spring Damper Environments. The diagram on the left the Mass Spring Damper (MSD) environment, and the diagram the right shows the Double Mass Spring Damper (DMSD) environment. the diagram, we have labelled the system parameters and the parts of the observation. The dotted line shows where the center mass is measured with respect to this point.
mass located with force applied, and the current position of
For both MSD and DMSD, the observations include the current mass position(s), the target reference position(s), and the last action played. Fach episode lasts for 100 time steps. For all RL methods, the action is difference in force applied the mass, but for the PID the action is simply the force
20

---

System Parameter
Fixed
Small
Large
Damping Constant
u(4.0, 4.0)
5(355
U(2.0, 10.0)
Spring Constant
2(2.0,2.0)
U(1.75,3.0
u(0.5, 6.0)
Mass
4(20.0, 20.0)
u(17.5,40.)
4(10.0, 100.0)
Table 10: MSD and DMSD System Parameter Distributions. Each episode system parameters uniformly random drawn from these bounds.
System Parameter
No Friction
Friction
Total Friction
u(0.0, 0.0)
u0005, 0.25)
Static Friction (Proportion)
U(0.0, 0.0)
(0.25, 0.75)
Mass
4(15.0, 25.0)
u(500, 35.0)
Table 11: Navigation System Parameter Distributions. Each episode system parameters are uniformly random drawn from these bounds. The static friction parameter drawn is the proportion of the total friction that static friction.
to be applied to the mass at that time. The force is bounded between -00 and 10 N for MSD and -30 and 30 N for DMSD. Each episode, system parameters are drawn from uniform distribution with bounds shown in Table 10 (they are the same for both MSD and DMSD). Targets are drawn to uniformly at random to be
1.5 m offset from the masses' resting positions.
E.2 Navigation Environment
Like the MSD and DMSD environments, the navigation experiment lasts 100 time steps each episode. Additionally, the observation includes position signal, target locations, and the last action. For all methods we set the action to be the change in force, and the total amount of force is bounded between -10 and 10 N. The penalty the reward is equal to 0.01 times the magnitude of the change in force. In addition, the maximum magnitude of the velocity for the agent is bounded by 1 1.0m/s. The agent always starts the location (0, 0), and the target is picked uniformly random to be within box of length 10 centered around the origin.
Every episode, the mass, kinetic friction coefficient, and static friction coefficient is sampled, The friction is sampled by first sampling the total amount friction the system, and then sampling what proportion of the total friction is static friction. All distributions for the system parameters are uniform, and we show the bounds in Table
E.3 Tokamak Control Environment
Simulator Our simulator version of the tokamak control is inspired by equations used by Boyer al. [9] Scoville al. [63]. In particular, we use the following relations for stored energy, E, and rotation, Uroe:
E
E
P
TF
Uoot
Croe
Tm
where P is the total power, T is the total torque, TE the energy confinement time, Tm is the momentum confinement time, and Cin is quantity relying on the ion density and major radius of the plasma. We treat Tm and Cra constants with values of 0.1 and 80.0 respectively.
We base the energy confinement time off of the ITERH-98 scaling [68]. This uses many measurements of the plasma, but we focus on subset of these and treat the rest as constants, In particular,
Cg1 0.95
0.95 0.15 p 0.69
where Cg is a constant value we set to be 200, I is the plasma current, and B is the toroidal magnetic field. To relate the stored energy to BN use the rough approximation
aB
NN C3
E
21

---

Minor Radius (m)
Plasma Current (MA)
Toroidal Magnetic Field (T)
(0,5589, 0.02)
(1e6,, 1e5)
(2.75 0.1)
Table 12:
Tokamak Control Simulator Distributions.
8
6
Total Torque (Nm)
4
2
2000
4000
6000
8000
10000
12000
14000
Total Power (kW)
Figure Power and Torque Bounds. The region outlined in blue shows the possible power-torque configurations. The dots show possible requests, and the corresponding red X marks show the actual achieved power-torque setting.
where C3 is a constant we set to be 5, and a is the minor radius of the plasma. For I, and B, we sample these from the distribution described in Table 12 for each episode. Lastly, we add momentum to the stored energy. That is, the stored energy derivative time f, Fr, is
E. + 0.5.,
E, = 0.5 Pt
The actions for control methods is the amount of change for power and torque. Because the total amount of power and torque injected rely on the beams, they are not totally disentangled. In Figure[7] we show the bounds for the action space. Furthermore, we bound the amount that power and torque can be changed by roughly 40MW/ and 35.Nm/s. respectively. Each step is 0.025 seconds.
Each episode lasts for 100 increments of 0.025 seconds. The observations are the current By and rotation values, their reference values, and the current power and torque settings. We make the initial 3y and rotation relatively small in order simulate the plasma ramping up. We let the 8N and rotation targets be distributed as (1.75, 2.75) and ((2500,5000) rad/s, respectively.
"Real" For the real versions of the tokamak control experiments, most the previous (such as action bounds and target distributions) stays the same. This data-driven simulator is based on the one from Char al. 114, and we refer the reader there for more details. That being said, there are some differences between the architecture presented there and the one used in this work. Our network is recurrent network that uses GRU, has four hidden layers with 512 units each, and outputs the mean and log variance of normal distribution describing how 3y and rotation will change. In addition to power and torque, it takes in measurements for the plasma current, toroidal magnetic field, nirms (a measurement related to the plasma' stability), and 13 other actuator requests for gas control and plasma shaping. addition sampling from the normal distribution outputted by the network, we train ensemble of ten networks, and ensemble member is selected every episode. We use five of
22

---

these models during training and the other five during testing. Along with an ensemble member being sampled each episode, we also sample a historical run, which determines the starting conditions of the plasma and how the other inputs to the neural network which are not modelled evolve over time. Recall that 100 fixed settings are used to evaluate the policy every epoch of training. In this case, a setting consists of targets, an ensemble member, and historical run.
F Additional Experiments
F.1
Experiments Using VIB + GRU
As shown in this work, using GRU for history encoder often results in a policy that is ill-equipped to handle changes in the dynamics not seen at train time. One may wonder whether using other robust RL techniques is able to mask this inadequacy of GRU. To test this, we look at adding Variational Information Bottlenecking ( (VIB) to our GRU baseline [4]. Previous works applying this concept to
RL usually do not consider the same class of POMDPs as us 42 33; however, Eysenbach et al. [21]
does have baseline that uses VIB with a recurrent policy.
To use VIB with RL, we alter the policy network so that it encodes input to a latent random variable, and the decodes into an action. Following the notation of Lu et al. 42), let this latent random variable be E and the random variable representing the input of the network be S. The goal is learn a policy that maximizes (T) subject to I (Z, S) S. Ic.. where ((2 S) is the mutual information between Z and S, and Ic some given threshold. In practice, we optimize the Lagrangian. Where , is a Lagrangian multiplier, p P (S) is the conditional density of Z outputted by the encoder, and q Z is the prior, the penalizer 5 [DKL standard multivarite normal.
Like other works, we assume that q( (Z) is a
We alter our GRU baseline for tracking tasks so that the policy uses VIB. This is not entirely straightforward since our policy network is already quite small. We choose to keep as close to original policy architecture as possible and set the dimension of the latent variable, Z, be 24. Note that this change has no affect on the history encoder; this only affects the policy network. For our experiments, we set / 0.1, but we note that we may be able to achieve better performance through more careful tuning or annealing of B.
In any case, we do see that VIB helps with robustness in many instances (see Table 13 However, the cases where there are improvements are instances where the GRU policy already did a good job at generalizing the test environment. These are primarily the MSD and DMSD environments where the system parameters drawn during training time are simply a subset of those drawn during testing time (interestingly, this notion of dynamics generalization matches the set up of the experiments presented in Lu et al. (42). Surprisingly, in the navigation and tokamak control experiments, where there are more complex differences between the train and test environments, VIB can sometimes hurt the final performance.
PTD Controller
GRU+
Transfommr
PIDE
GPIDE
MSD Fixed/ / Fixed
-6.
0.02
5.76
-5.73 0.01
-5.75 0.01
-5.69 +: 0.00
-5.76 + 0.01
MSD Fixed/ Large
11.39
0.11
-1.50
-10.87 0.05
-1.44
-1.61 0.07
MSD / Small
-7.99 = 0:03
-7.02
-7.01 0.00
-7.15
+ 0.02
-7.14-0011
-7.12+0.04
MSD Small / Large
11.18 0.09
9.82 0.07
99.57 0.003
10.01
+: 0.00
10.88 0.04
10.43 0.14
DMSD Fixed Tixed
15.33 a 0.14
16.20
1 0.31
15.83
0.228
15,41 1 0.13
-12.64 1 0.04
13.40 1. 0.22
DMSD Fixed/ /
-27.59 1 0.44
37.21
0.35
-35344
0.28
-28.16
1 0.17
-2.229
+ 0.18
27.54
1: 0.33
DMSD Small/ Small
21.78
22.40
0.34
22.51
0.24
20.56
0.16
18.09 0.04
18.67
0.17
DMSD Small Largge
26.57 0.22
31.27
1.366
30.93
0.34
-6.044
1: 0.24
23.82 - 0.13
23.65 :: 0.20
Sim/ Sim
-17.23
13.82
14.69
133668
-13.74
13.65 4: 0.00
Nav Sim / Real
-23.87 0.29
-29.85
39.57
1021
-2181+0111
-20.37 + 0.08
-21.23 0.12
Dx Sim Sim
-8.09 0.00
-7.19 0.00
0.01
0.00
-8.71
-7.21 0.00
ON Sim / Real
166.41 = 0.30
-31
-31.21 + 1.67
-32.19 + 1.19
- -31.49 3.06
-43.78 6-46
- 26.83 + 1.36
Bg Rotation Sim / Sim
27..56
3.06++0.08 0.08
18.53 0.02
18.61
0.12
18.79 0.00
-21.36
0.05
-18.45 + 0.03
9a Rotation Sim Real
30.000 0.95
-15.91 2.10
11.21 1 1.33
18:23:0.00
-00.23
3,20
-11.86 0.69
Averuge
18.33
-20.12
-21.14
-1877
19.58
-17.51
Table 13: Tracking Experiments with GRU We use green and red text to highlight significant
improvements and deteriorations in performance over vanilla GRU. We only highlight a subset of configurations since we are focused the robustness properties. This table shows average (unnormalized) returns.
23

---

F22 Lookback Size Ablations
To better understand the role of the maximum lookback size (i.e. the amount of history used to form the encoding) of GPIDE, we repeat the PyBullet experiments using lookback size of 4, 16, 64, and 1 128 with and without attention (labelled GPIDE and GPIDE-ESS, respectively). Figures X and 9 show performance curves for GPIDE and GPIDE-ESS respectively. It is clear that there is massive increase in improvement when expanding the maximum size of the lookback from 4 to 16. For the most part, this trend continues expanding the lookback size from 16 to 64; however, it seems pushing from 64 to 128 yields mixed results. For some tasks, such as HalfCheetah-P, expanding the lookback to 128 results in noticeable improvements both with and without attention. For other tasks, such as Hopper-V, this expansion yields slightly worse performance, possibly because of training stability issues.
One interesting observation is that, when attention is included, this decrease in performance from expanding lookback can occur when increasing from 16 to 64 (see HalfCheetah-V and Walker-V) At the same time, however, it appears the GPIDE can sometimes maintain good performance when expanding the lookback to 128 when GPIDE-ESS cannot (Walker-P).
GPIDE
128
Osade
Narkeaan
HolChhokrnBBT P.V
HupporBLTPPv2 P-w)
W.
3320
2.50
2000
1990
1000
HalO ooten ViT
1ed
HusperBLTIvve
AHELTV-6
10
2500
3000
2000
1900
0.
02
115
Figure 8: GPIDE Lookback Ablation. Each curve shows the average over four seeds and the standard error of each.
GPIDE ESS
Oraeii
Markovinn
1 S:hwo
Ardil: 1-P-0
3000
3530
2.00
2503
2090
2.50
2000
2530
1593
201
1950
1000
300
500
532
1.2
AYB.TMV0
2000
2000
10:0
1900
2300
530
-120
0.
EG Enirorrent Sloza
0.
6.6
1.0
9-5
1.0
Ervtormunt Skops
Stero Stess
Eniorrent Shoz3
Figure 9: GPIDE-ESS Lookback Ablation. Each curve shows the average over four seeds and the standard error of cach.
G Further Results
In this Appendix, we give further evaluation of the evaluation procedure. In addition, we give full tables of results for normalized and unnormalized scores for all methods. We also show performance
24

---

traces. Note that the percentage changes in Table since they report all combinations of environment variants.
3 do not necessarily reflect tables in this section
G.1 Evaluation Procedure
As stated in the main paper, for tracking tasks, we fix 100 settings (each comprised of targets, start state, and system parameters) that used to evaluate the policy for every epoch of training (i.e. for every epoch the evaluation returns is the average over all 100 settings returns). We use a separate 100
settings when tuning. For the final returns, we average over the last 10% of recorded evaluations.
For the PyBullet tasks, we use ten different rollouts for evaluation following Ni et al. [50] We also average over the last 20% of recorded evaluations like they do.
Normalized Table Scores. We now give an in-depth explanation of how the scores in the table are computed. Let ((b,i) be the policy trained with baseline method (e.g. with GPIDE, transformer, GRU encoder) environment variant i (e.g. fixed, small, or large). Let JJ (TT(..) be the evaluation of policy (bb.)) on environment variant j. i.c. the average returns over all seeds and episodes. The normalized score for policy ((b,)) variant is then
Jz( 8.2 ( ma 8.2
Note that we only min and max baseline methods presented in the table.
For PyBullet tasks, we the same procedure but normalize by the oracle policy's performance (sees both position and velocity and has no history encoder) and the Markovian policy's performance (sees only position or velocity and has no history encoder). For both of these policies, we use what was reported from Ni et al. [50]. Note the our normalized scores differ slightly from those used in Ni et al. [50) since they normalize based on the best and worst returns any policy; however, we believe our scheme gives more intuitive picture of how any given policy is performing.
G.2 MSD and DMSD Results
PID Contoller
Transfommer
PIDE
GPIDE-ES
GPIDE-ESS
GPIDE-AIt
Fixed / Fixed
-6.14 0.02
-0.76 0.02
5,75 11:11
-5.69 +: 0.00
-5.76
-3.75 0.00
-5.754.0.01
-5,33 0.02
Fixel Small
-7.51 + 0004
-7.56++0.00 +.00
-7.29 0.01
-7.37 + 0.01
0.04
-7.57 0.001
-.322+00.
-7.30 0.03
Fised/ / Large
-11.99 1.000
-1252 0.11
-10.87 2: 0.05
-11.48 0.00
-11.611.00
-11.48
-12.9010..19
-11.52 0.100
Small/ /Fxed
-6.26 + 0.06
-5.80 0.00
-1922 0.01
-6.90 = 0.01
-538
-3.89 0.00
-5.92
-5.9 #: 0.02
Small/ Small
-7.49 1.000.0.000
-7.02
-T.15= 0.01
-7.14
-..1220.114
-1.15
-7.32
Small
-11.18
-10.01
0.04
-1..43+0..14
-10.42
-10.3300112
Large/
Large Small
-7.78
-7.25 0.02
-7.44
0.02
-7.45
0.44
-7.48
Larg// Large
11.12.+0.06
-9.44 0.02
-0.67 0.05
-10.37
0.02
-9.68
-9.70
-9.60
Average
8.41
-7.92
.7.82
.8.00
-7.95
-7.93
4.05
-7.92
Table 14: Unnormalized MSD Results.
PID Controller
GRU
Transformer
PIDE
GPIDE
GPIDE-ES
GPIDE-ESS
GPIDE-Attn
Fixed/ Fined Fixed
58.09
1.66
93.18 1.16
9101 1 0.83
100.00 0.27
98.18
93.77 1:
96.20 1.22
87.16
1.78
Fixed
1.35
Fixed LLagge
36.58
2.36
0.00 a: 3.42
53.70
1.71
$1,92
0.95
29.55
2.32
33.62
1.71
0.60
6.25
32.51
3.20
Small Fisee
46.87 1: 5.88
89.05 +: 0.32
78.21
1.51
75.81
0.79
77.27
1.66
61.11
1.20
78.55
1.44
79.64
1.80
Small Small
38.25
3:44
100.00 + 0.98
83.49 3.07
84,88
0.81
87.76
5.31
90.06
2.02
83.97
2.10
87.57
2.65
Small /Large
61.44-1
53.21
Large Fixed
0.00 1: 15.12
63.36 +: 1.17
15.01
3.18
66.37
46.68
3.06
19.50
3.84
18.52
3.69
45.03
4.72
Large / Small
0.00 +: 16.75
70.44
1: 3.30
45.
6.95
45.73
2.47
43.66
4.47
44.71
6.45
45.21
5.42
39.64
7.78
I / Large
45.60 + 1.71
100.00 + 0.61
92.60
1.49
60.88
0.00
93.03
1.22
92.36 36
1.62
91.67
1.68
91.95
1.80
Average
62.86
66.53
67.65
63.74
Table 15: Normalized MSD Results.
25

---

PID
GPIDE
GRU
Transformer
Fixed
Small
Large
-5.75
Fixed
-7.5
-8.00
-14
6625
0.0
0.5
1.0
-5.75
Small
8.00
-7.5
-6.0
Large
-7.5
-6.5
-11
-7.0
-8,0
-12
0.5
1.0
0.5
Environment Stepše6
Environment Step&e6
Environment Stepše6
Figure 10: MSD Performance Curves. Each row corresponds to training environment, and each column corresponds to testing environment.
PID
GPIDE-Attn
GPIDE-ESS
GPIDE-ES
Fixed
Small
Large
7.4
Fixed
-6.0
-12
-7.6
-6.5
-14
1.0
-6.0
Small
-11
7.50
-6.5
-7.5
-12
0.5
1.0
7.5
-10
Large
-6.5
-7.0
8.0
-12
0.0
0.5
0.0
0.5
1.0
0.0
0.5
Environment Stepe6
Environment Stepke
Environment Steple6
Figure 11: MSD Performance Curve Ablations. Each row corresponds to training environment, and
each column corresponds to a testing environment.
PID Coaroller
GRU
Transformer
PIDE
GPIDE
GMDD-ES
GPIDE-ESS
GPIDE-Am
Fixed / Fixed
-15.33
0.14
-16.20 0.31
11541 0.13
12..44 0.04
-13.88 0.22
-13.92 0.09
-13.55
0.05
-16.77 +.1.3
Fixed/ Small
-21.29
0.29
-25221
0.52
-21.37
0..6
18.58 0.05
-99.77
1.24
-21.11 1 0.07
-20.09
0.08
-23.25
0.155
Fxed/Larg Lazge
-27.59 2 n.44
-1T.21
2:33 0.35
0217
25.29
0.18
-2354
-.111420.11
-2811
0.11
-31.44
Small/ Fixed
-18.15
0.01
-17.75++004
-15.56+-0.11
-33.43
0.00
-1437 0.11
-14.35-0.1
10:5--000 0.11
-15.7
0.10
-16.55++0.1
Small Small
-21.78
-22.99 0.34
-2036
-18.09
0.04
-18.7 1.17
-18.02
0.10
2: 0.07
-21.77 1.010
Small/
26.67
-2.044034
20..2.
Large
-21.00 0.2
22.41
2..30
-14.83
Langs
-26.63
19..6
19.99 0.16
221.14
0.07
Large Large
22.29 0.30
-29.54 0.30
-2143 0.21
-2106 0.03
-2.08 0.14
23.00
0.07
21.81 0.09
-25
Average
22.25
25.39
21.36
118.91
19.48
20.48
19.18
-2.91
Table 16: Unnormalized DMSD Results.
26

---

PID Controller
GRU
Transformer
PIDE
GPIDE
GPIDE-ES
GPIDE-ESS
GPIDE-Att
Hixed Hixd
0.50
3.75
0.97
Fixed "Large
62:47 2:
0.00 +: 2.24
58.78 1.11
77.38 = 1.14
02.76
2.13
39.
0.83
58.92 0.73
34.84
4.61
Small /Fixed
43.30
9.27
47.76 4.225 4.25
67 02 1.30
91.92 = 0.90
82.32
1.72
82
1.34
90.46 0.99
56.988
1.16
Small Small
56.04
47.82
70.07
0.48
88.95
100.00
1.11
Small Laage
60.11
38.57 2,33
72.
0.82
88.08
57
94.09 + 0.51
0.80
Large Fixed
4.64 + 6.34
0.00 + 3.30
41.37
9.00
77.62 + 1.24
66-16-
1.45
07 60
0.396
73.51
1.24
36.06
2.85
Large 7 Small
50,02 2: 5.07
0.00 1.56
53.
2,80
82.77 0.98
76.66
63.
0.85
79.95
1:03
39.71
3.65
Largo Large
Average
55.26
29.47
64.28
88.19
82.08
73.35
85.02
30,50
Table 17: Normalized DMSD Results.
PID
GPIDE
GRU
Transformer
PIDE
Fixed
Small
Large
-2
Fixed
-15
-25
-20
-30
-40
0.0
0.5
1.0
0.5
1.0
0.5
1.0
-15
-2
Small
-30
-20
-25
-40
0.0
0.5
1.0
0.0
0.5
1.0
0.5
1.0
-20
-25
Large
-20
-30
-30
-35
-30
0.0
0.5
0.5
1.0
0.0
0.5
Environment Stepše6
Environment Stepše6
Environment Stepše6
Figure 12: DMSD Performance Curves. Each row corresponds to training environment, and each column corresponds to testing environment.
PID
GPIDE-Attn
GPIDE-ESS
GPIDE-ES
Fixed
Small
Large
-20
Fixed
33
-25
-20
-35
0.0
0.5
1,0
0.0
0.5
1,0
0.0
0.5
1,0
Small
-15
-20
-25
-20
-30
1.0
-20
Large
-20
-25
-30
-30
0.0
0.5
1.0
0.0
0.5 Environment Stepte6
1.0
0.0
0.5
1.0
Environment Stepte6
Environment Stepte6
Figure 13: DMSD Performance Curve for Ablations. Each row corresponds to a training environment, and each column corresponds to a testing environment.
27

---

G.3
Navigation Results
Controller
GRU
Transfornee
PIDE
GPIDE
GPIDEES
GPIDE-ESS
Sim /Sim
28.94 4. 3.63
96.76 4: 0.15
90.57 =: 0.12
98.33
0.08
100.00 +: 0.07
98.64 0.06
99.66 0.06
99.81 0.08
Sim / Real
43.12 + 2.08
0.00
3.94
50.55 + 0.78
68.34 4: 0.57
62.16 + 0.89
63.17
00.21 1.15
02.52 0.50
Sim
100
Real Real
67.28 1: 2.05
97.29 1: 0.20
99.20 0.14
95.94 0.04
100.00 +: 0.21
99.19 0.09
99.11 1: 0.21
99.87 4: 0.17
Average
34,83
62,89
79.34
80.54
84.36
83.71
83.18
80,78
Table 18: Normalized Navigation Results. Note that these results are after million collected samples.
Comrelll
GRU
Transfoemer
PIDE
GPIDE
GPIDE-ES
GPIDEESS
GPIDX-AII
13.77
Sim Real
23.87 4. 0.29
29.85
0.55
-22.84
0.11
-20.37
0.08
4
-21.99
6.88
-21.64 0.16
-22.57 0.07
/ 5Smm
18.60 + 0.21
16.79
0.06
16.355
0.02
-15.00
0.04
-14.92
0.03
15.02
0.03
-4.93 + 0.03
-1611 -004
Real / Real
20.55 1.22
-16.36
0.00
-16.09
1 0.02
-16.55 1 0.01
-15.96 0.03
160.09 1 0.00
-16.11 1 0.08
-16.03 0.02
Average
-2008
-18.66
-16.97
-16.99
-1645
-16.47
-16.99
-1684
Table 19: Unnormalized Navigation Results. Note that these results are after million collected samples.
PID
GPIDE Sim
GRU
Transformer
PIDE
Real
-14
-20
-16
Sim
-30
-18
335
-20
-40
0.0
0.2
0.4
0.6
1
-5.00
1
-16
17.5
Real
-18
-00.0
-20
22.55
-22
-55.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps
1e6
Environment Steps
1e6
Figure 14: Navigation Performance Curves. Each row corresponds to a training environment, and each column corresponds to testing environment. Note that these runs are only done for one million transitions.
28

---

PID
GPIDE-Attn
GPIDE-ESS
GPIDE-ES
Sim
Real
-14
-22
-16
Sim
-24
-18
-20
-26
0.0
1.0
0.0
0.2
0.4
0.6
0.8
1
10
-16
17.5
Real
-18
20.0
-20
-2.55
-22
-25.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps
1e6
Environment Steps
1e6
Figure 15: Navigation Performance Curve for Ablations. Each row corresponds to training environment, and each column corresponds to testing environment. Note that these runs are only done for one million transitions.
29

---

G.4
Tokamak Control Results
PID Controller
GRU
Transformer
PIDE
GPIDE
GPIDE-ES
GPIDE-ESS
GFIDE-AtI
Sim / Sam
90.96 0.05
100.00 = 0.03
99.63 2 0.03
81.71 0.
98.78
0.06
98.91 = 0.02
99.90 2 0.02
99.47 0.01
Sim Real
89.15 4: 0.99
40.96 5.45
40.05 11.91
21.0
55.21
4.44
61.36 = 7.40
65.65 5.66
35.06 4.41
Real Real
98.45 = 0.77
98.21 0.38
98.74 0.29
100.00
0.23
99.50 0.01
98.39 = 0.33
98.35 0.33
98.27 0.37
Average
82.29
68.88
68.42
46.18
75.67
78.12
80.10
62.43
Table 20: Normalized BN Tracking Results.
PID Couzoller
GRU
Transfocmer
PIDE
GPIDE
GPIDE-ES
GPIDE ESS
GPIDE-Ata
Sim Sim
-8.00 + 0.00
-7.19 + 0.00
-7.22 0.00
-8.7 0.02
-721
0.01
-7.19++0000 0.00
-7.00 0.00
-7.26 0.00
Sin Sim/Real
-16.41 0.30
-31.21 4: 1.67
-31.49 3,66
43.78 6.46
-26883
1.30
-21.8
-2.63 L.7
-32.55 1.35
Real Sim
-12.12 0:40
-15.55 0.3.
-13.05
0
-12.84
0:40
-11.92
-11.56 + 0,22
-15.35-+0020
Real/ Real
-1356 0.25
-1362 0.12
-134771:00 0:000
13.08 0.07
-13.30 0.20
-1558 0:110
13.53 0.100
-13.61100.11
Average
-1255
-16.39
-16.46
-20.99
-14.92
-14.99
-3398
-17.31
Table 21: Unnormalized BN Tracking Results.
PID
GPIDE
GRU
Transformer
PIDE
Sim
Real
-20
Sim
-40
-9
-60
-10
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
-13.0
-12.5
-15.0
133.5
Real
-17.5
-14.0
-20.0
-22.5
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps
1e6
Environment Steps
1e6
Figure 16: By Tracking Performance Curves. Each row corresponds to a training environment, and each column corresponds to a testing environment.
30

---

PID
GPIDE-Attn
GPIDE-ESS
GPIDE-ES
Sim
Real
-7
-20
Sim
-8
-30
-40
-9
-50
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
100
-12.5
13.6
Real
-15.0
13.8
-17.5
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Environment Steps
1e6
Environment Steps
1e6
Figure 17: 8N Tracking Performance Curve Ablations. Each row corresponds training environment. and each column corresponds to a testing environment.
PID Controller
GRU
Transfornc
PIDE
GPIDE
GPIDE-ES
GPIDE-ESS
GPIDE-AIH
Sim / Sim
16.78 0.44
0.112
97.99 1.50
82 1 1:27
100.00 4: 0.15
99.64 0.19
99.97 0.12
96.18 :: 1.35
Sim. Real
83.48 2.63
39.65 5.80
33.22 0.09
0.00 8.87
30,96 1.92
54.36 4 2:07
52.56 4 2.44
42.51 2.97
Real Real
91.78 4: 0.84
98.07 0.52
96.05 + 0.31
97.94 0.23
09.73 0.48
57.62 + 0.46
100.00 1: 0.28
06.13 + 0.47
Average 55.51
64.63
58.62
50.85
67.40
68.58
64.43
62.59
Table 22: Normalized -otaatin Tracking Results.
Comroller
Transtormer
GPIDE-ES
-1853
-21.8
-19.10
Sim
0.95
45.91 2.2.10
-63 -6
0.25
60.25
5.20
0.00
11.25
0.58
-4..8
Real Sim
33.57
1.50
-1.92
1 0.42
-91.31 0.06
-1.72
0.22
-2.31
0.98
31.69
0.76
-3488
3.45
32.06
1.42
Reall Real
27 0.30
24.51
+ 0.19
25.54 0.11
24.86 + 0.08
24.21
0.16
21.98 0.17
24.12 0.10
25.44
0.17
Avenge
-3.08
-30.20
33.72
-34.54
-2.31
-2.94
-0.63
-30.9
Table 23: Unnormalized BN Rotation Tracking Results.
31

---

PID
GPIDE
GRU
Transformer
PIDE
Sim
Real
-17.5
-20.0
-40
Sim
-22.5
-25.0
-60
-27.5
-70
1.0
0.2
0.4
0.6
0.8
10
-24
100
33
Real
-34
-28
-36
-38
-30
0.0
0.2
Environment Steps
1e6
Environment Steps
1e6
Figure 18: 8x-Rotation Tracking Performance Curves. Each corresponds training environment, and each column corresponds testing environment.
PID
GPIDE-Attn
GPIDE-ESS
GPIDE-ES
Sim
Real
-30
-20
Sim
-40
-30
-50
-35
0.00
0.25
0.50
1.00
0.00
0.25
0.50
0.75
108
106
-24
-32
-26
Real
-34
-28
-38
-30
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Environment Steps
1e6
Environment Steps
1e6
Figure 19: -ooRaaioon Tracking Performance Curve for Ablations. Each row corresponds to a training environment, and each column corresponds to a testing environment.
32

---

G.5 PyBullet Results
For these results, SAC encodes observations, actions and rewards. TD3 encodes observations and actions since it is the best performing on average.
GPIDE
77.03 4.00
71.38 4: 6.57
6.32 6.332
30.73 1: 1.80
14,85+ 11.26
-1.81
63.10
SAC
38.26
SAC. GE---IIEEEEEE ESS
71 08 + 6.95
94.58 11.00
0T.78 0 3:78
96.09 2:50
67 17-14- + 3.00
12 12.85
70.13
SACGPIIDE
86 45 4.83
76.58 5:47
64 3.82
29
08. 20.30 + 4.88
SAC
72.63
1.50
63.18 66.82
1.00
66.66 - 2.24
90.98 1. 4.28
M6,00 + 16.50
SAC SAC- GPIDE
+ 3.46
82 63
96.61
SAC Translooeen
57 00 + 9:07
24.00 1430
-11.67 --157=2244
2244
0.28 8.99
-8.31 + 1.11
7,80
50
1:25
30030 0.17
351
1.00 0.20
-13/1
VIM
-10 10
3.531
3.50
-28.67
85. 80 6.15
3.30
1.30
TD3
366
00
57.43
-4833+
17.03
36.50
77.06 1.00
15.56
3.54 3.5H
12.44
11522 + 1.88
SACLSII
01.7
-16.99
16.26
32.55
-2210 0.13
1:73
6:52
58.13
+0037
-13.67
A2C 12C-000 CRU
-K00
-15.03
7 155
4:30
30 08 3.74
19.88 4.50
8.12 + 5.43
PP1 CRU
27.00
1.03
20.04
HallCheetall-P
HallCheetah-VV
Hopper
Wilker
Heppeer-
Khilker-V V
Avenge
Ant-P
Aut-VV
Table 24: Normalized PyBullet Scores.
33

---

07.77
171.41
1722 S1 - 115.22
256 67 1.30 20
5.3.07 228-34
594.24-- + 48
SAC-OPIDE-4-
2829-11
COC-CPPIEERT ESS
67
2706 81 91-30 -0
21200.1 1. 192 02.81
2450 08 4: 55.07
1572.41 4: 72.50
00% 335 + 9--084444
+
24344
SAC-OPID LAC-OPIDE-BS
92.18
1. 95.01
2506 63 11 66.36
770.33 770.51 1-4302
1587 -50 1
F 522
1814
13 10
983-21
2165. 52 28.10
2840. 10 1 68.46
1140. 7X # 140.22
SAC-OPIDE
096 1. 500
2002
1022
NANG
1642 + 199.10
008. 17 250.52
803 S1 = 52 83
40 32 43
667 20 08
7455 234
693.55
140.91 31.81
510 26 335
173 28 14.00
2716 - 3.18
403 403.80 21
1477 17
58L63 63 189 06
77700770 75.68
12. 24 36.14
11:3 17.08
TD3-GRU
2153-22
31072
1979.56
1853.5
1796.8 194-11
L. 272.77
2511 2511.4 155 80 13
182 18 37-80
178.08 + 35.37
SAC-LSTM
1180
1254,2
05.00
30 39
137.53
20 35
A9C-ORU
03.25:1111
10
260 32
288
837
222.
14006 -1 #: 102 12
501 08 1. 28.000
20125 52 3 58
5F7. 5777.80 d. 100.1
09 30 + 80
PPO-ORU
1000
1000
HollCheattin Phethtstth-PPP
HallChhenthh-
Hoppers-
Wallenn
Hopper
Walker
Average
Anc P
Ancy V
Table 25: Unnormalieed PyBuleet Scores.
34

---

VRM
HopporBLT aP
AMEELT-P-v0
3008
3000
2000
Rutu
1003
1500
1048
1008
500
0.
Har C.teetanBBL...v.
1.0
HoppordLT MM--0
0.
1.5
0.0
1.0
146
0.0
AVEBLT-VV
1/3
3000
5000
2003
1006
2008
1003
1908
2009
0.0
0.5
1.0
1.5
3.0
0.5
1.5
0.0
8.5
1/
0.
0.5
1.
Steps
Stioo
Elvrommnn Sxppo
Figure 20: PyBullet Performance Curves.
DE-AA
GPIDE-ES
GRIDE-ES
Coade
Markoden
HaftChestorBLT.PPW
HopperBi.T P.v0
WakerBLT-P00
AMBLTT +V
2000
3030
2000
2030
5930
1000
1000
900
500
0.5
0.0
HaMhChesthha.. TV
HopperBLT VNO
1ec
WakerBLITV
1Al
AnELTT
30:00
2000
2000
2000
1000
2000
1000
1535
530
1000
0.5
0.
166
Figure 21: PyBullet Performance Curve for Ablations.
Interestingly, we found that GPIDE policies often outperform the oracle policy on Hopper-P. While the oracle performance here was taken from Ni et al. [50), we confirmed this also happens with our own implementation of oracle policy. We hypothesize that this may be due to the fact the GPIDE policy gets to see actions and rewards and the oracle does not.
35

---

G.6 Attention Scheme Visualizations
We generate the attention visualizations (as seen Figure 4 by doing handful rollouts with GPIDE policy using only attention heads. During this rollout we collect all of the weighting schemes, iee softmax giikki generated throughout the rollouts and average them together. Below, we show additional attention visualizations. In all figures, each plot shows one of the different six heads. For each these, the policies were evaluated on the same version of the environment they were trained
22-
20
20
20-
Sergar
60-
66-
60+
10
56 Time Stop
50
55
15
Stop
Stop
Tims
Stop
Figure 22: MSD-Fixed Attention.
Sequiee Langtt
2
20
80
80-
Time Step
Time Stop
Time Stop
Tmm Stap
Tima Step
Time Stop
Figure 23: MSD-Small Attention.
Senouacio Langs
82-
40-
40-
80
Time
Time Sime
Stop
Tima Stop
Time
Time Stop
Figure 24: MSD-Large Attention.
Sequente Lenghh
28-
20-
60
50
56
50 Stop
52
Stop
Figure 25: DMSD-Fixed Attention.
Sequree Lango
2
Time Stop
Time St0p
Time Stop
Time Stop
Timz Step
Tine Stop
Figure 26: DMSD-Small Attention.
36

---

Seq 12
56 Time
Time
Time Step
Time
Time
Figure 27: DMSD-Large Attention.
Serguence Lamghh
50
Figure 28: Navigation Friction Attention.
Seq Lengt
20
10
62
60
50
56
50 Step
53
53
Figure 29: Navigation Friction Attention.
Sequitc Lereth
20
Time Step
Time Seep
Time Stap
Time Step
Time Step
Time Step
Figure 30: 8x Tracking Sim Attention.
Sequence Langth
48-
43-
80
55
Time
Time
Time Tmmp5lep
Tim
Time Step
Figure 31: AN Tracking Rotation Attention.
Sequance Langh
28
20
20
62
60
10
56
50
53
53 Stop
Stop
Time
Figure 32: -N -y-oottioon Tracking Sim Attention.
Pegurre Langon
20
30
80
Time Skep
TimeSttp
Time Step
Tim Stap
Tima Step
Time Stp
Figure 33: B. -Rotation Tracking Rotation Attention.
37

---

60
25
56
25
50
25
50
25
53
25
Step
Stpp
Figure 34: HalfCheetah-P Attention.
Scepanoo Langth
25
10
25 40
26
Stop
Tim Stop
5t0p
Figure 35: HalfCheetah- V Attention.
Saquertee Lampih
20
6G-
40-
60-
60
60
23
25
Figure 36: Hopper-P Attention.
SeCu4nte sergt
20-
20
20-
40
60
23
25
Time
Time
Figure 37: Hopper-V Attention.
Soquente Length
20
40
25
25
Time
Tim
Figure 38: Walker-P Attention.
DEGHTOO Langet
42
K6-
40
60-
25
25
23
30
25
Stop
Figure 39: Walker-V V Attention.
20
40
50
50
60
25
25
Figure 40: Ant-P Attention.
38

---

Time Stop
Time St0p
Time Stop
TimeStes
Time Step
Tine Step
Figure 41: Ant-V Attention. Note that total path length is less than 64 here since the agent falls down pretty fast.
39


Reviews:
[{"review_id": "C96IVU9aCx", "reviewer": "Reviewer_vEC9", "summary": "This paper studies the problem of learning from histories in partially observable MDPs where a key question is how to design an architecture that is general enough so as to work on a large set of problems, yet specific enough to be sample efficient.\n\nInspired by the success of PID in the more classical control literature, the idea here is to design an architecture that is solely comprised of summation, attention, and linear operations. \n\nExperiments show that on tracking problems, where PID is originally proposed for, as well as more complicated control problems, this simple architecture is good enough to achieve competitive results.", "questions": "I would like to ideally see an acknowledgment that as we move to much more difficult domains, GPIDE may be incapable of doing well simply because the actual function to be represented in not in the function class. But such an example is currently absent. This may give a false sense that the paper is claiming that GPIDE is a general-purpose memory learner but that is not true, and I am not saying that authors say it is true, but having examples of failure cases would really clarify this point.\n\nIf I may say, I would also like to see experiments on different types of problems. For example, can we expect GPIDE to perform reasonably on image-based domains? Like the paper mentions, we can hope to apply the GPIDE on the embedding space learned by some more capable function approximator, but it seems to me that to learn good embeddings that are useful for downstream memory learning, one needs to leverage some kind of GRU/Transformer/LSTM architecture, and so it is not immediately clear to me how to leverage GPIDE in this setting.\n\nMy other question is also that as we know RNNs struggle to learn important events from the distant past because the vanishing/exploding gradients. LSTMs and Transformers are less prone to this issue but I wonder if we could better compare this capability of GPIDE in contrast to the existing architectures. In the Half-Cheetah experiment, for example, do we have a good sense of how far back one needs to go to be able to accurately learn the position and/or the velocity signal?\n\nAlso, from the intro the paper reads \"Another hurdle stems from the fact that policies are often trained in an imperfect simulator, which is likely different from the true environment. Combining these two challenges necessitates striking a balance between extracting useful information from the history and avoiding overfitting to modelling error.\"\n\nMy thoughts reading this sentence was that we are going to either have actual sim2real experiments or some kind of a transfer learning benchmark, where we learn the memory function in some settings, and test the learned memory function in another test setting. This would have been more convincing in terms of claiming that GPIDE is not prone to overfitting. Did I get it right that this experiment is absent?", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper shows that using a complicated architecture is an overkill for many of the domains that the RL community considers as benchmarks. On this note I'd like to mention that, while the contexts are somewhat different, a similar conclusion can be made from \"Towards Generalization and Simplicity in Continuous Control\" Rajeswaran et al, where they show that simpler architectures such as linear models are equipped to deal with Mujoco Tasks. The difference is that in their paper they were considering reward-maximization in presence of the full state, whereas here the goal is learn a memory-like function in conjunction with performing well in the given task.\n\nIn any case, I generally like results that show simple stuff work, because simple things are easy to understand, implement, and execute. ", "weaknesses": "Overall I like this paper but I also have some concerns:\n\nFundamentally, I feel like the results in this paper are pertinent in terms of a bigger question, namely how rich we want our hypothesis class to be for learning a certain target. The solution adopted by the ML community nowadays seems to be: we really do not know how complicated a task is, so by choosing a very complicated architecture and the fact that we are at the mercy of universal function approximation capability, we can always in principle find the right fit. Moreover, the double descent phenomenon  (\"Deep double descent: where bigger models and more data hurt\", Nakkiran et al) tells us that empirically we are very likely to find a good fit with larger and more complicated models and that, despite more classical belief, such an overparameterized fit happens to also be quite robust. (\"A Universal Law of Robustness via Isoperimetry\" Bubeck and Sellke)\n\nNow, moving to the RL setting, these insights from supervised learning do not really carry over because we usually learn by bootstrapping off of Bellman targets, and by increasing the complexity of our function approximator, we also have to learn from increasingly more complicated targets (unlike supervised learning where targets are fixed).\n\nSo I think, at least with our current RL algorithms, unlike SL we always need to choose our function class carefully and so adding inductive bias, such as ones provided in this paper, can usually be helpful. My worry is that adding such biases are helpful in a limited sense: It is not clear how much inductive bias should I add for a given task when I don't know too much about that task. Sure, I can always choose environments and problem settings where I know that my inductive bias is conducive to solving the problem, but do we really want to bring back the burden to ML practitioners to carefully think about the kind of inductive biases that are useful for their task?\n\n\nGrounding my concerns more so in the experimental results provided here, for example we see that the inductive bias provided is already giving small (or negative) gains as we move to some of the more complicated domains such as HalfCheetah and Ant (based on Figures 18 and 19 in the Appendix), so I ask authors, are you not worried that as we move to much larger domains then it becomes increasingly more difficult to show performance gains for GPIDE compared to other approaches? In other words, does the inductive bias scale reasonably with problem complexity or there is only so much this could be useful?\n\nTo further play devil's advocate, to say that GPIDE is capable of beating the transformer architecture is sort of a weird claim, because as authors say GPIDE could be thought of as a special case of transformers, and further as you show the attention layer is often not very useful. This tells me that with proper hyper-parameter tuning and regularization, the transformer architecture should in principle be able to always do better than GPIDE, and we can still hope to find the transformer useful as we move to larger problems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and your thoughtful review! We greatly appreciate the feedback, and we are glad that you liked the paper.\n\n* Your comments highlight a fascinating problem which is the tradeoff between the benefits of high capacity modeling and the benefits of useful inductive biases. As you point out, the community is getting a better handle on these tradeoffs for supervised learning, but for RL many of these questions are fairly open. We agree that our Limitation section should better acknowledge GPIDE’s reduced capacity being a potential problem for high dimensional or complex environments, and we will expand our discussion in the final version of the paper.  \n\n   While a quick look at the margin of victory might suggest we do better on lower dimensional environments, we note that GPIDE consistently improves upon or matches the performance of the best competitor (see global response for more information on HalfCheetah-V). It is important to note this best competitor is not always the same. In particular, while SAC-LSTM is the best performing competitor by a substantial margin on Ant-P, it is also the worst performing competitor on Ant-V. We believe this robustness across environments is encouraging for many other problems including larger and more complex ones. \n\n   Regarding the possibility of transformers always doing better than GPIDE, it is not clear to us that this is true with the right hyper-parameters and regularization. We view this as analogous to CNNs vs fully connected networks (FCNs) on image tasks. While the FCNs are a more expressive class, they are rarely as performant as their CNN counterparts, which utilize an intelligent inductive bias. As you already pointed out, there are still many open questions around RL and whether these same phenomena apply. \n\n* We have done an additional experiment on the amount of lookback for HalfCheetah. Please refer to the global response.\n\n* We do have substantial experiments on sim2real and the generalization capabilities of PIDE and GPIDE. For the MSD and DMSD environments, we train the policy on a reduced set of system parameters and test on a larger set. This is the same type of experimental setup used in “Assessing generalization in deep reinforcement learning” (Packer et al 2018). We expand on this in the navigation experiments by having the “sim” not model friction whereas the “real” environment has friction. Perhaps most realistic, in our fusion experiment we create a simulator using first-principle equations. For the “real” environment we use a data-driven dynamics model trained using data from the actual device. In all of these experiments, PIDE and GPIDE showed an impressive level of generalization, especially when compared to using a GRU.\n"}, {"review_id": "84wLNa4TVY", "reviewer": "Reviewer_EZtX", "summary": "This paper proposes a new way to encode features in partially observable environments using PID. Experiments show superior results over several domains compared with recurrent and transformer encoders.", "questions": "Questions,\n\nPlease see the pros and cons part about the assumptions, technical details, ablations, and limitations.\n\n", "limitations": "Please see the pros and cons part about the limitations.\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is easy to read overall.\n\nExperiments show promising performance on multiple domains.", "weaknesses": "Some assumptions and technical details need to be clarified.\n\n1) Using the difference between observations as the derivative of the error (D) is trivial,  assuming that the reference value for the observation at different timesteps is the same, which is not true.\n\n2) The author mentioned in Line 102 that using M separate PID controllers to control M signals has some clear shortcomings. This paper models the MIMO problem in a centralized manner, inputting all inputs and outputting a 3M dimensional vector. However, this would not be feasible if there is a large scale of signals that need to control.\n\n3) It said the $f_{\\theta}^h$ is a learnable linear projection, how to learn it is not discussed. Also with $g_{\\theta}$.\n\n4) How to determine the suitable length of the history, As for the length $1:t$ in $v_{1:t}^{h}$?\n\nMore ablations should be provided.\n\n1) The ablations investigate the influence of different types of heads, while why choosing these three settings (ES, ES+Sum, Attention) is not explained. What about ES+Attention? Why five ES + 1 Sum? Are there any insights in choosing the number and combination of different heads?\n\n2) The influence of different lengths of history should also be investigated.\n\nMore limitations should be discussed. For example, the encoder processing time is highly dependent on the length of the history.\n\nSome related works are not discussed in this paper. For example, [1-3].\n\n[1] An adaptive deep reinforcement learning approach for MIMO PID control of mobile robots\n\n[2] Self-Tuning Two Degree-of-Freedom Proportional–Integral Control System Based on Reinforcement Learning for a Multiple-Input Multiple-Output Industrial Process That Suffers from Spatial Input Coupling\n\n[3] Robust Adaptive PID Control based on Reinforcement Learning for MIMO Nonlinear Six-joint Manipulator\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and your review! We will respond to each of your weaknesses in a corresponding list. Starting with the list of assumptions and technical details,\n\n1. For your first point, you are correct that the reference value need not be the same throughout time. However, GPIDE can still recover the correct D term. To demonstrate, consider an M=1 dimensional tracking problem. As per line 89, $o_t = (x_t, \\sigma_t)$. Then, $o_t - o_{t - 1} = (x_t - x_{t-1}, \\sigma_t - \\sigma_{t - 1})$. If we let our $f$ be a projection that simply subtracts the second coordinate from the first we get $f(o_t - o_{t - 1}) = (x_t - \\sigma_t) - (x_{t - 1} - \\sigma_{t - 1})$. Thus, we have recovered the correct D term up to a constant.\n\n2. While we agree that learning a policy becomes difficult as the dimensionality $M$ grows, we assert that all approaches will struggle as $M$ becomes large, especially since policies need to remember a history of observations to perform well in the POMDP setting. For example, a GRU-based approach would learn a $D$-dimensional encoding of the history, which effectively makes the policy over a $(M + D)$-dimensional input. We found that $D$ needs to be relatively large for good performance (64 for tracking experiments and 256 for locomotion experiments), and it would likely need to grow as $M$ grows. Moreover, the mapping from the history to the $D$-dimensional encoding is often learned simultaneously with the policy. Adding up all these factors, we assert that learning a policy that takes in a fixed, $3M$-dimensional input is actually a less daunting task than using a GRU or LSTM.\n\n3. $f$ and $g$ are simply fully connected layers with no activation functions. Everything is trained end-to-end using the Adam optimizer. We will add a mention of this in the final paper.\n\n4. For the lookback $t$, we simply use the full episode for tracking problems and the same setting as was used by the baseline methods in Ni et al. In practice $t$ is a hyperparameter; however, this is not unique to our method. Although recurrent networks can encode arbitrary history lengths at test time, one must often select how far to look back during training the recurrent network. Please refer to the global response for more discussion on this.\n\nFor your comment on the ablations:\n\n1. We chose the ablations we did because we wanted to investigate two questions: how important are heads that accumulate information (hence ES vs ES + Sum)? and do we even need ES if attention has the capacity to represent it already (hence ES vs Attention)? This reasoning was not presented well in the paper, and our final version will flesh this out. While the results of ES + Attention would be interesting to see, we did not feel the need to devote computational resources to this configuration since it would not answer either of our two questions. As to why we picked 5 ES heads and 1 summation head, ES can capture several different time scales depending on $\\alpha$ whereas summation can only capture one. Therefore, it seemed logical to only have one summation head.\n\n2. Please refer to the global comment for time horizon ablations.\n\nThank you for bringing these related works to our attention. We will cite them in our final paper. While they cover how RL can improve PID rather than how PID can improve RL (like our paper), we still think it is important related work to discuss. \n\nIf you feel that we have adequately addressed your concerns, we would respectfully ask you to consider raising your score. Thank you!\n"}, {"review_id": "kLRZCssb3X", "reviewer": "Reviewer_QQYm", "summary": "This paper considers history encoding for deep RL POMDP control problems through a PID-inspired lens. Specifically, authors introduce PIDE, a method of directly using PID control which extends to multiple-input multiple-output problems, and GPIDE, a PID-inspired encoder architecture. GPIDE consists of a series of heads that takes as input model observations and past history, and internally computes differences between observations (similar to the PID D-term) and aggregation/summation over all past timesteps (similar to the PID I-term).\n\nThe method is then evaluated on tasks including mass-spring-damper, navigation, tolemak control, and pyBullet locomotion. Across these experiments, the method is compared against baselines of direct PID control, GRU, transformer, and several RL algorithms. GPIDE is found to have strong performance across both simple and complex tasks and be robust to domain transfer scenarios. Further ablation studies provide insight into design choices for GPIDE heads, visualize attention schemes, and evaluate a GRU+VIB baseline.\n", "questions": "#### Limitations\nThe authors mention that control tasks which require memory may not be suitable - it would be further interesting to consider from the perspective of task similarity to that of PID control. Do the authors think that there would be correlation between similarity to PID / tracking tasks and successful application of GPIDE?\n\nFor instance, a common image-based task would be visual grasping which involves reasoning about object geometry, where it may be less clear that results would directly transfer.\n\nFurthermore, some tasks may be less partially observable in nature than others - given this method focuses on a history encoder, would the relevance of history generally for certain tasks be relevant to consider?\n", "limitations": "The limitations are generally adequate (there are some questions in the section above).\n", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "#### Originality\n\nThe ideas of the paper are generally original, building from prior deep RL work and focusing on the design of the feature encoder, especially designing a novel history encoding architecture which draws from PID controllers. The pieces of the architecture draw from general temporal aggregation principles and transformers.\n\n#### Quality\n\nThe paper is generally high-quality. The experiments compare against relevant baselines and prior work, are run with multiple seeds, and the training / setup / tasks are clearly defined. The code is provided and looks cleanly written from initial observation, though I haven’t run it to verify. The ablation studies answer my initial questions about the architecture design, especially in how important the attention heads are.\n\n#### Clarity\n\nThe manuscript is well written and clearly conveys the derivation of the method from PID controllers and explains the architecture and design choices in a very understandable way. Relevant figures are provided for context of the architecture and tasks. The experiments are well motivated and help understand the performance of PIDE / GPIDE across tasks, RL methods, and transfer scenarios.\n\n\n#### Significance\n\nGPIDE seems broadly applicable across deep RL control tasks, especially those related to tracking and navigation / locomotion as demonstrated by experiments. The method is an approach to handling the POMDP problem on control problems as a drop-in architecture for history encoding.\n", "weaknesses": "#### Experiments\nCompared to RNNs such as GRU which consider only the current timestep and state to predict the next action, GPIDE seems to consider all past timesteps from 1 to t within each step, as part of the weighted summation resembling the PID I step, which may be more expensive than other methods if the time / observations are large. It may be worthwhile to consider potential inference time compute differences because of this, and also to consider ablations or baselines which also have such history context.\n\n#### Significance\nOn line 116-117, the authors mention that LSTM, GRU, and transformers were shown to be powerful tools for NLP because of complex relationships within the task. Because control tasks may not have such complexities, such methods may overfit rather than help, especially in cases of domain transfer. However, these methods have also been applied to success in domains including robotics and visual encoding (e.g. https://arxiv.org/abs/2212.06817, https://arxiv.org/abs/2010.11929, https://arxiv.org/abs/2106.01345) - perhaps this indicates that the best targets for using GPIDE would be cases where highly-overparameterized / “large” models may overfit, which would be towards the simpler side.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and your thorough review! We were happy to see that you thought the paper was high-quality and that the method seems broadly applicable across RL tasks. We have touched on many of your points in the weakness section in the global response (particularly computational expense, ablation of history context, and extension to image-based tasks). You mentioned that perhaps the best targets for using GPIDE are cases in which the tasks or dynamics are relatively simple. This may very well be the case; however, we would like to point out that the papers you linked use a supervised loss function during training. Besides a few works we mention in the paper, it seems that most successes applying transformers to control use supervised losses. We found that training transformers in an online RL setting where there are bootstrapped target values is far more challenging. As such, it is not clear to us exactly when a transformer should be used over GPIDE for online RL."}, {"review_id": "M18v4cpPyx", "reviewer": "Reviewer_64AN", "summary": "The paper proposes a new, simplified architecture for reinforcement learning in the partially observable setting inspired by PID control. Through experiments on a number of tracking problems, and some locomotion environments, strong performance is obtained.", "questions": "See the final weakness, it would be nice to see computational cost measured.", "limitations": "Limitations are discussed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The exposition of the argument is very clear, even for a reader somewhat unfamiliar with some aspects.\n- Simplification of complex (and computationally expensive) recurrent architectures is a worthwhile endeavor.\n- Ablations are informative, and come with demonstrations of issues (Figure 4).", "weaknesses": "- The largest weakness is in the evaluation. Most of the evaluation is devoted to low-dimensional tracking problems. The choice of state-based locomotion with only positions or velocities observable also seems a little artificial when there are many tasks that have existing sources of partial observability (long-horizon, first-person viewpoints).\n- One possibly beneficial use of an alternative state summary like GPIDE would be computational savings, but there isn't any measurement of that in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and your review! For your note on computational savings, please refer to the global response.\n\nWhile it is true the majority of the environment variants (10 out of the 18) are lower-dimensional tracking problems (either 3 or 6 dimensions), we do not believe that they are any less valuable than the high dimensional ones. It would be one thing if all baselines performed optimally on the lower dimensional experiments, but we find that is not the case and that pre-existing (and widely used) methods can be very suboptimal. How can we expect a method to perform optimally on high-dimensional tasks if it cannot on basic low-dimensional ones? \n\nLastly, while we agree there are many different types of partial observability that occur in real-world tasks, partial observability of positions or velocities is a common benchmark in the literature (Han et al 2019, Meng et al 2021, Yang and Nguyen 2021, Ni et al 2022). \n"}, {"review_id": "vkTuZKKBHP", "reviewer": "Reviewer_RWrF", "summary": "The paper proposes two history encoders for RL in partially observable control tasks. The history encoders are designed with PID-inspired inductive bias. Specifically, the inductive bias comes from lifting the observation into features consisting of summation and difference of observation over the historical horizon, which are analogous to the integral and derivative of tracking errors in PID. It was shown that the proposed history encoders, especially the generalized PID encoder, could achieve better performance and robustness than GRU and transformers in tracking and locomotion tasks when integrated with RL algorithms. ", "questions": "1. Please see the three issues listed in the Weaknesses section. I would have a more positive impression of the paper if the authors could properly address these three issues during the rebuttal. \n\n2. In addition, I want to ask the authors to clarify the role of attention heads in GPIDE. In the ablation study, the authors stated:\n>\n> It appears that the attention scheme for HalfCheetah is simply a poor reproduction of exponential smoothing, making it redundant and suboptimal. In fact, we found this phenomenon to be true across all attention heads and PyBullet tasks. We believe that the periodicity that appears here is due to the oscillatory nature of the problem and lack of positional encoding (although we found including positional encoding degrades performance).\n>\nIf that is the case, does it mean that the attention heads could be simply replaced by exponential smoothing heads? It seems to be supported by the ablation study results (i.e., ES+SUM has similar or better performance than GPIDE with multiple types of heads). In lines 227-230, the authors said:\n>\n>This choice was not optimized but rather was picked so that all types of heads were included and so that GPIDE has roughly the same amount of parameters as our GRU baseline.\n>\nI don't think it is a good justification for the experimental design. The authors should attempt to find an optimized configuration so that the audience can have a better sense of the necessity of each proposed head, especially the attention head. \n", "limitations": "The authors acknowledged the limitation of the proposed methods, which is the proposed PID-inspired inductive bias might not apply to all control tasks. While I personally doubt if the proposed method can ever be extended to systems with image-based observations, I do not think the authors are obligated to justify the feasibility of extension to visual control tasks in this paper. However, as suggested above, I do think the authors should provide more information and insights on the failure of GPIDE in Cheetah-V. ", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "I particularly appreciate the new perspective presented by the authors in combining conventional control techniques with reinforcement learning. The idea of incorporating PID architecture into history encoders is interesting and novel to the best of my knowledge. The advantages of the generalized PID encoders over GRU and transformers, especially in the locomotion tasks, validate that the PID-inspired inductive bias is indeed effective in some control tasks. This paper could inspire the RL community to discover more powerful and generalizable architectures with insights from control theory and conventional control techniques. ", "weaknesses": "1. In lines 102-104, the authors stated:\n>\n> In the case of MIMO tracking problems, where there are M signals with M corresponding actuators, one can control the system with M separate PID controllers. However, this assumes there is a clear breakdown of which actuator influences which signal.\n>\nDecomposing a MIMO system into multiple SISO systems is indeed a practical way to synthesize MIMO PID controllers. However, there exist methods to directly synthesize PID controllers for MIMO systems without decomposition (e.g., [1]). While I am okay with the authors only comparing the proposed methods with a decomposed PID controller, the authors should appropriately assert the capacity of PID controllers based on the state-of-the-art literature. \n\n2. The related work section should include a review of the literature on the combination of conventional control techniques with RL. For instance, there have been extensive efforts in tuning PID or learning an adaptive PID weight schedule with RL, which could be considered a special case of the control policy with the proposed PID encoder. I understand the paper is mainly targeting audiences from the RL community, but I think it is necessary to place the proposed method in the control literature as well. \n\n3. While I am generally impressed by the experimental results, I am concerned about the failure of SAC-GPIDE in the HalfCheetah-V environment. It makes me wonder whether the PID-based inductive bias could limit the expressiveness of the policy network in some control tasks. For instance, I would doubt if the proposed method could still work when image-based observation is used. The authors do not provide any explanation or hypothesis for the failure in HalfCheetah-V. I would suggest providing more information and insights regarding it during the rebuttal and in the updated version of the paper.  \n\n[1] Boyd, Stephen, Martin Hast, and Karl Johan Åström. \"MIMO PID tuning via iterated LMI restriction.\" International Journal of Robust and Nonlinear Control 26, no. 8 (2016): 1718-1731.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and your thoughtful review! We are glad to hear that you were generally impressed with the experimental results. For the failure of HalfCheetah-V, we agree that the current iteration of the paper could benefit from more information. Please see the global response for what we plan to add.\n\nWe also agree we need to do a better job at appropriately framing the capacity of PID based on state-of-the-art literature. On top of this, while our paper is about how PID can help RL, we agree that we do need to reference literature on how RL can help PID to give readers the full picture. In addition to the works recommended by reviewer EZtX, we also plan on referencing the works shown at the end of this response in our discussion.\n\nFor your second question, although there are a few cases where attention may be helpful, generally it seems that the best performing variant of GPIDE does not have attention. We thought the version of the paper that would be most coherent to readers is to first run experiments with a “complete” GPIDE. That is, a version of the method that, although possibly suboptimal, uses all of the  heads discussed in the Methodology section. Through the ablation section we hoped to give readers insights into the importance of each of the proposed heads and hone in on a more optimal version of GPIDE.\n\nIf you feel that we have adequately addressed your concerns, we would respectfully ask you to consider raising your score. Thank you!\n\nLawrence, N. P., Stewart, G. E., Loewen, P. D., Forbes, M. G., Backstrom, J. U., & Gopaluni, R. B. (2020). Reinforcement learning based design of linear fixed structure controllers. IFAC-PapersOnLine, 53(2), 230-235.\n\nGuan, Z., & Yamamoto, T. (2021). Design of a Reinforcement Learning PID controller. IEEJ Transactions on Electrical and Electronic Engineering, 16(10), 1354-1360.\n\nJesawada, H., Yerudkar, A., Del Vecchio, C., & Singh, N. (2022). A Model-Based Reinforcement Learning Approach for PID Design. arXiv preprint arXiv:2206.03567.\n\n"}]

Author rebuttal:
Thank you to all of the reviewers for taking the time to read our paper and give thoughtful reviews. We value your feedback and hope to use it to strengthen our work. In this global comment we will address some points that were raised in multiple reviews.

### Performance of HalfCheetah-V

Some reviewers pointed out that GPIDE achieves substantially worse performance on HalfCheetah-V and were concerned this may indicate the GPIDE architecture does not have enough flexibility to handle higher dimensional tasks. However, we assert this is not due to the GPIDE architecture but due to attention heads. Looking at Table 28 in the Appendix, we see that removing attention heads more than doubles the average performance ($20.39 \pm 29.60$ to $53.14 \pm 5.86$), making it competitive with the TD3-GRU, which is the best performing method ($59.03 \pm 2.88$). Moreover, GPIDE-Attention is one of the worst performers, showing attention heads result in particularly poor performance in this environment. We agree that the current iteration of the paper is lacking this explanation, and we will update our paper with these details.

### Computational Cost of GPIDE

Many reviewers inquired about the computational cost of GPIDE. This information can be found in Appendix E, and the final version of the paper will do a better job of referencing this in the main body. Using attention is slower since the current query must be compared with all previous keys in order to form the encoding. We see in practice this results in a roughly 20% slow down on the tracking problems. However, we would like to emphasize that if no attention heads are used (which ablations suggest is optimal), $w_{t-1}$ can be cached at every time step and only $v_t$ need be computed to calculate $w_t$. Since $v_t$ is the result of a linear projection, this would likely be faster than using a GRU or LSTM. Unfortunately, our implementation of GPIDE-ES and GPIDE-ESS does not take this shortcut into consideration since our code was created with flexibility of head type in mind. 

### Impact of Different Lookback Lengths

Multiple reviews wished to see the impact of lookback length on the policy performance. We were able to run preliminary experiments with varying lookback on the HalfCheetah environments for GPIDE (see one page pdf), and will run similar experiments for all locomotion environments and variations of GPIDE found in the ablations for the full paper. As a reminder, we picked a lookback of 64 since this is the sequence length that competitor algorithms trained on; however, we agree that investigating the lookback is important to understanding GPIDE.

For HalfCheetah-P, it is clear that a greater amount of lookback corresponds to higher returns. In fact, it seems that we are able to get even stronger performance by increasing the history from 64 to 128.

For HalfCheetah-V, there is a clear increase in performance going from 4 to 16 lookback; however, performance drops off after this and crashes for a lookback of 128. As stated in the global response addressing HalfCheetah-V, we believe attention heads are the cause for this, and it seems that there are instabilities that occur as lookback grows. We do not expect to see these same instabilities when we run the same experiments for GPIDE-ES and GPIDE-ESS. As initial evidence to this claim, we also have preliminary data for GPIDE-ESS with a lookback of 128. Although these jobs are still running, it is clear that the stability problems have been eliminated, and it seems the final performance will be greater than any of the other GPIDE variants.  

### Extension to Image-Based Tasks
Many reviewers also mentioned the extension to image-based tasks. While we are optimistic about the application of GPIDE on such tasks given the right image encoding, working with images comes with its own set of challenges which we leave for future work.


Meta-review: This paper addresses the problem of encoding an RL agent's history to provide a state representation in partially observable RL tasks. The proposed architecture is a feedforward encoder that learns multiple "heads" which each encode past history in a different way. The architecture is evaluated in RL tasks, mainly focusing on tracking but also partially observable locomotion tasks

Main strengths: The architecture is novel to the best of reviewer knowledge. Seems like an interesting in-between architecture between recurrent architectures and the feedforward approach of simply concatenating history. Empirical evaluation shows improved policy learning in a range of partially observable learning tasks.

Main Concerns: I find the connection to PID control tenuous and would encourage the authors to clarify how the novel architecture is PID-like or to downplay the PID connection. It was hard to see the connection to PID control. I think the empirical study quality could be improved with additional trials ran for experiments to provide more confidence in results. I downplay this concern some because of the large number of tasks considered. A nice improvement could be for the authors to consider using the tools provided by the author of this paper: https://arxiv.org/pdf/2108.13264.pdf.

History encodings for handling partially observable tasks are a useful contribution for RL and the proposed architecture seems useful for this. I would recommend acceptance.
Decision: Accept (poster)