Title: PID-Inspired Inductive Biases for Deep Reinforcement Learning in Partially Observable Control Tasks
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reinforcement Learning

Control

POMDP
Abstract: Deep reinforcement learning (RL) has shown immense potential for learning to control systems through data alone. However, one challenge deep RL faces is that the full state of the system is often not observable. When this is the case, the policy needs to leverage the history of observations to infer the current state. At the same time, differences between the training and testing environments makes it critical for the policy not to overfit to the sequence of observations it sees at training time. As such, there is an important balancing act between having the history encoder be flexible enough to extract relevant information, yet be robust to changes in the environment. To strike this balance, we look to the PID controller for inspiration. We assert the PID controller's success shows that only summing and differencing are needed to accumulate information over time for many control tasks. Following this principle, we propose two architectures for encoding history: one that directly uses PID features and another that extends these core ideas and can be used in arbitrary control tasks. When compared with prior approaches, our encoders produce policies that are often more robust and achieve better performance on a variety of tracking tasks. Going beyond tracking tasks, our policies achieve 1.7x better performance on average over previous state-of-the-art methods on a suite of locomotion control tasks.

PID-Inspired Inductive Biases for Deep Reinforcement Learning in Partially Observable Control Tasks

Abstract

Deep reinforcement learning (RL) has shown immense potential for learning to control systems through data alone. However, one challenge deep RL faces is that the full state of the system is often not observable. When this is the case, the policy needs to leverage the history of observations to infer the current state. At the same time, differences between the training and testing environments makes it critical for the policy not to overfit to the sequence of observations it sees at training time. As such, there is an important balancing act between having the history encoder be flexible enough to extract relevant information, yet be robust to changes in the environment. To strike this balance, we look to the PID controller for inspiration. We assert the PID controller's success shows that only summing and differencing are needed to accumulate information over time for many control tasks. Following this principle, we propose two architectures for encoding history: one that directly uses PID features and another that extends these core ideas and can be used in arbitrary control tasks. When compared with prior approaches, our encoders produce policies that are often more robust and achieve better performance on a variety of tracking tasks. Going beyond tracking tasks, our policies achieve 1.7x better performance on average over previous state-of-the-art methods on a suite of locomotion control tasks. 1

1 Introduction

Deep reinforcement learning (RL) holds great potential for solving complex tasks through data alone, and there have already been exciting applications of RL in video game playing [70], language model tuning [53], and robotic control [2]. Despite these successes, there still remain significant challenges in controlling real-world systems that stand in the way of realizing RL's full potential [20]. One major hurdle is the issue of partial observability, resulting in a Partially Observable Markov Decision Process (POMDP). In this case, the true state of the system is unknown and the policy must leverage its history of observations. Another hurdle stems from the fact that policies are often trained in an imperfect simulator, which is likely different from the true environment. Combining these two challenges necessitates striking a balance between extracting useful information from the history and avoiding overfitting to modelling error. Therefore, introducing the right inductive biases to the training procedure is crucial.

The use of recurrent network architectures in deep RL for POMDPs was one of the initial proposed solutions [29] and remains a prominent approach for control tasks [47, 75, 50]. Theses architectures are certainly flexible; however, it is unclear whether they are the best choice for control tasks,

especially since they were originally designed with other applications in mind such as natural language processing.

In contrast with deep RL methods, the Proportional-Integral-Derivative (PID) controller remains a cornerstone of modern control systems despite its simplicity and the fact it is over 100 years old [5, 48]. PID controllers are single-input single-output (SISO) feedback controllers designed for tracking problems, where the goal is to maintain a signal at a given reference value. The controller adjusts a single actuator based on the weighted sum of three terms: the current error between the signal and its reference, the integral of this error over time, and the temporal derivative of this error. PID controllers are far simpler than recurrent architectures and yet are still able to perform well in SISO tracking problems despite having no model for the system's dynamics. We assert that PID's success teaches us that in many cases only two operations are needed for successful control: summing and differencing.

To investigate this assertion, we conduct experiments on a variety of SISO and multi-input multioutput (MIMO) tracking problems using the same featurizations as a PID controller to encode history. We find that this encoding often achieves superior performance and is significantly more resilient to changes in the dynamics during test time. The biggest shortcoming with this method, however, is that it can only be used for tracking problems. As such, we propose an architecture that is built on the same principles as the PID controller, but is general enough to be applied to arbitrary control problems. Not only does this architecture exhibit similar robustness benefits, but policies trained with it achieve an average of 1.7x better performance than previous state-of-the-art methods on a suite of locomotion control tasks.

2 Preliminaries

The MDP and POMDP We define the discrete time, infinite horizon Markov Decision Process (MDP) to be the tuple ( S , A , r, T, T 0 , γ ) , where S is the state space, A is the action space, r : S × A × S → R is the reward function, T : S × A → ∆( S ) is the transition function, T 0 ⊂ ∆( S ) is the initial state distribution, and γ is the discount factor. We use ∆( S ) to denote the space of distributions over S . Importantly, the Markov property holds for the transition function, i.e. the distribution over a next state s ′ depends only on the current state, s , and current action, a . Knowing previous states and actions does not provide any more information. The objective is to learn a policy π : S → ∆( A ) that maximizes the objective J ( π ) = E [ ∑ ∞ t =0 γ t r ( s t , a t , s t +1 )] , where s 0 ∼ T 0 , a t ∼ π ( s t ) , and s t +1 ∼ T ( s t , a t ) . When learning a policy, it is often key to learn a corresponding value function, Q π : S × A → R , which outputs the expected discounted returns after playing action a at state s and then following π afterwards.

In a Partially Observable Markov Decision Process (POMDP), the observations that the policy receives are not the true states of the process. In control this may happen for a variety of reasons such as noisy observations made by sensors, but in this work we specifically focus on the case where aspects of the state space remain unmeasured. In any case, the POMDP is defined as the tuple ( S , A , r, T, T 0 , Ω , O , γ ) , where Ω is the space of possible observations, O : S × A → ∆(Ω) is the conditional distribution of seeing an observation, and the rest of the elements of the tuple remain the same as before. The objective remains the same as the MDP, but now the policy and value functions are not allowed access to the state.

Crucially, the Markov property does not hold for observations in the POMDP. That is, where o 1: t +1 := o 1 , o 2 , . . . , o t +1 are observations seen at times 1 through t +1 , o 1: t -1 ̸⊥ o t +1 | o t , a t . A naive solution to this problem is to instead have the policy take in the history of the episode so far. Of course, it is usually infeasible to learn a policy that takes in the entire history for long episodes since the space of possible histories grows exponentially with the length of the episode. Instead, one can encode the information into a more compact representation. In particular, one can use an encoder ϕ which outputs an encoding z t = ϕ ( o 1: t , a 1: t -1 , r 1: t -1 ) (note that encoders need not always take in the actions and rewards). Then, the policy and Q-value functions are augmented to take in ( o t , z t ) and ( o t , a t , z t ) , respectively.

Tracking Problems and PID Controllers. We first focus on the tracking problem, in which there are a set of signals that we wish to maintain at given reference values. For example, in espresso machines the temperature of the boiler (i.e. the signal) must be maintained at a constant reference temperature, and a controller is used to vary the boiler's on-off time so the temperature

is maintained at that value [43]. Casting tracking problems as discrete time POMDPs, we let o t = ( x (1) t , . . . , x ( M ) t , σ (1) t , . . . , σ ( M ) t ) be the observation at time t , where x ( i ) t and σ ( i ) t are the i th signal and corresponding reference value, respectively. The reward at time t is simply the negative error summed across dimensions, i.e. -∑ M m =1 ∣ ∣ ∣ x ( m ) t -σ ( m ) t ∣ ∣ ∣ .

When dealing with a single-input single-output (SISO) system (with one signal and one actuator that influences the signal), one often uses a Proportional-Integral-Derivative (PID) controller: a feedback controller that is often paired with feedforward control. This controller requires no knowledge of the dynamics, and simply sets the action via a linear combination of three terms: the error (P), the integral of the error (I), and the derivative of the error (D). When comparing other architectures to the PID controller, we will use orange colored text and blue colored text to highlight similarities between the I and D terms, respectively. Concretely, the policy corresponding to a discrete-time PID controller is defined as

where K P , K I , and K D are scalar values known as gains that must be tuned. PID controllers are designed for SISO control problems, but many real-world systems are multi-input multi-output (MIMO). In the case of MIMO tracking problems, where there are M signals with M corresponding actuators, one can control the system with M separate PID controllers. However, this assumes there is a clear breakdown of which actuator influences which signal. Additionally, there are often interactions between the different signals, which the PID controllers do not account for. Beyond tracking problems, it is less clear how to use PID controllers without substantial engineering efforts.

3 Methodology

To motivate the following, consider the task of controlling a tokamak: a toroidal device that magnetically confines plasma and is used for nuclear fusion. Nuclear fusion holds the promise of providing an energy source with few drawbacks and an abundant fuel source. As such, there has recently been a surge of interest in applying machine learning [1, 13], and especially RL [14, 45, 17, 71, 65, 64, 44], for tokamak control. However, applying deep RL has the same problems as mentioned earlier; the state is partially observable since there are aspects of the plasma's state that cannot be measured in real time, and the policy must be trained before-hand on an imperfect simulator since operation of the actual device is extremely expensive.

How should one choose a historical encoder with these challenges in mind? Previous works [50, 46] suggest using Long Short Term Memory (LSTM) [31], Gated Recurrent Units [15], or transformers [69]. These architectures have been shown to be powerful tools in natural language processing, where there exist complicated relationships between words and how they are positioned with respect to each other. However, do the same complex temporal relationships exist in something like tokamak control? The fact that PID controllers have been successfully applied for feedback control on tokamaks suggests this may not be the case [72, 27]. In reality, the extra flexibility of these architectures may become a hindrance when deployed on the physical device if they overfit to quirks in the simulator.

In this section, we present two historical encoders that we believe have good inductive biases for control. They are inspired by the PID controller in that they only sum and difference in order to combine information throughout time. Following this, in Section 5, we empirically show the benefits of these encoders on a number of control tasks including tokamak control.

The PID Encoder. Under the framework of a policy that uses a history encoder, the standard PID controller (1) is simply a linear policy with an encoder that outputs the tracking error, the integral of the tracking error, and the derivative of the tracking error. This notion can be extended to MIMO problems and arbitrary policy classes, resulting in the PID-Encoder (PIDE). Given input o 1: t , this encoder outputs a 3 M dimensional vector consisting of ( x ( m ) t -σ ( m ) t ) , ∑ t i =1 ( x ( m ) i -σ ( m ) i ) dt , and

( x ( m ) t -σ ( m ) t ) -( x ( m ) t -1 -σ ( m ) t -1 ) dt ∀ m = 1 , . . . , M . For SISO problems, policies with this encoder have access to the same information as a PID controller. However, for MIMO problems the policy has

access to all the information that each PID controller, acting in isolation, would have. Ideally a sophisticated policy would coordinate each actuator setting well.

Figure 1: Architecture for GPIDE. The diagram shows how one encoding, z t , is formed. Each of the gray, rounded boxes corresponds to one of the heads that makes up GPIDE. Each green box shows a function to be learned from data, and the orange box shows the weighted summation of all previous vectors, v h 1: t . We write the difference in observations in blue text to highlight the part of GPIDE that relates to a PID controller's D term. Note that q h 1: t and k h 1: t only play a role in this process if head h uses attention; as such, we write these terms in gray text.

The Generalized PID Encoder. A shortcoming of PIDE is that it is only applicable to tracking problems since it operates over tracking error explicitly. A more general encoder should instead accumulate information over arbitrary features of each observation. With this in mind, we introduce the Generalized-PID-Encoder (GPIDE).

GPIDE consists of a number of 'heads', each accumulating information about the history in a different manner. When there are H heads, GPIDE forms history encoding, z t , through the following:

Here, GPIDE is parameterized by θ . For head h , f h θ is a linear projection of the previous observation, action, reward, and difference between the current and previous observation to R D , and ℓ h is a weighted summation of these projections. g θ is a decoder which combines all of the information from the heads. A diagram of this process is shown in Figure 1. Note that θ is trained along with the policy and Q networks end-to-end with a gradient based optimizer.

Notice that the key aspects of the PID controller are present here. The difference in observations is explicitly taken before the linear projection f h θ . We found that this simple method works best for representing differences when the observations are scalar descriptions of the state (e.g. joint positions). Although we do not consider image observations in this work, we imagine a similar technique could be done by taking the differences in image encodings. Like the integral term of the PID, ℓ h also accumulates information over time. In the following, we consider several possibilities for ℓ h , and we will refer to these different choices as 'head types' throughout this work. We omit the superscript h below for notational convenience.

Summation. Most in line with PID, the projections can be summed, i.e. ℓ ( v 1: t ) = ∑ t i =1 v i .

Exponential Smoothing. In order to weight recent observations more heavily, exponential smoothing can be used. That is, ℓ ( v 1: t ) = (1 -α ) t -1 v 1 + ∑ t i =2 α (1 -α ) t -i v i , where 0 ≤ α ≤ 1 is the smoothing parameter. Unlike summation, this head type cannot accumulate information in the same way because it is a convex combination.

Attention. Instead of hard-coding a weighted summation of the projections, this weighting can be learned through attention [69]. Attention is one of the key components of transformers because of its ability to learn relationships between tokens. To implement this, two additional linear functions should be learned that project concatenate ( o i -1 , a i -1 , r i -1 , o i -o i -1 ) to R D . These new projections are referred to as they key and query vectors, denoted as k i and q i respectively. The softmax between their inner products is then used to form the weighting scheme for v 1: t . We can rewrite the first two

steps of GPIDE for a head that uses attention as

Here, q 1: t , k 1: t , and v 1: t are treated as t × D dimensional matrices. Since it results in a convex combination, attention has the capacity to reproduce exponential smoothing but not summation.

To anchor the GPIDE architecture back to the PID controller, we note that the P, I, and D terms can be formed exactly. At a high level, this is achieved when f h θ simply subtracts the target from the state measurement and when using summation and exponential smoothing heads with α = 1 . We write down the specific instance of GPIDE that results in these terms in Appendix A.1. While it is trivial for GPIDE to reconstruct the P, I, and D terms, it is less clear how an LSTM or GRU would achieve this, especially because of the I term. At the same time, GPIDE is much more flexible than the PID-representation since altering f h θ results in different representations at each time step and altering the type of head results in different temporal relationships.

4 Related Work

A control task may be partially observable for a myriad of reasons including unmeasured state variables [28, 75, 29], sensor noise[47], and unmeasured system parameters [76, 54]. When there are unmeasured system parameters, this is usually framed as a meta-reinforcement learning (MetaRL) [73] problem. This is a specific subclass of POMDPs where there is a collection of MDPs, and each episode, an MDP is sampled from this collection. Although these works do consider system parameters varying between episodes, the primary focus of the experiments usually tends to be on the multi-task setting (i.e. different reward functions instead of transition functions) [77, 18, 60]. We consider not only differing system parameters but also the presence of unmeasured state variables; therefore, the class of POMDPs considered in this paper is broader than the one studied in MetaRL.

Using recurrent networks has long been an approach for tackling POMDPs [29], and is still a common way to do so in a wide variety of settings [19, 73, 70, 50, 47, 75, 66, 11, 2]. Moreover implementations are publicly available both for on-policy [41, 30] and off-policy [50, 75, 11] algorithms, making it an easy pick for those wanting a quick solution. Some works [32, 77, 28, 18, 3] use recurrent networks to estimate the belief state [37], which is a distribution over the agent's true state. However, Ni et al. [50] recently showed that well-implemented, recurrent versions of SAC [26] and TD3 [23] perform competitively with many of these specialized algorithms. In either case, we believe works that estimate the belief state are not in conflict with our own since their architectures can be modified to use GPIDE instead of a recurrent unit.

Beyond recurrent networks, there has been a surge of interest in applying transformers to reinforcement learning [40]. However, we were unable to find many instances of transformers being used as history encoders in the online setting, perhaps because of their difficulty to train. Parisotto et al. [55] introduced a new architecture to remedy these difficulties; however, Melo [46] applied transformers to MetaRL and asserted that careful weight initialization is the only thing needed for stability in training. We note that GPIDE with only attention heads is similar to a single multi-headed self-attention block that appears in many transformer architectures; however, we show that attention is the least important type of head in GPIDE and often hurts performance (see Section 5.3).

Perhaps closest to our proposed architecture is PEARL [60], which does a multiplicative combination of Gaussian distributions corresponding to each state-action-reward tuple. However, their algorithm is designed for the MetaRL setting specifically. Additionally, we note that the idea of summations and averaging has been shown to be powerful in prior works. Specifically, Oliva et al. [52] introduced the Statistical Recurrent Unit, an alternative architecture to LSTMs and GRUs that leverages moving averages and performs competitively across several supervised learning tasks.

There are many facets of RL where improvements can be made to robustness, and many works focus on altering the training procedure. They use techniques such as optimizing the policy's worst-case performance [59, 36] or using variational information bottlenecking (VIB) [4] to limit the information used by the policy [42, 33, 21]. In contrast, our work specifically focuses on how architecture choices of history encoders affect robustness, but we note our developments can be used in conjunctions with

these other directions, possibly resulting in improved robustness. We perform additional experiments that consider VIB in Appendix F.1.

Lastly, we note that there is a plethora of work interested in the intersection of reinforcement learning and PID control [35, 25, 39, 22, 74, 12]. These works focus on using reinforcement learning to tune the coefficients of PID controllers (often in MIMO settings). We view these as important works on how to improve PID control using reinforcement learning; however, we view our own work as how to improve deep reinforcement learning by leveraging ideas from PID control.

5 Experiments

In this section, we experimentally compare PIDE and GPIDE against recurrent and transformer encoders. In particular, we explore the following questions:

How does the performance of a policy using PIDE or GPIDE do on tracking problems? In addition, how well can policies adapt to different system parameters and how robust to modelling error are they on these problems? (Section 5.1)

Going beyond tracking problems, how well does GPIDE perform on higher dimensional locomotion control tasks (Section 5.2)

How important is each type of head in GPIDE? (Section 5.3)

For the following tracking problems we use the Soft Actor Critic (SAC) [26] algorithm with each of the different methods for encoding observation history. Following Ni et al. [50], we make two separate instantiations of the encoders for the policy and value networks, respectively. Since the tracking problems are relatively simple, we use a small policy network consisting of 1 hidden layer with 24 units; however, we found that we still needed to use a relatively large Q network consisting of 2 hidden layers with 256 units each to solve the problems. All hyperparameters remain fixed across baselines and tracking tasks; only the history encoders change.

For the recurrent encoder, we use a GRU and follow the implementation of Ni et al. [50] closely. Our transformer encoder closely resembles the GPT2 architecture [58], and it also includes positional encodings for the observation history. For GPIDE, we use H = 6 heads: one summation head, two attention heads, and three exponential smoothing heads (with α = 0 . 25 , 0 . 5 , 1 . 0 ). This choice was not optimized, but rather was picked so that all types of heads were included and so that GPIDE has roughly the same amount of parameters as our GRU baseline. As a reference point for these RL methods, we also evaluate the performance of a tuned PID controller. Not only do PID controllers have an incredibly small number of parameters compared to the other RL-based controllers, but the training procedure is also much more straightforward since it can be posed as a black-box optimization over the returns. While there exists many sophisticated extensions of the PID controller (especially in MIMO systems [8]), we only consider the vanilla PID controller since we believe it serves as a good reference point. All methods are built on top of the rlkit library [57]. More details about implementations, hyperparameters, and computation can be found in Appendices B, C, and D, respectively. We also include additional experiments regarding variational information bottlenecking (VIB) and lookback size ablations in Appendices F.1 and F.2. Implementations can be found at https://github.com/IanChar/GPIDE .

5.1 Tracking Problems

In this subsection we consider a number of tracking problems. For each environment, the observation consists of the current signals, the reference values, and additional information about the last action made. Unless stated otherwise, the reward is as described in Section 2. More information about environments can be found in Appendix E. To make a fair comparison against PID controls, we choose to only encode the history of observations. For evaluation, we use 100 fixed settings of the environment (each setting consists of targets and system parameters). To avoid overfitting to these 100 settings, we used a separate set of 100 settings and averaged over 3 seeds when developing our methods. We evaluate policies throughout training, but report the average over the last 10% of evaluations as the final returns. We allow each policy to collect one million environment transitions, and all scores are averaged over 5 seeds. Lastly, each table shows scores formed by scaling the returns by the best and worst average returns across all methods in a particular variant of the environment, where scores of 0 and 100 correspond to the worst and best returns respectively.

Mass Spring Damper Tracking The first tracking task is the control of a classic 1D toy physics system in which there is a mass attached to a wall by a spring and damper. The goal is then to apply a force to the mass in order to move it to a given reference location. There are three system parameters to consider here: the mass, spring constant, and damping factor. We also consider the substantially more difficult problem in which there are two masses sandwiched between two walls, and the masses are connected to the walls and each other by springs and dampers (see Appendix E.1 for a diagram of this). Overall there are eight system parameters (three spring constants, three damping factors, and two masses) and two actuators (a force applied to each mass). We refer to the first problem as Mass-Spring-Damper (MSD) and the second problem as Double-Mass-Spring-Damper (DMSD).

Additionally, we test how adaptive these policies are by changing system parameters in a MetaRLtype fashion (i.e. for each episode we randomly select system parameters and then fix them for the rest of the episode). Similar to Packer et al. [54], we train the policies on three versions of the environment: one with no variation in system parameters, one with a small amount of variation, and one with a large amount of variation. We evaluate all policies on the version of the environment with large system parameter variation to test generalization capabilities.

Table 1 shows the scores achieved for each of the settings. While GRU and transformers seem to do a good job at encoding history for the MSD environment, both are significantly worse on the more complex DMSD task when compared to our proposed encoders. This is true especially for GRU, which performs worse than two independent PID controllers for every configuration. Additionally, while it seems that GRU can generalize to large amounts of variation in system parameters when a small amount is present, it fails horribly when trained on fixed system parameters. On the other hand, transformers are able to generalize surprisingly well when trained on both fixed system parameters and with small variation. We hypothesize the autoregressive nature of GRU may make it particularly susceptible to overfitting. Comparing PIDE and GPIDE, we see that PIDE tends to shine in the straightforward cases where there is little change in system parameters, whereas GPIDE is able to adapt when there is a large variation in parameters since it has additional capacity.

Navigation Environment To emulate the setting where the policy is trained on an imperfect simulator, we consider an environment in which the agent is tasked with moving itself across a surface to a specified 2D target as quickly and efficiently as possible. At every point in time, the agent can apply some force to move itself, but a penalty term proportional to the magnitude of the force is subtracted from the reward. Suppose that we have access to a simulator of the environment that is perfect except for the fact that it does not model friction between the agent and the surface. We refer to this simulator and the real environment as the 'No Friction' and 'Friction' environment, respectively. In both environments, the mass of the agent is treated as a system parameter that is sampled for each episode; however, the Friction environment has a larger range of masses and also randomly samples the coefficient of friction each episode.

Figure 2 shows the average returns recorded during training for both navigation environments and when the policies trained in No Friction are evaluated in Friction. A table of final scores can be found in Appendix G.3. One can see that GPIDE not only achieves the best returns in the environments it was trained in, but is also robust when going from the frictionless environment to the one with friction. On the other hand, PIDE has less capacity and therefore cannot achieve the same results; however, it

1

2

3

4

5

Environment Steps

1e6

14.0

13.9

13.8

13.7

13.6

Returns

No Friction

No Friction

GRU

Transformer

PIDE

GPIDE

1

2

3

4

5

Environment Steps

1e6

30

28

26

24

22

20

No Friction

Friction

1

2

3

4

5

Environment Steps

1e6

17.0

16.8

16.6

16.4

16.2

16.0

15.8

15.6

Friction

Friction

is immediately more robust than the other methods, although it begins to overfit over time. It is also clear that using GRU is less sample efficient and less robust to changes in the test environment.

Tokamak Control For our last tracking experiment we return to tokamak control. In particular, we focus on the DIII-D tokamak, a device operated by General Atomics in San Diego, California. We aim to control two quantities: β N , the normalized ratio between plasma and magnetic pressure, and rotation, i.e. how fast the plasma is spinning around the toroid. These are important quantities to track because β N serves as an approximate economic indicator and rotation control of the plasma has been suggested to be key for stability [7, 67, 10, 61, 56]. The policy has control over the eight neutral beams [24], which are able to inject power and torque by blasting neutrally charged particles into the plasma. Importantly, two of the eight beams can be oriented in the opposite direction from the others, which decouples the total combined power and torque to some extent (see Figure 3).

To emulate the sim-to-real training experience, we create a simulator based on the equations described in Boyer et al. [9] and Scoville et al. [63]. This simulator has two major shortcomings: it assumes that certain states of the plasma (e.g. its shape) are fixed for entire episodes, and it assumes that there are no events that cause loss of confinement of the plasma. We make up for part of the former by randomly sampling plasma states each episode. The approximate

Table 3: PyBullet Task Results. Each score is averaged over four seeds and we report the standard errors. Unlike before, we scale the returns by the returns of an oracle policy (i.e. one which sees position and velocity) and a policy which does not encode any history. For the environment names, 'P' and 'V' denote only position or only velocity in the observation, resepctively.

'real' environment addresses these shortcomings by using a data-driven simulator. This approach to simulating has been shown to be relatively accurate [14, 65, 64, 1], and we use an adapted version of the simulator appearing in Char et al. [14] for our work. This simulator accounts for a greater set of the plasma's state, and the additional information is rich enough that loss of confinement events play a role in the dynamics.

We consider two versions of this task: the first is a SISO task where total power is controlled to achieve a β N target, and the second is a MIMO task where total power and torque is controlled to achieve β N and rotation targets. The results for both of these tasks are shown in Table 2. Most of the RL techniques are able to do well if tested in the same environment they were trained in; the exception of this is PIDE, which curiously is unable to perform well in the simulator environment. While no reinforcement learning method matches the robustness of a PID controller, policies trained with GPIDE fare significantly better.

5.2 PyBullet Locomotion Tasks

Moving past tracking problems, we evaluate GPIDE on the PyBullet [16] benchmark proposed by Han et al. [28] and adapted in Ni et al. [50]. The benchmark has four robots: halfcheetah, hopper, walker, and ant. For each of these, either the current position information or velocity information is hidden from the agent. Except for GPIDE and transformer encoder, we use all of the performance traces given by Ni et al. [50]. In addition to SAC, they also train using PPO [62], A2C [49], TD3 [23], and VRM [28], a variational method that uses recurrent units to estimate the belief state. We reproduce as much of the training and evaluation procedure as possible, including using the same hyperparameters in the SAC algorithm and giving the history encoders access to actions and rewards. For more information see Appendix C.2. Table 3 shows the performance of GPIDE along with a subset of best performing methods (more results can be found in Appendix G.5). These results make it clear that GPIDE is powerful in arbitrary control tasks besides tracking since the average score achieved across all tasks is a 73% improvement over TD3-GRU, which we believe is the previous state-of-the-art for this benchmark at the time of this work.

Moreover, GPIDE dominates performance for every robot except HalfCheetah. The only setting where GPIDE achieves significantly worse performance is HalfCheetah-V. This setting seemed to cause stability issues in some seeds, lowering the average score. We believe that these stability issues stem from the attention heads, and we found that removing these heads fixed stability issues and resulted in a competitive average score (see Appendix G.5).

5.3 GPIDE Ablations

To investigate the role of each type of head, we reran all experiments with three variants of GPIDE: one with six exponential smoothing heads (ES), one with five exponential smoothing heads and

one summation head (ES + Sum), and one with six attention heads (see Appendix C.3 for details). We choose these three configurations specifically to better understand the roles that attention and summation play.

Table 4 shows the differences in the average scores for each environment. The first notable takeaway is that having summation is often important in some of the more complex environments. The other takeaway is that much of the heavy lifting is being done by the exponential smoothing. GPIDE fares far worse when only having attention heads, especially in DMSD and the PyBullet environments.

We visualize some of the attention schemes learned by GPIDE for MSD with small variation and HalfCheetah (Figure 4). While the attention scheme learned for MSD could potentially be useful since it recalls information from near the beginning of the episode when the most movement is happening, it appears that the attention scheme for HalfCheetah is simply a poor reproduction of exponential smoothing, making

0

50

Time Step

0

20

40

60

80

Sequence Length

MSD-Small

0

25

50

Time Step

0

20

40

60

HalfCheetah-P

it redundant and suboptimal. In fact, we found this phenomenon to be true across all attention heads and PyBullet tasks. We believe that the periodicity that appears here is due to the oscillatory nature of the problem and lack of positional encoding (although we found including positional encoding degrades performance).

6 Discussion

In this work, we introduced the PIDE and GPIDE history encoders to be used for reinforcement learning in partially observable control tasks. Although both are far simpler than prior methods of encoding, they often result in powerful yet robust controllers. We hope that this work inspires the research community to think about how pre-existing control methods can inform architecture choices.

Limitations There are many different ways a control task may be partially observable, and we do not believe that our proposed methods are solutions to all of them. For example, we do not think GPIDE is necessarily suited for tasks where the agent needs to remember events (e.g. picking up a key to unlock a door).

As with any bias, the PID-inspired biases that we propose in this work come at the cost of flexibility. For the experiments considered in this work, this trade off is beneficial and results in better policies. However, it is unclear whether this trade off is always worth making. It is possible that in higher dimensional environments or environments with more complex dynamics that having more flexibility is preferable to our proposed architecture.

Lastly, some tasks may require the policy to act on images as observations. We are optimistic that PIDE and GPIDE are still useful architectures in this setting, but we speculate that this is contingent on training an image encoder that is well-suited for these architectures, and we leave this research direction for future work.

References

Joseph Abbate, R Conlin, and E Kolemen. Data-driven profile prediction for diii-d. Nuclear Fusion , 61(4):046027, 2021.

Ananye Agarwal, Ashish Kumar, Jitendra Malik, and Deepak Pathak. Legged locomotion in challenging terrains using egocentric vision. In Conference on Robot Learning , pages 403-415. PMLR, 2023.

Kei Akuzawa, Yusuke Iwasawa, and Yutaka Matsuo. Estimating disentangled belief about hidden state and hidden task for meta-reinforcement learning. In Learning for Dynamics and Control , pages 73-86. PMLR, 2021.

Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410 , 2016.

Lucas Alpi. Pid control: Breaking the time barrier, November 2019. URL https://www.novusautomation.com/en/article_PID_control#:~:text=In% 201911%20the%20first%20PID,still%20widely%20used%20in%20automation.

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

Laszlo Bardoczi, NC Logan, and EJ Strait. Neoclassical tearing mode seeding by nonlinear three-wave interactions in tokamaks. Physical Review Letters , 127(5):055002, 2021.

Stephen Boyd, Martin Hast, and Karl Johan Åström. Mimo pid tuning via iterated lmi restriction. International Journal of Robust and Nonlinear Control , 26(8):1718-1731, 2016.

MDBoyer, KG Erickson, BA Grierson, DC Pace, JT Scoville, J Rauch, BJ Crowley, JR Ferron, SR Haskey, DA Humphreys, et al. Feedback control of stored energy and rotation with variable beam energy and perveance on diii-d. Nuclear Fusion , 59(7):076004, 2019.

RJ Buttery, RJ La Haye, P Gohil, GL Jackson, H Reimerdes, EJ Strait, and DIII-D Team. The influence of rotation on the β n threshold for the 2/ 1 neoclassical tearing mode in diii-d. Physics of Plasmas , 15(5):056115, 2008.

Massimo Caccia, Jonas Mueller, Taesup Kim, Laurent Charlin, and Rasool Fakoor. Taskagnostic continual reinforcement learning: In praise of a simple baseline. arXiv preprint arXiv:2205.14495 , 2022.

Ignacio Carlucho, Mariano De Paula, and Gerardo G Acosta. An adaptive deep reinforcement learning approach for mimo pid control of mobile robots. ISA transactions , 102:280-294, 2020.

Ian Char, Youngseog Chung, Willie Neiswanger, Kirthevasan Kandasamy, Andrew O Nelson, Mark Boyer, Egemen Kolemen, and Jeff Schneider. Offline contextual bayesian optimization. Advances in Neural Information Processing Systems , 32, 2019.

Ian Char, Joseph Abbate, László Bardóczi, Mark Boyer, Youngseog Chung, Rory Conlin, Keith Erickson, Viraj Mehta, Nathan Richner, Egemen Kolemen, et al. Offline model-based reinforcement learning for tokamak control. In Learning for Dynamics and Control Conference , pages 1357-1372. PMLR, 2023.

Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoderdecoder for statistical machine translation. arXiv preprint arXiv:1406.1078 , 2014.

Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. http://pybullet.org , 2016-2021.

Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature , 602(7897): 414-419, 2022.

Ron Dorfman, Idan Shenfeld, and Aviv Tamar. Offline meta learning of exploration. arXiv preprint arXiv:2008.02598 , 2020.

Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. RL 2 : Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.02779 , 2016.

Gabriel Dulac-Arnold, Nir Levine, Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning , 110(9):2419-2468, 2021.

Ben Eysenbach, Russ R Salakhutdinov, and Sergey Levine. Robust predictable control. Advances in Neural Information Processing Systems , 34:27813-27825, 2021.

Fumitake Fujii, Akinori Kaneishi, Takafumi Nii, Ryu'ichiro Maenishi, and Soma Tanaka. Selftuning two degree-of-freedom proportional-integral control system based on reinforcement learning for a multiple-input multiple-output industrial process that suffers from spatial input coupling. Processes , 9(3):487, 2021.

Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning , pages 1587-1596. PMLR, 2018.

BA Grierson, MA Van Zeeland, JT Scoville, B Crowley, I Bykov, JM Park, WW Heidbrink, A Nagy, SR Haskey, and D Liu. Testing the diii-d co/counter off-axis neutral beam injected power and ability to balance injected torque. Nuclear Fusion , 61(11):116049, 2021.

Zhe Guan and Toru Yamamoto. Design of a reinforcement learning pid controller. IEEJ Transactions on Electrical and Electronic Engineering , 16(10):1354-1360, 2021.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Sang-hee Hahn, YJ Kim, BG Penaflor, JG Bak, H Han, JS Hong, YM Jeon, JH Jeong, M Joung, JW Juhn, et al. Progress and plan of kstar plasma control system upgrade. Fusion Engineering and Design , 112:687-691, 2016.

Dongqi Han, Kenji Doya, and Jun Tani. Variational recurrent models for solving partially observable control tasks. arXiv preprint arXiv:1912.10703 , 2019.

Nicolas Heess, Jonathan J Hunt, Timothy P Lillicrap, and David Silver. Memory-based control with recurrent neural networks. arXiv preprint arXiv:1512.04455 , 2015.

Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines. https: //github.com/hill-a/stable-baselines , 2018.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8): 1735-1780, 1997.

Maximilian Igl, Luisa Zintgraf, Tuan Anh Le, Frank Wood, and Shimon Whiteson. Deep variational reinforcement learning for pomdps. In International Conference on Machine Learning , pages 2117-2126. PMLR, 2018.

Maximilian Igl, Kamil Ciosek, Yingzhen Li, Sebastian Tschiatschek, Cheng Zhang, Sam Devlin, and Katja Hofmann. Generalization in reinforcement learning with selective noise injection and information bottleneck. Advances in neural information processing systems , 32, 2019.

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning , pages 448-456. pmlr, 2015.

Hozefa Jesawada, Amol Yerudkar, Carmen Del Vecchio, and Navdeep Singh. A model-based reinforcement learning approach for pid design. arXiv preprint arXiv:2206.03567 , 2022.

Yuankun Jiang, Chenglin Li, Wenrui Dai, Junni Zou, and Hongkai Xiong. Monotonic robust policy optimization with model discrepancy. In International Conference on Machine Learning , pages 4951-4960. PMLR, 2021.

Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence , 101(1-2):99-134, 1998.

Andrej Karpathy. nanogpt, 2022-. URL https://github.com/karpathy/nanoGPT .

Nathan P Lawrence, Gregory E Stewart, Philip D Loewen, Michael G Forbes, Johan U Backstrom, and R Bhushan Gopaluni. Reinforcement learning based design of linear fixed structure controllers. IFAC-PapersOnLine , 53(2):230-235, 2020.

Wenzhe Li, Hao Luo, Zichuan Lin, Chongjie Zhang, Zongqing Lu, and Deheng Ye. A survey on transformers in reinforcement learning. arXiv preprint arXiv:2301.03044 , 2023.

Eric Liang, Richard Liaw, Philipp Moritz, Robert Nishihara, Roy Fox, Ken Goldberg, Joseph E Gonzalez, Michael I Jordan, and Ion Stoica. Rllib: Abstractions for distributed reinforcement learning. arxiv e-prints, page. arXiv preprint arXiv:1712.09381 , 2017.

Xingyu Lu, Kimin Lee, Pieter Abbeel, and Stas Tiomkin. Dynamics generalization via information bottleneck in deep reinforcement learning. arXiv preprint arXiv:2008.00614 , 2020.

La Marzocco. A brief history of the pid, October 2015. URL https://home.lamarzoccousa. com/history-of-the-pid/ .

Viraj Mehta, Biswajit Paria, Jeff Schneider, Stefano Ermon, and Willie Neiswanger. An experimental design perspective on model-based reinforcement learning. arXiv preprint arXiv:2112.05244 , 2021.

Viraj Mehta, Ian Char, Joseph Abbate, Rory Conlin, Mark D Boyer, Stefano Ermon, Jeff Schneider, and Willie Neiswanger. Exploration via planning for information about the optimal trajectory. arXiv preprint arXiv:2210.04642 , 2022.

Luckeciano C Melo. Transformers are meta-reinforcement learners. In International Conference on Machine Learning , pages 15340-15359. PMLR, 2022.

Lingheng Meng, Rob Gorbet, and Dana Kuli´ c. Memory-based deep reinforcement learning for pomdps. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 5619-5626. IEEE, 2021.

Nicolas Minorsky. Directional stability of automatically steered bodies. Journal of the American Society for Naval Engineers , 34(2):280-309, 1922.

Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning , pages 1928-1937. PMLR, 2016.

Tianwei Ni, Benjamin Eysenbach, and Ruslan Salakhutdinov. Recurrent model-free RL can be a strong baseline for many POMDPs. In International Conference on Machine Learning , pages 16691-16723. PMLR, 2022.

Fernando Nogueira. Bayesian Optimization: Open source constrained global optimization tool for Python, 2014-. URL https://github.com/fmfn/BayesianOptimization .

Junier B Oliva, Barnabás Póczos, and Jeff Schneider. The statistical recurrent unit. In International Conference on Machine Learning , pages 2671-2680. PMLR, 2017.

OpenAI. Gpt-4 technical report. arXiv , 2023.

Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song. Assessing generalization in deep reinforcement learning. arXiv preprint arXiv:1810.12282 , 2018.

Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, et al. Stabilizing transformers for reinforcement learning. In International conference on machine learning , pages 7487-7498. PMLR, 2020.

PA Politzer, CC Petty, RJ Jayakumar, TC Luce, MR Wade, JC DeBoo, JR Ferron, P Gohil, CT Holcomb, AW Hyatt, et al. Influence of toroidal rotation on transport and stability in hybrid scenario plasmas in diii-d. Nuclear Fusion , 48(7):075001, 2008.

Vitchyr Pong and Ashvin Nair. rlkit. https://github.com/rail-berkeley/rlkit , 2018-.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine. Epopt: Learning robust neural network policies using model ensembles. arXiv preprint arXiv:1610.01283 , 2016.

Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In International conference on machine learning , pages 5331-5340. PMLR, 2019.

H Reimerdes, AM Garofalo, GL Jackson, M Okabayashi, EJ Strait, Ming-Sheng Chu, Y In, RJ La Haye, MJ Lanctot, YQ Liu, et al. Reduced critical rotation for resistive-wall mode stabilization in a near-axisymmetric configuration. Physical review letters , 98(5):055001, 2007.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

JT Scoville, DA Humphreys, JR Ferron, and P Gohil. Simultaneous feedback control of plasma rotation and stored energy on the diii-d tokamak. Fusion engineering and design , 82(5-14): 1045-1050, 2007.

J Seo, Y-S Na, B Kim, CY Lee, MS Park, SJ Park, and YH Lee. Development of an operation trajectory design algorithm for control of multiple 0d parameters using deep reinforcement learning in kstar. Nuclear Fusion , 62(8):086049, 2022.

Jaemin Seo, Y-S Na, B Kim, CY Lee, MS Park, SJ Park, and YH Lee. Feedforward beta control in the kstar tokamak by deep reinforcement learning. Nuclear Fusion , 61(10):106010, 2021.

Open Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michael Mathieu, et al. Openended learning leads to generally capable agents. arXiv preprint arXiv:2107.12808 , 2021.

B Tobias, M Chen, IGJ Classen, CW Domier, R Fitzpatrick, BA Grierson, NC Luhmann Jr, CMMuscatello, M Okabayashi, KEJ Olofsson, et al. Rotation profile flattening and toroidal flow shear reversal due to the coupling of magnetic islands in tokamaks. Physics of Plasmas , 23 (5):056107, 2016.

ITER Physics Expert Group on Confinement Transport, , ITER Physics Expert Group on Confinement Modelling Database, , and ITER Physics Basis Editors. Chapter 2: Plasma confinement and transport. Nuclear Fusion , 39(12):2175-2249, 1999.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature , 575(7782):350-354, 2019.

T Wakatsuki, T Suzuki, N Oyama, and N Hayashi. Ion temperature gradient control using reinforcement learning technique. Nuclear Fusion , 61(4):046036, 2021.

ML Walker, DA Humphreys, JA Leuer, JR Ferron, and BG Penaflor. Implementation of model-based multivariable control on diii-d. GA-A23468 , 2000.

Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint arXiv:1611.05763 , 2016.

Junhang Wang, Jianing Zhu, Chao Zou, Linlin Ou, and Xinyi Yu. Robust adaptive pid control based on reinforcement learning for mimo nonlinear six-joint manipulator. In 2021 China Automation Congress (CAC) , pages 2633-2638. IEEE, 2021.

Zhihan Yang and Hai Huu Nguyen. Recurrent off-policy baselines for memory-based continuous control. In Deep RL Workshop NeurIPS 2021 , 2021.

Wenhao Yu, Jie Tan, C Karen Liu, and Greg Turk. Preparing for the unknown: Learning a universal policy with online system identification. arXiv preprint arXiv:1702.02453 , 2017.

Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep rl via metalearning. arXiv preprint arXiv:1910.08348 , 2019.

Appendices

Contents

A Additional GPIDE Details

A.1 Forming the P, I, and D Features with GPIDE

As mentioned in Section 3, the P, I, and D features associated with a PID controller can be easily formed with GPIDE, and in this Appendix we show this concretely. Assume that the observations take the form o t = ( x (1) t , . . . , x ( M ) t , σ (1) t , . . . , σ ( M ) t ) , and assume that the actions and rewards are not given to GPIDE for simplicity. Then the linear projection for each head, f h θ , takes in x (1) t -1 , . . . , x ( M ) t -1 , σ (1) t -1 , . . . , σ ( M ) t -1 , ( x (1) t -x (1) t -1 ) , . . . , ( x ( M ) t -x ( M ) t -1 ) , ( σ (1) t -σ (1) t -1 ) , . . . , ( σ ( M ) t -σ ( M ) t -1 ) at each time step t . We use a GPIDE architecture with three heads and where g θ is the identity function. The linear projections for each head is as follows:

Note that f 1 θ and f 2 θ form the current error at time t (i.e. the P term), and f 3 θ forms the change in error (i.e. the D term). For accumulation strategies, using exponential smoothing with α = 1 for the first and third heads and a summation head for the second head will recover the P, I, and D terms for heads 1, 2, and 3, respectively. Note that the above assumes dt = 1 , but the linear projections can be adjusted to take different dt values into account.

B Implementation Details

Code Release All code for implementations are provided in the supplemental material along with instructions for how to run experiments. The code can also be found at https://github.com/ IanChar/GPIDE . The only experiment that cannot be run are the 'real' cases for tokamak control.

Architecture We use the same general architecture for each of the RL methods in this paper (see Figure 5). Each input to the history encoders, policy functions, and Q -value functions have corresponding encoders. This setup closely follows what was done in Ni et al. [50]. The encoders are simply linear projections; however, in the case of our GRU history encoder we do linear projections followed by a ReLU activation (as done in Ni et al. [50]). Although hypothetically the policy only needs to take in history encoding, z t , since int includes the current observation, we found it essential for the current observation to be passed in independently and have its own encoder.

Shortcut

Encoder

Observation

Encoder

Action

Encoder

Reward

Encoder

Transition

Encoder

GPIDE / Transformer

Shortcut

Encoder

Observation

Encoder

Action

Encoder

Reward

Encoder

Transition

Encoder

GPIDE / Transformer

B.1 GPIDE Implementation Details

In addition to what is mentioned in Section 3, we found that there were several choices that helped with training. First, there may be some scaling issues because o t -o t -1 may be small or the result of summation type heads may result in large encodings. To account for this, we use batch normalization layers [34] before each input encoding and after each ℓ h .

There are very few nonlinear components of GPIDE. The only one that remains constant across all experiments is that a tanh activation is used for the final output of the encoder. For tracking tasks, the decoder g θ has 1 hidden layer with 64 units and uses a ReLU activation function. For PyBullet tasks, g θ is a linear function.

B.2 Recurrent and Transformer Baseline Details

Recurrent Encoder. For the recurrent encoder, we tried to match as many details as Ni et al. [50] as possible. We double checked our implementation against theirs and confirmed that it achieves similar performance.

Transformer Encoder. We follow the GPT2 architecture [58] for inspiration, and particularly the code provided in Karpathy [38]. In particular, we use a number of multi-headed self-attention blocks in sequence with residual connections. We use layer normalization [6] before multi-headed attention and out projections; however, we do not use dropout. The out projection for each multi-headed self-attention block has one hidden layer with four times the number of units as the embedding dimension. Although Melo [46] suggests using T-Fixup weight initialization, we found that more reliably high performance was achieved with the weight initialization of Radford et al. [58]. Lastly, we used the same representation for the history as GPIDE, i.e. ( o t -1 , a t -1 , r t -1 , o t -o t -1 ) , since it results in better performance.

B.3 PID Baseline

To tune our PID baseline, we used Bayesian Optimization over the three (for SISO) or six (for MIMO) dimensional space. Specifically we use the library provided by Nogueira [51]. The output of the blackbox optimization is the average over 100 different settings (independent from the 100 settings used for testing). We allow the optimization procedure to collect as many samples as the RL methods. The final performance reported uses the PID controller with the best gains found during the

optimization procedure. The bounds for each of the tracking tasks were eyeballed to be appropriate, which potentially preferably skews performance.

C Hyperparameters

Because of resource restrictions, we were unable to do full hyperparameter tuning for each benchmark presented in this paper. Instead, we focused on ensuring that all history encoding methods were roughly comparable, e.g. dimension of encoding, number of parameters, etc. Tables 5 show selected hyperparameters, and the following subsections describe how an important subset of these hyperparameters were picked. Any tuning that was done was over three seeds using 100 fixed settings (different from the 100 settings used for testing).

C.1 Hyperparamters for Tracking Tasks

For tracking tasks, we tried using a history encoding size of 32 and 64 for GRU, and we found that performance was better with 64. This is surprising since PIDE can perform well in these environments even though its history encoding is much smaller (3 or 6 dimensional). To make it a fair comparison, we set the history encoding dimension for GPIDE and transformer to be 64 as well. We use one layer for GRU. For the transformer-specific hyperparameters we choose half of what appears in the PyBullet tasks.

C.2 Hyperparameters for PyBullet Task

For the PyBullet tasks, we simply tried to emulate most of the hyperparameters found in Ni et al. [50]. For the transformer, we choose to use similar hyperparameters to those found in Melo [46]. However, we found that, unlike the tracking tasks, positional encoding hurts performance. As such, we do not include it for PyBullet experiments.

C.3 Hyperparameters for Ablations

For the ablations of GPIDE, we use α = 0 . 01 , 0 . 1 , 0 . 25 , 0 . 5 , 0 . 9 , 1 . 0 for the smoothing parameters when only exponential smoothing is used. When using exponential smoothing and summation, the α = 0 . 01 head is replaced with a summation head. The attention version of GPIDE replaces all six of these heads with attention.

D Computation Details

We used an internal cluster of machines to run these experiments. We mostly leveraged Nvidia Titan X GPUs for this, but also used a few Nvidia GTX 1080s. It is difficult to get an accurate estimate of run time since job loads vary drastically on our cluster from other users. However, to train a single policy on DMSD to completion (1 million transitions collected, or 1,000 epochs) using PIDE takes roughly 4.5 hours, using GPIDE takes roughly 17.25 hours, using a GRU takes roughly 14.5 hours, and using a transformer takes roughly 21 hours. This is similar for other tracking tasks. For PyBullet tasks, using GPIDE took roughly 43.2 hours and using a transformer took roughly 64.2 hours. We note that our implementation of GPIDE is somewhat naive and could be vastly improved. In particular, for exponential smoothing and summation heads, w t can be cached to save on compute, which is not being done currently. This is a big advantage in efficiency that GPIDE (especially one without attention heads) has over transformers.

E Environment Descriptions

E.1 Mass Spring Damper

MSD

DMSD

For both MSD and DMSD, the observations include the current mass position(s), the target reference position(s), and the last action played. Each episode lasts for 100 time steps. For all RL methods, the action is a difference in force applied to the mass, but for the PID the action is simply the force

to be applied to the mass at that time. The force is bounded between -10 and 10 N for MSD and -30 and 30 N for DMSD. Each episode, system parameters are drawn from a uniform distribution with bounds shown in Table 10 (they are the same for both MSD and DMSD). Targets are drawn to uniformly at random to be -1 . 5 to 1 . 5 m offset from the masses' resting positions.

E.2 Navigation Environment

Like the MSD and DMSD environments, the navigation experiment lasts 100 time steps each episode. Additionally, the observation includes position signal, target locations, and the last action. For all methods we set the action to be the change in force, and the total amount of force is bounded between -10 and 10 N . The penalty on the reward is equal to 0 . 01 times the magnitude of the change in force. In addition, the maximum magnitude of the velocity for the agent is bounded by 1 . 0 m/s . The agent always starts at the location (0 , 0) , and the target is picked uniformly at random to be within a box of length 10 centered around the origin.

Every episode, the mass, kinetic friction coefficient, and static friction coefficient is sampled, The friction is sampled by first sampling the total amount of friction in the system, and then sampling what proportion of the total friction is static friction. All distributions for the system parameters are uniform, and we show the bounds in Table 11.

E.3 Tokamak Control Environment

Simulator Our simulator version of the tokamak control is inspired by equations used by Boyer et al. [9], Scoville et al. [63]. In particular, we use the following relations for stored energy, E , and rotation, v rot :

where P is the total power, T is the total torque, τ E is the energy confinement time, τ m is the momentum confinement time, and C rot is a quantity relying on the ion density and major radius of the plasma. We treat τ m and C rot as constants with values of 0 . 1 and 80 . 0 respectively.

Webase the energy confinement time off of the ITERH-98 scaling [68]. This uses many measurements of the plasma, but we focus on a subset of these and treat the rest as constants. In particular,

where C E is a constant value we set to be 200 , I is the plasma current, and B is the toroidal magnetic field. To relate the stored energy to β N we use the rough approximation

2000

4000

6000

8000

10000

12000

14000

Total Power (kW)

0

2

4

6

8

Total Torque (Nm)

where C β is a constant we set to be 5 , and a is the minor radius of the plasma. For a , I , and B , we sample these from the distribution described in Table 12 for each episode. Lastly, we add momentum to the stored energy. That is, the stored energy derivative at time t , ˙ E t , is

The actions for all control methods is the amount of change for the power and torque. Because the total amount of power and torque injected rely on the beams, they are not totally disentangled. In Figure 7, we show the bounds for the action space. Furthermore, we bound the amount that power and torque can be changed by roughly 40 MW/s and 35 Nm/s , respectively. Each step is 0 . 025 seconds.

Each episode lasts for 100 increments of 0 . 025 seconds. The observations are the current β N and rotation values, their reference values, and the current power and torque settings. We make the initial β N and rotation relatively small in order to simulate the plasma ramping up. We let the β N and rotation targets be distributed as U (1 . 75 , 2 . 75) and U (25 . 0 , 50 . 0) rad/s , respectively.

'Real' For the real versions of the tokamak control experiments, most of the previous (such as action bounds and target distributions) stays the same. This data-driven simulator is based on the one from Char et al. [14], and we refer the reader there for more details. That being said, there are some differences between the architecture presented there and the one used in this work. Our network is a recurrent network that uses a GRU, has four hidden layers with 512 units each, and outputs the mean and log variance of a normal distribution describing how β N and rotation will change. In addition to power and torque, it takes in measurements for the plasma current, the toroidal magnetic field, n1rms (a measurement related to the plasma' stability), and 13 other actuator requests for gas control and plasma shaping. In addition to sampling from the normal distribution outputted by the network, we train an ensemble of ten networks, and an ensemble member is selected every episode. We use five of

these models during training and the other five during testing. Along with an ensemble member being sampled each episode, we also sample a historical run, which determines the starting conditions of the plasma and how the other inputs to the neural network which are not modelled evolve over time. Recall that 100 fixed settings are used to evaluate the policy every epoch of training. In this case, a setting consists of targets, an ensemble member, and a historical run.

F Additional Experiments

F.1 Experiments Using VIB + GRU

As shown in this work, using a GRU for a history encoder often results in a policy that is ill-equipped to handle changes in the dynamics not seen at train time. One may wonder whether using other robust RL techniques is able to mask this inadequacy of GRU. To test this, we look at adding Variational Information Bottlenecking (VIB) to our GRU baseline [4]. Previous works applying this concept to RL usually do not consider the same class of POMDPs as us [42, 33]; however, Eysenbach et al. [21] does have a baseline that uses VIB with a recurrent policy.

To use VIB with RL, we alter the policy network so that it encodes input to a latent random variable, and the decodes into an action. Following the notation of Lu et al. [42], let this latent random variable be Z and the random variable representing the input of the network be S . The goal is to learn a policy that maximizes J ( π ) subject to I ( Z, S ) ≤ I C , where I ( Z, S ) is the mutual information between Z and S , and I C is some given threshold. In practice, we optimize the Lagrangian. Where β is a Lagrangian multiplier, p ( Z | S ) is the conditional density of Z outputted by the encoder, and q ( Z ) is the prior, the penalizer is -β E S [ D KL ( p ( Z | S ) || q ( Z ))] . Like other works, we assume that q ( Z ) is a standard multivarite normal.

We alter our GRU baseline for tracking tasks so that the policy uses VIB. This is not entirely straightforward since our policy network is already quite small. We choose to keep as close to original policy architecture as possible and set the dimension of the latent variable, Z , to be 24. Note that this change has no affect on the history encoder; this only affects the policy network. For our experiments, we set β = 0 . 1 , but we note that we may be able to achieve better performance through more careful tuning or annealing of β .

In any case, we do see that VIB helps with robustness in many instances (see Table 13). However, the cases where there are improvements are instances where the GRU policy already did a good job at generalizing to the test environment. These are primarily the MSD and DMSD environments where the system parameters drawn during training time are simply a subset of those drawn during testing time (interestingly, this notion of dynamics generalization matches the set up of the experiments presented in Lu et al. [42]). Surprisingly, in the navigation and tokamak control experiments, where there are more complex differences between the train and test environments, VIB can sometimes hurt the final performance.

F.2 Lookback Size Ablations

To better understand the role of the maximum lookback size (i.e. the amount of history used to form the encoding) of GPIDE, we repeat the PyBullet experiments using a lookback size of 4, 16, 64, and 128 with and without attention (labelled GPIDE and GPIDE-ESS, respectively). Figures 8 and 9 show performance curves for GPIDE and GPIDE-ESS respectively. It is clear that there is a massive increase in improvement when expanding the maximum size of the lookback from 4 to 16. For the most part, this trend continues expanding the lookback size from 16 to 64; however, it seems pushing from 64 to 128 yields mixed results. For some tasks, such as HalfCheetah-P, expanding the lookback to 128 results in noticeable improvements both with and without attention. For other tasks, such as Hopper-V, this expansion yields slightly worse performance, possibly because of training stability issues.

One interesting observation is that, when attention is included, this decrease in performance from expanding lookback can occur when increasing from 16 to 64 (see HalfCheetah-V and Walker-V). At the same time, however, it appears the GPIDE can sometimes maintain good performance when expanding the lookback to 128 when GPIDE-ESS cannot (Walker-P).

0.0

0.5

1.0

1.5

1e6

0

1000

2000

3000

Returns

HalfCheetahBLT-P-v0

0.0

0.5

1.0

1.5

1e6

0

1000

2000

HopperBLT-P-v0

4

16

64

128

Oracle

0.0

0.5

1.0

1.5

1e6

0

500

1000

1500

2000

2500

WalkerBLT-P-v0

0.0

0.5

1.0

1.5

1e6

1000

2000

3000

AntBLT-P-v0

0.0

0.5

1.0

1.5

Environment Steps

1e6

1000

0

1000

2000

3000

Returns

HalfCheetahBLT-V-v0

0.0

0.5

1.0

1.5

Environment Steps

1e6

0

500

1000

1500

2000

2500

HopperBLT-V-v0

0.0

0.5

1.0

1.5

Environment Steps

1e6

0

500

1000

1500

2000

WalkerBLT-V-v0

0.0

0.5

1.0

1.5

Environment Steps

1e6

1000

2000

3000

GPIDE

0.0

0.5

1.0

1.5

1e6

0

500

1000

1500

2000

2500

3000

Returns

HalfCheetahBLT-P-v0

0.0

0.5

1.0

1.5

1e6

0

500

1000

1500

2000

2500

3000

HopperBLT-P-v0

4

16

64

128

Oracle

0.0

0.5

1.0

1.5

1e6

0

500

1000

1500

2000

2500

WalkerBLT-P-v0

0.0

0.5

1.0

1.5

1e6

500

1000

1500

2000

2500

3000

3500

AntBLT-P-v0

0.0

0.5

1.0

1.5

Environment Steps

1e6

1000

0

1000

2000

3000

Returns

HalfCheetahBLT-V-v0

0.0

0.5

1.0

1.5

Environment Steps

1e6

0

500

1000

1500

2000

2500

HopperBLT-V-v0

0.0

0.5

1.0

1.5

Environment Steps

1e6

0

500

1000

1500

2000

WalkerBLT-V-v0

0.0

0.5

1.0

1.5

Environment Steps

1e6

500

1000

1500

2000

2500

3000

3500

GPIDE-ESS

G Further Results

In this Appendix, we give further evaluation of the evaluation procedure. In addition, we give full tables of results for normalized and unnormalized scores for all methods. We also show performance

traces. Note that the percentage changes in Table 4 do not necessarily reflect tables in this section since they report all combinations of environment variants.

G.1 Evaluation Procedure

As stated in the main paper, for tracking tasks, we fix 100 settings (each comprised of targets, start state, and system parameters) that are used to evaluate the policy for every epoch of training (i.e. for every epoch the evaluation returns is the average over all 100 settings returns). We use a separate 100 settings when tuning. For the final returns, we average over the last 10% of recorded evaluations.

For the PyBullet tasks, we use ten different rollouts for evaluation following Ni et al. [50]. We also average over the last 20% of recorded evaluations like they do.

Normalized Table Scores. We now give an in-depth explanation of how the scores in the table are computed. Let π ( b,i ) be the policy trained with baseline method b (e.g. with GPIDE, transformer, or GRU encoder) on environment variant i (e.g. fixed, small, or large). Let J j ( π ( b,i ) ) be the evaluation of policy π ( b,i ) on environment variant j , i.e. the average returns over all seeds and episodes. The normalized score for policy π ( b,i ) on variant j is then

Note that we only min and max over baseline methods presented in the table.

For PyBullet tasks, we do the same procedure but normalize by the oracle policy's performance (sees both position and velocity and has no history encoder) and the Markovian policy's performance (sees only position or velocity and has no history encoder). For both of these policies, we use what was reported from Ni et al. [50]. Note the our normalized scores differ slightly from those used in Ni et al. [50] since they normalize based on the best and worst returns of any policy; however, we believe our scheme gives a more intuitive picture of how any given policy is performing.

G.2 MSD and DMSD Results

0.0

0.5

1.0

1e6

6.25

6.00

5.75

Fixed

Fixed

PID

GPIDE

GRU

Transformer

PIDE

0.0

0.5

1.0

1e6

8.0

7.5

Small

0.5

1.0

1e6

14

12

Large

0.5

1.0

1e6

6.25

6.00

5.75

Small

0.5

1.0

1e6

7.5

7.0

0.5

1.0

1e6

11

10

0.5

1.0

Environment Steps

1e6

7.0

6.5

6.0

Large

0.5

1.0

Environment Steps

1e6

8.0

7.5

0.5

1.0

Environment Steps

1e6

12

11

10

0.0

0.5

1.0

1e6

6.5

6.0

Fixed

Fixed

PID

GPIDE-Attn

GPIDE-ESS

GPIDE-ES

0.0

0.5

1.0

1e6

7.6

7.4

Small

0.0

0.5

1.0

1e6

14

12

Large

0.0

0.5

1.0

1e6

6.5

6.0

Small

0.0

0.5

1.0

1e6

7.75

7.50

7.25

0.0

0.5

1.0

1e6

12

11

10

0.0

0.5

1.0

Environment Steps

1e6

7.0

6.5

Large

0.0

0.5

1.0

Environment Steps

1e6

8.0

7.5

0.0

0.5

1.0

Environment Steps

1e6

12

10

0.0

0.5

1.0

1e6

20

15

Fixed

Fixed

PID

GPIDE

GRU

Transformer

PIDE

0.5

1.0

1e6

30

25

20

Small

0.5

1.0

1e6

40

30

Large

0.0

0.5

1.0

1e6

20

15

Small

0.0

0.5

1.0

1e6

25

20

0.5

1.0

1e6

40

30

0.0

0.5

1.0

Environment Steps

1e6

30

20

Large

0.0

0.5

1.0

Environment Steps

1e6

30

20

0.0

0.5

1.0

Environment Steps

1e6

35

30

25

0.0

0.5

1.0

1e6

20

15

Fixed

Fixed

PID

GPIDE-Attn

GPIDE-ESS

GPIDE-ES

0.0

0.5

1.0

1e6

25

20

Small

0.0

0.5

1.0

1e6

35

30

Large

0.0

0.5

1.0

1e6

20

15

Small

0.0

0.5

1.0

1e6

25

20

0.0

0.5

1.0

1e6

35

30

25

0.0

0.5

1.0

Environment Steps

1e6

30

20

Large

0.0

0.5

1.0

Environment Steps

1e6

25

20

0.0

0.5

1.0

Environment Steps

1e6

30

25

G.3 Navigation Results

0.0

0.2

0.4

0.6

0.8

1.0

1e6

20

18

16

14

PID

GPIDE

GRU

Transformer

PIDE

0.0

0.2

0.4

0.6

0.8

1.0

1e6

40

35

30

25

20

Real

0.0

0.2

0.4

0.6

0.8

1.0

Environment Steps

1e6

22

20

18

16

Real

0.0

0.2

0.4

0.6

0.8

1.0

Environment Steps

1e6

25.0

22.5

20.0

17.5

15.0

0.0

0.2

0.4

0.6

0.8

1.0

1e6

20

18

16

14

PID

GPIDE-Attn

GPIDE-ESS

GPIDE-ES

0.0

0.2

0.4

0.6

0.8

1.0

1e6

26

24

22

Real

0.0

0.2

0.4

0.6

0.8

1.0

Environment Steps

1e6

22

20

18

16

Real

0.0

0.2

0.4

0.6

0.8

1.0

Environment Steps

1e6

25.0

22.5

20.0

17.5

G.4 Tokamak Control Results

0.0

0.2

0.4

0.6

0.8

1.0

1e6

10

9

8

7

PID

GPIDE

GRU

Transformer

PIDE

0.0

0.2

0.4

0.6

0.8

1.0

1e6

60

40

20

Real

0.0

0.2

0.4

0.6

0.8

1.0

Environment Steps

1e6

22.5

20.0

17.5

15.0

12.5

Real

0.0

0.2

0.4

0.6

0.8

1.0

Environment Steps

1e6

14.0

13.5

13.0

0.00

0.25

0.50

0.75

1.00

1e6

9

8

7

PID

GPIDE-Attn

GPIDE-ESS

GPIDE-ES

0.00

0.25

0.50

0.75

1.00

1e6

50

40

30

20

Real

0.00

0.25

0.50

0.75

1.00

Environment Steps

1e6

17.5

15.0

12.5

Real

0.00

0.25

0.50

0.75

1.00

Environment Steps

1e6

13.8

13.6

0.2

0.4

0.6

0.8

1.0

1e6

27.5

25.0

22.5

20.0

17.5

PID

GPIDE

GRU

Transformer

PIDE

0.2

0.4

0.6

0.8

1.0

1e6

70

60

50

40

30

Real

0.2

0.4

0.6

0.8

1.0

Environment Steps

1e6

38

36

34

32

30

Real

0.0

0.2

0.4

0.6

0.8

1.0

Environment Steps

1e6

30

28

26

24

0.00

0.25

0.50

0.75

1.00

1e6

35

30

25

20

PID

GPIDE-Attn

GPIDE-ESS

GPIDE-ES

0.00

0.25

0.50

0.75

1.00

1e6

50

40

30

Real

0.00

0.25

0.50

0.75

1.00

Environment Steps

1e6

38

36

34

32

Real

0.00

0.25

0.50

0.75

1.00

Environment Steps

1e6

30

28

26

24

G.5 PyBullet Results

For these results, SAC encodes observations, actions and rewards. TD3 encodes observations and actions since it is the best performing on average.

PPO-GRU

SAC-LSTM

VRM

SAC-Transformer

SAC-GPIDE

SAC-GPIDE-ES

SAC-GPIDE-ESS

SAC-GPIDE-Attn

HalfCheetah-P

27

.

09

±

7

.

85

-

22

.

00

±

5

.

13

77

.

06

±

7

.

96

85

.

80

±

5

.

15

-

107

.

00

±

1

.

39

37

.

00

±

9

.

97

82

.

63

±

3

.

46

85

.

45

±

4

.

83

71

.

08

±

6

.

95

77

.

63

±

4

.

60

Hopper-P

49

.

00

±

5

.

22

-

2

.

29

±

3

.

33

64

.

36

±

9

.

94

84

.

63

±

8

.

33

3

.

53

±

1

.

63

59

.

54

±

19

.

64

93

.

27

±

13

.

56

111

.

48

±

4

.

20

113

.

95

±

4

.

67

104

.

87

±

8

.

76

Walker-P

1

.

67

±

4

.

39

-

10

.

48

±

1

.

73

40

.

92

±

15

.

56

29

.

08

±

9

.

67

-

3

.

89

±

1

.

25

24

.

89

±

14

.

80

96

.

61

±

1

.

60

76

.

58

±

5

.

47

94

.

58

±

11

.

00

71

.

36

±

6

.

57

Ant-P

39

.

48

±

3

.

74

-

13

.

06

±

6

.

52

60

.

97

±

3

.

54

-

36

.

36

±

3

.

35

-

36

.

39

±

0

.

17

-

10

.

57

±

2

.

34

66

.

66

±

2

.

94

64

.

73

±

3

.

82

57

.

78

±

3

.

78

63

.

19

±

5

.

32

HalfCheetah-V

19

.

68

±

11

.

71

-

50

.

13

±

9

.

50

18

.

54

±

33

.

09

59

.

03

±

2

.

88

-

80

.

49

±

2

.

97

-

41

.

31

±

26

.

15

20

.

39

±

29

.

60

51

.

03

±

13

.

93

53

.

14

±

5

.

86

-

54

.

70

±

19

.

89

Hopper-V

13

.

86

±

4

.

80

-

0

.

60

±

3

.

33

16

.

26

±

12

.

44

57

.

43

±

8

.

63

10

.

08

±

3

.

51

0

.

28

±

8

.

49

90

.

98

±

4

.

28

72

.

63

±

19

.

28

90

.

09

±

2

.

50

30

.

73

±

1

.

60

Walker-V

8

.

12

±

5

.

43

-

8

.

02

±

0

.

57

-

1

.

57

±

1

.

88

-

4

.

63

±

1

.

30

-

1

.

80

±

0

.

70

-

8

.

21

±

1

.

31

36

.

90

±

16

.

59

68

.

30

±

4

.

33

67

.

54

±

3

.

60

14

.

85

±

11

.

26

1

.

43

±

3

.

26

-

13

.

67

±

1

.

83

-

16

.

95

±

1

.

29

17

.

03

±

6

.

55

-

13

.

41

±

0

.

12

0

.

81

±

1

.

31

18

.

03

±

5

.

10

4

.

56

±

5

.

20

12

.

85

±

1

.

67

-

1

.

84

±

5

.

76

Average

20.04

-15.03

32.45

36.50

-28.67

7.80

63.18

66.84

70.13

38.26

PPO-GRU

SAC-LSTM

VRM

SAC-Transformer

SAC-GPIDE

SAC-GPIDE-ES

SAC-GPIDE-ESS

SAC-GPIDE-Attn

HalfCheetah-P

1445

.

81

±

166

.

79

403

.

35

±

108

.

97

2506

.

88

±

168

.

93

2692

.

53

±

109

.

43

-

1401

.

67

±

29

.

62

1656

.

13

±

211

.

75

2625

.

13

±

73

.

49

2684

.

98

±

102

.

57

2379

.

79

±

147

.

67

2519

.

06

±

97

.

72

Hopper-P

1436

.

43

±

102

.

09

433

.

19

±

65

.

09

1736

.

81

±

194

.

51

2133

.

42

±

162

.

93

546

.

93

±

31

.

81

1642

.

63

±

384

.

10

2302

.

31

±

265

.

21

2658

.

48

±

82

.

18

2706

.

81

±

91

.

39

2529

.

31

±

171

.

41

Walker-P

501

.

06

±

76

.

99

288

.

10

±

30

.

39

1189

.

28

±

272

.

77

981

.

63

±

169

.

46

403

.

60

±

21

.

85

908

.

17

±

259

.

52

2165

.

52

±

28

.

10

1814

.

40

±

95

.

91

2129

.

91

±

192

.

87

1722

.

81

±

115

.

22

Ant-P

2025

.

52

±

84

.

58

837

.

57

±

147

.

53

2511

.

54

±

80

.

13

310

.

72

±

75

.

68

310

.

24

±

3

.

83

893

.

84

±

52

.

83

2640

.

16

±

66

.

46

2596

.

63

±

86

.

26

2439

.

48

±

85

.

37

2561

.

67

±

120

.

21

HalfCheetah-V

1005

.

13

±

289

.

84

-

723

.

40

±

235

.

29

977

.

02

±

819

.

24

1979

.

56

±

71

.

40

-

1475

.

15

±

73

.

42

-

505

.

00

±

647

.

43

1022

.

93

±

732

.

93

1781

.

36

±

344

.

95

1833

.

60

±

145

.

14

-

836

.

47

±

492

.

45

Hopper-V

534

.

05

±

105

.

85

215

.

22

±

73

.

48

587

.

10

±

274

.

42

1495

.

11

±

190

.

42

450

.

77

±

77

.

35

234

.

49

±

187

.

36

2235

.

02

±

94

.

45

1830

.

26

±

425

.

16

2215

.

47

±

55

.

16

906

.

05

±

35

.

29

Walker-V

377

.

80

±

109

.

11

53

.

25

±

11

.

45

182

.

97

±

37

.

89

121

.

44

±

26

.

14

178

.

28

±

14

.

09

49

.

32

±

26

.

43

956

.

43

±

333

.

46

1587

.

56

±

87

.

15

1572

.

41

±

72

.

46

513

.

07

±

226

.

34

684

.

36

±

89

.

48

269

.

32

±

50

.

35

178

.

98

±

35

.

57

1113

.

19

±

179

.

93

276

.

33

±

3

.

18

667

.

20

±

35

.

98

1140

.

73

±

140

.

22

770

.

51

±

143

.

02

998

.

35

±

46

.

04

594

.

54

±

158

.

48

Average

1001.27

222.08

1233.82

1353.45

-88.84

693.35

1886.03

1965.52

2034.48

1313.76

0.0

0.5

1.0

1.5

1e6

1000

0

1000

2000

3000

Returns

HalfCheetahBLT-P-v0

0.0

0.5

1.0

1.5

1e6

0

500

1000

1500

2000

2500

HopperBLT-P-v0

GPIDE

Transformer

SAC-LSTM

PPO-GRU

VRM

Oracle

0.0

0.5

1.0

1.5

1e6

0

500

1000

1500

2000

2500

WalkerBLT-P-v0

0.0

0.5

1.0

1.5

1e6

0

500

1000

1500

2000

2500

3000

3500

AntBLT-P-v0

0.0

0.5

1.0

1.5

Environment Steps

1e6

1000

0

1000

2000

3000

Returns

HalfCheetahBLT-V-v0

0.0

0.5

1.0

1.5

Environment Steps

1e6

0

500

1000

1500

2000

2500

HopperBLT-V-v0

0.0

0.5

1.0

1.5

Environment Steps

1e6

0

500

1000

1500

2000

WalkerBLT-V-v0

0.0

0.5

1.0

1.5

Environment Steps

1e6

0

500

1000

1500

2000

2500

3000

3500

0.0

0.5

1.0

1.5

1e6

0

500

1000

1500

2000

2500

3000

Returns

HalfCheetahBLT-P-v0

0.0

0.5

1.0

1.5

1e6

500

1000

1500

2000

2500

3000

HopperBLT-P-v0

GPIDE-Attn

GPIDE-ESS

GPIDE-ES

Oracle

0.0

0.5

1.0

1.5

1e6

500

1000

1500

2000

2500

WalkerBLT-P-v0

0.0

0.5

1.0

1.5

1e6

500

1000

1500

2000

2500

3000

3500

AntBLT-P-v0

0.0

0.5

1.0

1.5

Environment Steps

1e6

1000

0

1000

2000

3000

Returns

HalfCheetahBLT-V-v0

0.0

0.5

1.0

1.5

Environment Steps

1e6

500

1000

1500

2000

2500

HopperBLT-V-v0

0.0

0.5

1.0

1.5

Environment Steps

1e6

0

500

1000

1500

2000

WalkerBLT-V-v0

0.0

0.5

1.0

1.5

Environment Steps

1e6

500

1000

1500

2000

2500

3000

3500

Interestingly, we found that GPIDE policies often outperform the oracle policy on Hopper-P. While the oracle performance here was taken from Ni et al. [50], we confirmed this also happens with our own implementation of an oracle policy. We hypothesize that this may be due to the fact the GPIDE policy gets to see actions and rewards and the oracle does not.

G.6 Attention Scheme Visualizations

We generate the attention visualizations (as seen in Figure 4) by doing a handful of rollouts with a GPIDE policy using only attention heads. During this rollout we collect all of the weighting schemes, i.e. softmax ( q 1: t k T 1: t √ D ) , generated throughout the rollouts and average them together. Below, we show additional attention visualizations. In all figures, each plot shows one of the different six heads. For each of these, the policies were evaluated on the same version of the environment they were trained on.

0

50

Time Step

0

20

40

60

80

Sequence Length

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

Sequence Length

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

Sequence Length

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

Sequence Length

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

Sequence Length

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

Sequence Length

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

Sequence Length

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

Sequence Length

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

Sequence Length

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

Sequence Length

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

Sequence Length

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

Sequence Length

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

0

50

Time Step

0

20

40

60

80

Sequence Length

Sequence Length

Sequence Length

0

20

40

60

0

20

40

60

0

20

40

60

0

0

0

0

0

0

25

50

Time Step

25

50

Time Step

25

50

Time Step

50

25

Time Step

0

50

25

0

25

50

0

Time Step

Time Step

25

50

Time Step

25

50

Time Step

25

50

Time Step

25

50

Time Step

0

25

50

Time Step

0

20

40

60

Sequence Length

0

25

50

Time Step

0

20

40

60

0

25

50

Time Step

0

20

40

60

0

25

50

Time Step

0

20

40

60

0

25

50

Time Step

0

20

40

60

0

25

50

Time Step

0

20

40

60

Sequence Length

Sequence Length

Sequence Length

0

20

40

60

0

20

40

60

0

20

40

60

0

20

40

60

0

20

40

60

0

20

40

60

0

0

20

40

60

20

40

60

0

20

40

60

0

20

40

60

0

20

40

60

0

20

40

60

25

50

Time Step

25

50

Time Step

25

50

Time Step

0

20

40

60

0

20

40

60

0

20

40

60

0

0

0

0

0

0

25

50

Time Step

0

25

50

0

25

50

0

25

50

Time Step

Time Step

Time Step

0

0

20

40

60

20

40

60

0

20

40

60

25

50

Time Step

0

50

25

0

25

50

0

25

50

Time Step

Time Step

Time Step

0

0

20

40

60

20

40

60

0

20

40

60

25

50

Time Step

0

25

50

0

25

50

0

Time Step

Time Step

0

0

20

40

60

20

40

60

0

20

40

60

25

50

Time Step

0

25

50

0

25

50

0

Time Step

Time Step

0

0

20

40

60

20

40

60

0

20

40

60

25

50

Time Step

25

50

0

25

50

0

Time Step

Time Step

0

20

40

60

0

20

40

60

0

20

40

60

0

25

50

Time Step

25

50

Time Step

25

50

Time Step

0

20

40

60

0

20

40

60

0

20

40

60

0

0

0

0

0

0

25

50

Time Step

25

50

Time Step

25

50

Time Step

0

10

20

Time Step

0

10

20

Sequence Length

0

10

20

Time Step

0

10

20

0

10

20

Time Step

0

10

20

0

10

20

Time Step

0

10

20

0

10

20

Time Step

0

10

20

0

10

20

Time Step

0

10

20

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes two history encoders for RL in partially observable control tasks. The history encoders are designed with PID-inspired inductive bias. Specifically, the inductive bias comes from lifting the observation into features consisting of summation and difference of observation over the historical horizon, which are analogous to the integral and derivative of tracking errors in PID. It was shown that the proposed history encoders, especially the generalized PID encoder, could achieve better performance and robustness than GRU and transformers in tracking and locomotion tasks when integrated with RL algorithms.\n\nSTRENGTHS:\nI particularly appreciate the new perspective presented by the authors in combining conventional control techniques with reinforcement learning. The idea of incorporating PID architecture into history encoders is interesting and novel to the best of my knowledge. The advantages of the generalized PID encoders over GRU and transformers, especially in the locomotion tasks, validate that the PID-inspired inductive bias is indeed effective in some control tasks. This paper could inspire the RL community to discover more powerful and generalizable architectures with insights from control theory and conventional control techniques.\n\nWEAKNESSES:\n1. In lines 102-104, the authors stated:\n>\n> In the case of MIMO tracking problems, where there are M signals with M corresponding actuators, one can control the system with M separate PID controllers. However, this assumes there is a clear breakdown of which actuator influences which signal.\n>\nDecomposing a MIMO system into multiple SISO systems is indeed a practical way to synthesize MIMO PID controllers. However, there exist methods to directly synthesize PID controllers for MIMO systems without decomposition (e.g., [1]). While I am okay with the authors only comparing the proposed methods with a decomposed PID controller, the authors should appropriately assert the capacity of PID controllers based on the state-of-the-art literature. \n\n2. The related work section should include a review of the literature on the combination of conventional control techniques with RL. For instance, there have been extensive efforts in tuning PID or learning an adaptive PID weight schedule with RL, which could be considered a special case of the control policy with the proposed PID encoder. I understand the paper is mainly targeting audiences from the RL community, but I think it is necessary to place the proposed method in the control literature as well. \n\n3. While I am generally impressed by the experimental results, I am concerned about the failure of SAC-GPIDE in the HalfCheetah-V environment. It makes me wonder whether the PID-based inductive bias could limit the expressiveness of the policy network in some control tasks. For instance, I would doubt if the proposed method could still work when image-based observation is used. The authors do not provide any explanation or hypothesis for the failure in HalfCheetah-V. I would suggest providing more information and insights regarding it during the rebuttal and in the updated version of the paper.  \n\n[1] Boyd, Stephen, Martin Hast, and Karl Johan Åström. \"MIMO PID tuning via iterated LMI restriction.\" International Journal of Robust and Nonlinear Control 26, no. 8 (2016): 1718-1731.\n\nQUESTIONS:\n1. Please see the three issues listed in the Weaknesses section. I would have a more positive impression of the paper if the authors could properly address these three issues during the rebuttal. \n\n2. In addition, I want to ask the authors to clarify the role of attention heads in GPIDE. In the ablation study, the authors stated:\n>\n> It appears that the attention scheme for HalfCheetah is simply a poor reproduction of exponential smoothing, making it redundant and suboptimal. In fact, we found this phenomenon to be true across all attention heads and PyBullet tasks. We believe that the periodicity that appears here is due to the oscillatory nature of the problem and lack of positional encoding (although we found including positional encoding degrades performance).\n>\nIf that is the case, does it mean that the attention heads could be simply replaced by exponential smoothing heads? It seems to be supported by the ablation study results (i.e., ES+SUM has similar or better performance than GPIDE with multiple types of heads). In lines 227-230, the authors said:\n>\n>This choice was not optimized but rather was picked so that all types of heads were included and so that GPIDE has roughly the same amount of parameters as our GRU baseline.\n>\nI don't think it is a good justification for the experimental design. The authors should attempt to find an optimized configuration so that the audience can have a better sense of the necessity of each proposed head, especially the attention head.\n\nLIMITATIONS:\nThe authors acknowledged the limitation of the proposed methods, which is the proposed PID-inspired inductive bias might not apply to all control tasks. While I personally doubt if the proposed method can ever be extended to systems with image-based observations, I do not think the authors are obligated to justify the feasibility of extension to visual control tasks in this paper. However, as suggested above, I do think the authors should provide more information and insights on the failure of GPIDE in Cheetah-V.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

comments: SUMMARY:
The paper proposes a new, simplified architecture for reinforcement learning in the partially observable setting inspired by PID control. Through experiments on a number of tracking problems, and some locomotion environments, strong performance is obtained.

STRENGTHS:
- The exposition of the argument is very clear, even for a reader somewhat unfamiliar with some aspects.
- Simplification of complex (and computationally expensive) recurrent architectures is a worthwhile endeavor.
- Ablations are informative, and come with demonstrations of issues (Figure 4).

WEAKNESSES:
- The largest weakness is in the evaluation. Most of the evaluation is devoted to low-dimensional tracking problems. The choice of state-based locomotion with only positions or velocities observable also seems a little artificial when there are many tasks that have existing sources of partial observability (long-horizon, first-person viewpoints).
- One possibly beneficial use of an alternative state summary like GPIDE would be computational savings, but there isn't any measurement of that in the paper.

QUESTIONS:
See the final weakness, it would be nice to see computational cost measured.

LIMITATIONS:
Limitations are discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers history encoding for deep RL POMDP control problems through a PID-inspired lens. Specifically, authors introduce PIDE, a method of directly using PID control which extends to multiple-input multiple-output problems, and GPIDE, a PID-inspired encoder architecture. GPIDE consists of a series of heads that takes as input model observations and past history, and internally computes differences between observations (similar to the PID D-term) and aggregation/summation over all past timesteps (similar to the PID I-term).\n\nThe method is then evaluated on tasks including mass-spring-damper, navigation, tolemak control, and pyBullet locomotion. Across these experiments, the method is compared against baselines of direct PID control, GRU, transformer, and several RL algorithms. GPIDE is found to have strong performance across both simple and complex tasks and be robust to domain transfer scenarios. Further ablation studies provide insight into design choices for GPIDE heads, visualize attention schemes, and evaluate a GRU+VIB baseline.\n\nSTRENGTHS:\n#### Originality\n\nThe ideas of the paper are generally original, building from prior deep RL work and focusing on the design of the feature encoder, especially designing a novel history encoding architecture which draws from PID controllers. The pieces of the architecture draw from general temporal aggregation principles and transformers.\n\n#### Quality\n\nThe paper is generally high-quality. The experiments compare against relevant baselines and prior work, are run with multiple seeds, and the training / setup / tasks are clearly defined. The code is provided and looks cleanly written from initial observation, though I haven’t run it to verify. The ablation studies answer my initial questions about the architecture design, especially in how important the attention heads are.\n\n#### Clarity\n\nThe manuscript is well written and clearly conveys the derivation of the method from PID controllers and explains the architecture and design choices in a very understandable way. Relevant figures are provided for context of the architecture and tasks. The experiments are well motivated and help understand the performance of PIDE / GPIDE across tasks, RL methods, and transfer scenarios.\n\n\n#### Significance\n\nGPIDE seems broadly applicable across deep RL control tasks, especially those related to tracking and navigation / locomotion as demonstrated by experiments. The method is an approach to handling the POMDP problem on control problems as a drop-in architecture for history encoding.\n\nWEAKNESSES:\n#### Experiments\nCompared to RNNs such as GRU which consider only the current timestep and state to predict the next action, GPIDE seems to consider all past timesteps from 1 to t within each step, as part of the weighted summation resembling the PID I step, which may be more expensive than other methods if the time / observations are large. It may be worthwhile to consider potential inference time compute differences because of this, and also to consider ablations or baselines which also have such history context.\n\n#### Significance\nOn line 116-117, the authors mention that LSTM, GRU, and transformers were shown to be powerful tools for NLP because of complex relationships within the task. Because control tasks may not have such complexities, such methods may overfit rather than help, especially in cases of domain transfer. However, these methods have also been applied to success in domains including robotics and visual encoding (e.g. https://arxiv.org/abs/2212.06817, https://arxiv.org/abs/2010.11929, https://arxiv.org/abs/2106.01345) - perhaps this indicates that the best targets for using GPIDE would be cases where highly-overparameterized / “large” models may overfit, which would be towards the simpler side.\n\nQUESTIONS:\n#### Limitations\nThe authors mention that control tasks which require memory may not be suitable - it would be further interesting to consider from the perspective of task similarity to that of PID control. Do the authors think that there would be correlation between similarity to PID / tracking tasks and successful application of GPIDE?\n\nFor instance, a common image-based task would be visual grasping which involves reasoning about object geometry, where it may be less clear that results would directly transfer.\n\nFurthermore, some tasks may be less partially observable in nature than others - given this method focuses on a history encoder, would the relevance of history generally for certain tasks be relevant to consider?\n\nLIMITATIONS:\nThe limitations are generally adequate (there are some questions in the section above).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new way to encode features in partially observable environments using PID. Experiments show superior results over several domains compared with recurrent and transformer encoders.\n\nSTRENGTHS:\nThe paper is easy to read overall.\n\nExperiments show promising performance on multiple domains.\n\nWEAKNESSES:\nSome assumptions and technical details need to be clarified.\n\n1) Using the difference between observations as the derivative of the error (D) is trivial,  assuming that the reference value for the observation at different timesteps is the same, which is not true.\n\n2) The author mentioned in Line 102 that using M separate PID controllers to control M signals has some clear shortcomings. This paper models the MIMO problem in a centralized manner, inputting all inputs and outputting a 3M dimensional vector. However, this would not be feasible if there is a large scale of signals that need to control.\n\n3) It said the $f_{\\theta}^h$ is a learnable linear projection, how to learn it is not discussed. Also with $g_{\\theta}$.\n\n4) How to determine the suitable length of the history, As for the length $1:t$ in $v_{1:t}^{h}$?\n\nMore ablations should be provided.\n\n1) The ablations investigate the influence of different types of heads, while why choosing these three settings (ES, ES+Sum, Attention) is not explained. What about ES+Attention? Why five ES + 1 Sum? Are there any insights in choosing the number and combination of different heads?\n\n2) The influence of different lengths of history should also be investigated.\n\nMore limitations should be discussed. For example, the encoder processing time is highly dependent on the length of the history.\n\nSome related works are not discussed in this paper. For example, [1-3].\n\n[1] An adaptive deep reinforcement learning approach for MIMO PID control of mobile robots\n\n[2] Self-Tuning Two Degree-of-Freedom Proportional–Integral Control System Based on Reinforcement Learning for a Multiple-Input Multiple-Output Industrial Process That Suffers from Spatial Input Coupling\n\n[3] Robust Adaptive PID Control based on Reinforcement Learning for MIMO Nonlinear Six-joint Manipulator\n\nQUESTIONS:\nQuestions,\n\nPlease see the pros and cons part about the assumptions, technical details, ablations, and limitations.\n\nLIMITATIONS:\nPlease see the pros and cons part about the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of learning from histories in partially observable MDPs where a key question is how to design an architecture that is general enough so as to work on a large set of problems, yet specific enough to be sample efficient.\n\nInspired by the success of PID in the more classical control literature, the idea here is to design an architecture that is solely comprised of summation, attention, and linear operations. \n\nExperiments show that on tracking problems, where PID is originally proposed for, as well as more complicated control problems, this simple architecture is good enough to achieve competitive results.\n\nSTRENGTHS:\nThe paper shows that using a complicated architecture is an overkill for many of the domains that the RL community considers as benchmarks. On this note I'd like to mention that, while the contexts are somewhat different, a similar conclusion can be made from \"Towards Generalization and Simplicity in Continuous Control\" Rajeswaran et al, where they show that simpler architectures such as linear models are equipped to deal with Mujoco Tasks. The difference is that in their paper they were considering reward-maximization in presence of the full state, whereas here the goal is learn a memory-like function in conjunction with performing well in the given task.\n\nIn any case, I generally like results that show simple stuff work, because simple things are easy to understand, implement, and execute.\n\nWEAKNESSES:\nOverall I like this paper but I also have some concerns:\n\nFundamentally, I feel like the results in this paper are pertinent in terms of a bigger question, namely how rich we want our hypothesis class to be for learning a certain target. The solution adopted by the ML community nowadays seems to be: we really do not know how complicated a task is, so by choosing a very complicated architecture and the fact that we are at the mercy of universal function approximation capability, we can always in principle find the right fit. Moreover, the double descent phenomenon  (\"Deep double descent: where bigger models and more data hurt\", Nakkiran et al) tells us that empirically we are very likely to find a good fit with larger and more complicated models and that, despite more classical belief, such an overparameterized fit happens to also be quite robust. (\"A Universal Law of Robustness via Isoperimetry\" Bubeck and Sellke)\n\nNow, moving to the RL setting, these insights from supervised learning do not really carry over because we usually learn by bootstrapping off of Bellman targets, and by increasing the complexity of our function approximator, we also have to learn from increasingly more complicated targets (unlike supervised learning where targets are fixed).\n\nSo I think, at least with our current RL algorithms, unlike SL we always need to choose our function class carefully and so adding inductive bias, such as ones provided in this paper, can usually be helpful. My worry is that adding such biases are helpful in a limited sense: It is not clear how much inductive bias should I add for a given task when I don't know too much about that task. Sure, I can always choose environments and problem settings where I know that my inductive bias is conducive to solving the problem, but do we really want to bring back the burden to ML practitioners to carefully think about the kind of inductive biases that are useful for their task?\n\n\nGrounding my concerns more so in the experimental results provided here, for example we see that the inductive bias provided is already giving small (or negative) gains as we move to some of the more complicated domains such as HalfCheetah and Ant (based on Figures 18 and 19 in the Appendix), so I ask authors, are you not worried that as we move to much larger domains then it becomes increasingly more difficult to show performance gains for GPIDE compared to other approaches? In other words, does the inductive bias scale reasonably with problem complexity or there is only so much this could be useful?\n\nTo further play devil's advocate, to say that GPIDE is capable of beating the transformer architecture is sort of a weird claim, because as authors say GPIDE could be thought of as a special case of transformers, and further as you show the attention layer is often not very useful. This tells me that with proper hyper-parameter tuning and regularization, the transformer architecture should in principle be able to always do better than GPIDE, and we can still hope to find the transformer useful as we move to larger problems.\n\nQUESTIONS:\nI would like to ideally see an acknowledgment that as we move to much more difficult domains, GPIDE may be incapable of doing well simply because the actual function to be represented in not in the function class. But such an example is currently absent. This may give a false sense that the paper is claiming that GPIDE is a general-purpose memory learner but that is not true, and I am not saying that authors say it is true, but having examples of failure cases would really clarify this point.\n\nIf I may say, I would also like to see experiments on different types of problems. For example, can we expect GPIDE to perform reasonably on image-based domains? Like the paper mentions, we can hope to apply the GPIDE on the embedding space learned by some more capable function approximator, but it seems to me that to learn good embeddings that are useful for downstream memory learning, one needs to leverage some kind of GRU/Transformer/LSTM architecture, and so it is not immediately clear to me how to leverage GPIDE in this setting.\n\nMy other question is also that as we know RNNs struggle to learn important events from the distant past because the vanishing/exploding gradients. LSTMs and Transformers are less prone to this issue but I wonder if we could better compare this capability of GPIDE in contrast to the existing architectures. In the Half-Cheetah experiment, for example, do we have a good sense of how far back one needs to go to be able to accurately learn the position and/or the velocity signal?\n\nAlso, from the intro the paper reads \"Another hurdle stems from the fact that policies are often trained in an imperfect simulator, which is likely different from the true environment. Combining these two challenges necessitates striking a balance between extracting useful information from the history and avoiding overfitting to modelling error.\"\n\nMy thoughts reading this sentence was that we are going to either have actual sim2real experiments or some kind of a transfer learning benchmark, where we learn the memory function in some settings, and test the learned memory function in another test setting. This would have been more convincing in terms of claiming that GPIDE is not prone to overfitting. Did I get it right that this experiment is absent?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}