SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ian Char  
Machine Learning Department  
Carnegie Mellon University  
Pittsburgh, PA 15213  
ichar@cs.cmu.edu
Jeff Schneider  
Machine Learning Department, Robotics Institute  
Carnegie Mellon University  
Pittsburgh, PA 15213  
schneide@cs.cmu.edu
Deep reinforcement learning (RL) has shown immense potential for learning to control systems through data alone. However, one challenge deep RL faces is that the full state of the system is often not observable. When this is the case, the policy needs to leverage the history of observations to infer the current state. At the same time, differences between the training and testing environments makes it critical for the policy not to overfit to the sequence of observations it sees at training time. As such, there is an important balancing act between having the history encoder be flexible enough to extract relevant information, yet be robust to changes in the environment. To strike this balance, we look to the PID controller for inspiration. We assert the PID controller's success shows that only summing and differencing are needed to accumulate information over time for many control tasks. Following this principle, we propose two architectures for encoding history: one that directly uses PID features and another that extends these core ideas and can be used in arbitrary control tasks. When compared with prior approaches, our encoders produce policies that are often more robust and achieve better performance on a variety of tracking tasks. Going beyond tracking tasks, our policies achieve 1.7x better performance on average over previous state-of-the-art methods on a suite of locomotion control tasks.<sup>1</sup>
Deep reinforcement learning (RL) holds great potential for solving complex tasks through data alone, and there have already been exciting applications of RL in video game playing [70], language model tuning [53], and robotic control [2]. Despite these successes, there still remain significant challenges in controlling real-world systems that stand in the way of realizing RL's full potential [20]. One major hurdle is the issue of partial observability, resulting in a Partially Observable Markov Decision Process (POMDP). In this case, the true state of the system is unknown and the policy must leverage its history of observations. Another hurdle stems from the fact that policies are often trained in an imperfect simulator, which is likely different from the true environment. Combining these two challenges necessitates striking a balance between extracting useful information from the history and avoiding overfitting to modelling error. Therefore, introducing the right inductive biases to the training procedure is crucial.
The use of recurrent network architectures in deep RL for POMDPs was one of the initial proposed solutions [29] and remains a prominent approach for control tasks [47, 75, 50]. Theses architectures are certainly flexible; however, it is unclear whether they are the best choice for control tasks,
$^{1}$ Code available at https://github.com/IanChar/GPIDE
especially since they were originally designed with other applications in mind such as natural language processing.
In contrast with deep RL methods, the Proportional-Integral-Derivative (PID) controller remains a cornerstone of modern control systems despite its simplicity and the fact it is over 100 years old [5, 48]. PID controllers are single-input single-output (SISO) feedback controllers designed for tracking problems, where the goal is to maintain a signal at a given reference value. The controller adjusts a single actuator based on the weighted sum of three terms: the current error between the signal and its reference, the integral of this error over time, and the temporal derivative of this error. PID controllers are far simpler than recurrent architectures and yet are still able to perform well in SISO tracking problems despite having no model for the system's dynamics. We assert that PID's success teaches us that in many cases only two operations are needed for successful control: summing and differencing.
To investigate this assertion, we conduct experiments on a variety of SISO and multi-input multi-output (MIMO) tracking problems using the same featurizations as a PID controller to encode history. We find that this encoding often achieves superior performance and is significantly more resilient to changes in the dynamics during test time. The biggest shortcoming with this method, however, is that it can only be used for tracking problems. As such, we propose an architecture that is built on the same principles as the PID controller, but is general enough to be applied to arbitrary control problems. Not only does this architecture exhibit similar robustness benefits, but policies trained with it achieve an average of  $1.7x$  better performance than previous state-of-the-art methods on a suite of locomotion control tasks.
The MDP and POMDP We define the discrete time, infinite horizon Markov Decision Process (MDP) to be the tuple  $(\mathcal{S},\mathcal{A},r,T,T_0,\gamma)$ , where  $\mathcal{S}$  is the state space,  $\mathcal{A}$  is the action space,  $r: S\times \mathcal{A}\times S\to \mathbb{R}$  is the reward function,  $T:S\times \mathcal{A}\rightarrow \Delta (\mathcal{S})$  is the transition function,  $T_{0}\subset \Delta (\mathcal{S})$  is the initial state distribution, and  $\gamma$  is the discount factor. We use  $\Delta (\mathcal{S})$  to denote the space of distributions over  $\mathcal{S}$ . Importantly, the Markov property holds for the transition function, i.e. the distribution over a next state  $s^{\prime}$  depends only on the current state,  $s$ , and current action,  $a$ . Knowing previous states and actions does not provide any more information. The objective is to learn a policy  $\pi :S\to \Delta (\mathcal{A})$  that maximizes the objective  $J(\pi) = \mathbb{E}\left[\sum_{t = 0}^{\infty}\gamma^{t}r(s_{t},a_{t},s_{t + 1})\right]$ , where  $s_0\sim T_0$ ,  $a_{t}\sim \pi (s_{t})$ , and  $s_{t + 1}\sim T(s_t,a_t)$ . When learning a policy, it is often key to learn a corresponding value function,  $Q^{\pi}:S\times \mathcal{A}\to \mathbb{R}$ , which outputs the expected discounted returns after playing action  $a$  at state  $s$  and then following  $\pi$  afterwards.
In a Partially Observable Markov Decision Process (POMDP), the observations that the policy receives are not the true states of the process. In control this may happen for a variety of reasons such as noisy observations made by sensors, but in this work we specifically focus on the case where aspects of the state space remain unmeasured. In any case, the POMDP is defined as the tuple  $(S, \mathcal{A}, r, T, T_0, \Omega, \mathcal{O}, \gamma)$ , where  $\Omega$  is the space of possible observations,  $\mathcal{O}: S \times \mathcal{A} \to \Delta(\Omega)$  is the conditional distribution of seeing an observation, and the rest of the elements of the tuple remain the same as before. The objective remains the same as the MDP, but now the policy and value functions are not allowed access to the state.
Crucially, the Markov property does not hold for observations in the POMDP. That is, where  $o_{1:t+1} \coloneqq o_1, o_2, \ldots, o_{t+1}$  are observations seen at times 1 through  $t+1$ ,  $o_{1:t-1} \not\sim o_{t+1}|o_t, a_t$ . A naive solution to this problem is to instead have the policy take in the history of the episode so far. Of course, it is usually infeasible to learn a policy that takes in the entire history for long episodes since the space of possible histories grows exponentially with the length of the episode. Instead, one can encode the information into a more compact representation. In particular, one can use an encoder  $\phi$  which outputs an encoding  $z_t = \phi(o_{1:t}, a_{1:t-1}, r_{1:t-1})$  (note that encoders need not always take in the actions and rewards). Then, the policy and Q-value functions are augmented to take in  $(o_t, z_t)$  and  $(o_t, a_t, z_t)$ , respectively.
Tracking Problems and PID Controllers. We first focus on the tracking problem, in which there are a set of signals that we wish to maintain at given reference values. For example, in espresso machines the temperature of the boiler (i.e. the signal) must be maintained at a constant reference temperature, and a controller is used to vary the boiler's on-off time so the temperature
is maintained at that value [43]. Casting tracking problems as discrete time POMDPs, we let  $o_{t} = \left(x_{t}^{(1)},\dots ,x_{t}^{(M)},\sigma_{t}^{(1)},\dots ,\sigma_{t}^{(M)}\right)$  be the observation at time  $t$ , where  $x_{t}^{(i)}$  and  $\sigma_t^{(i)}$  are the  $i^{\mathrm{th}}$  signal and corresponding reference value, respectively. The reward at time  $t$  is simply the negative error summed across dimensions, i.e.  $-\sum_{m = 1}^{M}\left|x_{t}^{(m)} - \sigma_{t}^{(m)}\right|$ .
When dealing with a single-input single-output (SISO) system (with one signal and one actuator that influences the signal), one often uses a Proportional-Integral-Derivative (PID) controller: a feedback controller that is often paired with feedforward control. This controller requires no knowledge of the dynamics, and simply sets the action via a linear combination of three terms: the error (P), the integral of the error (I), and the derivative of the error (D). When comparing other architectures to the PID controller, we will use orange colored text and blue colored text to highlight similarities between the I and D terms, respectively. Concretely, the policy corresponding to a discrete-time PID controller is defined as
$$
\pi^ {\mathrm {P I D}} \left(o _ {t}\right) = K _ {P} \left(x _ {t} ^ {(1)} - \sigma_ {t} ^ {(1)}\right) + K _ {I} \sum_ {i = 1} ^ {t} \left(x _ {i} ^ {(1)} - \sigma_ {i} ^ {(1)}\right) d t + K _ {D} \frac {\left(x _ {t} ^ {(1)} - \sigma_ {t} ^ {(1)}\right) - \left(x _ {t - 1} ^ {(1)} - \sigma_ {t - 1} ^ {(1)}\right)}{d t} \tag {1}
$$
where  $K_{P}$ ,  $K_{I}$ , and  $K_{D}$  are scalar values known as gains that must be tuned. PID controllers are designed for SISO control problems, but many real-world systems are multi-input multi-output (MIMO). In the case of MIMO tracking problems, where there are  $M$  signals with  $M$  corresponding actuators, one can control the system with  $M$  separate PID controllers. However, this assumes there is a clear breakdown of which actuator influences which signal. Additionally, there are often interactions between the different signals, which the PID controllers do not account for. Beyond tracking problems, it is less clear how to use PID controllers without substantial engineering efforts.
To motivate the following, consider the task of controlling a tokamak: a toroidal device that magnetically confines plasma and is used for nuclear fusion. Nuclear fusion holds the promise of providing an energy source with few drawbacks and an abundant fuel source. As such, there has recently been a surge of interest in applying machine learning [1, 13], and especially RL [14, 45, 17, 71, 65, 64, 44], for tokamak control. However, applying deep RL has the same problems as mentioned earlier; the state is partially observable since there are aspects of the plasma's state that cannot be measured in real time, and the policy must be trained before-hand on an imperfect simulator since operation of the actual device is extremely expensive.
How should one choose a historical encoder with these challenges in mind? Previous works [50, 46] suggest using Long Short Term Memory (LSTM) [31], Gated Recurrent Units [15], or transformers [69]. These architectures have been shown to be powerful tools in natural language processing, where there exist complicated relationships between words and how they are positioned with respect to each other. However, do the same complex temporal relationships exist in something like tokamak control? The fact that PID controllers have been successfully applied for feedback control on tokamaks suggests this may not be the case [72, 27]. In reality, the extra flexibility of these architectures may become a hindrance when deployed on the physical device if they overfit to quirks in the simulator.
In this section, we present two historical encoders that we believe have good inductive biases for control. They are inspired by the PID controller in that they only sum and difference in order to combine information throughout time. Following this, in Section 5, we empirically show the benefits of these encoders on a number of control tasks including tokamak control.
The PID Encoder. Under the framework of a policy that uses a history encoder, the standard PID controller (1) is simply a linear policy with an encoder that outputs the tracking error, the integral of the tracking error, and the derivative of the tracking error. This notion can be extended to MIMO problems and arbitrary policy classes, resulting in the PID-Encoder (PIDE). Given input  $o_{1:t}$ , this encoder outputs a  $3M$  dimensional vector consisting of  $(x_t^{(m)} - \sigma_t^{(m)})$ ,  $\sum_{i=1}^{t} (x_i^{(m)} - \sigma_i^{(m)}) dt$ , and  $\frac{(x_t^{(m)} - \sigma_t^{(m)}) - (x_{t-1}^{(m)} - \sigma_{t-1}^{(m)})}{dt} \forall m = 1, \ldots, M$ . For SISO problems, policies with this encoder have access to the same information as a PID controller. However, for MIMO problems the policy has
access to all the information that each PID controller, acting in isolation, would have. Ideally a sophisticated policy would coordinate each actuator setting well.
Figure 1: Architecture for GPIDE. The diagram shows how one encoding,  $z_{t}$ , is formed. Each of the gray, rounded boxes corresponds to one of the heads that makes up GPIDE. Each green box shows a function to be learned from data, and the orange box shows the weighted summation of all previous vectors,  $v_{1:t}^{h}$ . We write the difference in observations in blue text to highlight the part of GPIDE that relates to a PID controller's D term. Note that  $q_{1:t}^{h}$  and  $k_{1:t}^{h}$  only play a role in this process if head  $h$  uses attention; as such, we write these terms in gray text.
The Generalized PID Encoder. A shortcoming of PIDE is that it is only applicable to tracking problems since it operates over tracking error explicitly. A more general encoder should instead accumulate information over arbitrary features of each observation. With this in mind, we introduce the Generalized-PID-Encoder (GPIDE).
GPIDE consists of a number of "heads", each accumulating information about the history in a different manner. When there are  $H$  heads, GPIDE forms history encoding,  $z_{t}$ , through the following:
$$
\begin{array}{l} v _ {i} ^ {h} = f _ {\theta} ^ {h} \left(\text {c o n c a t e n a t e} \left(o _ {i - 1}, a _ {i - 1}, r _ {i - 1}, o _ {i} - o _ {i - 1}\right)\right) \quad \forall i \in \{1, \dots , t \}, h \in \{1 \dots , H \} \\ w _ {t} ^ {h} = \ell^ {h} \left(v _ {1: t} ^ {h}\right) \quad \forall h \in \{1 \dots , H \} \\ z _ {t} = g _ {\theta} \left(\operatorname {c o n c a t e n a t e} \left(w _ {t} ^ {1}, w _ {t} ^ {2}, \dots , w _ {t} ^ {h}\right)\right) \\ \end{array}
$$
Here, GPIDE is parameterized by  $\theta$ . For head  $h$ ,  $f_{\theta}^{h}$  is a linear projection of the previous observation, action, reward, and difference between the current and previous observation to  $\mathbb{R}^D$ , and  $\ell^{h}$  is a weighted summation of these projections.  $g_{\theta}$  is a decoder which combines all of the information from the heads. A diagram of this process is shown in Figure 1. Note that  $\theta$  is trained along with the policy and Q networks end-to-end with a gradient based optimizer.
Notice that the key aspects of the PID controller are present here. The difference in observations is explicitly taken before the linear projection  $f_{\theta}^{h}$ . We found that this simple method works best for representing differences when the observations are scalar descriptions of the state (e.g. joint positions). Although we do not consider image observations in this work, we imagine a similar technique could be done by taking the differences in image encodings. Like the integral term of the PID,  $\ell^{h}$  also accumulates information over time. In the following, we consider several possibilities for  $\ell^{h}$ , and we will refer to these different choices as "head types" throughout this work. We omit the superscript  $h$  below for notational convenience.
Summation. Most in line with PID, the projections can be summed, i.e.  $\ell(v_{1:t}) = \sum_{i=1}^{t} v_i$ .
Exponential Smoothing. In order to weight recent observations more heavily, exponential smoothing can be used. That is,  $\ell(v_{1:t}) = (1 - \alpha)^{t-1}v_1 + \sum_{i=2}^{t}\alpha(1 - \alpha)^{t-i}v_i$ , where  $0 \leq \alpha \leq 1$  is the smoothing parameter. Unlike summation, this head type cannot accumulate information in the same way because it is a convex combination.
Attention. Instead of hard-coding a weighted summation of the projections, this weighting can be learned through attention [69]. Attention is one of the key components of transformers because of its ability to learn relationships between tokens. To implement this, two additional linear functions should be learned that project concatenate  $(o_{i - 1},a_{i - 1},r_{i - 1},o_i - o_{i - 1})$  to  $\mathbb{R}^D$ . These new projections are referred to as they key and query vectors, denoted as  $k_{i}$  and  $q_{i}$  respectively. The softmax between their inner products is then used to form the weighting scheme for  $v_{1:t}$ . We can rewrite the first two
steps of GPIDE for a head that uses attention as
$$
\begin{array}{l} v _ {i}, k _ {i}, q _ {i} = f _ {\theta} (\text {c o n c a t e n a t e} (o _ {i - 1}, a _ {i - 1}, r _ {i - 1}, o _ {i} - o _ {i - 1})) \quad \forall i \in \{1, \dots , t \} \\ w _ {1: t} = \ell \left(q _ {1: t}, k _ {1: t}, v _ {1: t}\right) = \operatorname {s o f t m a x} \left(\frac {q _ {1 : t} k _ {1 : t} ^ {T}}{\sqrt {D}}\right) v _ {1: t} \\ \end{array}
$$
Here,  $q_{1:t}$ ,  $k_{1:t}$ , and  $v_{1:t}$  are treated as  $t \times D$  dimensional matrices. Since it results in a convex combination, attention has the capacity to reproduce exponential smoothing but not summation.
To anchor the GPIDE architecture back to the PID controller, we note that the P, I, and D terms can be formed exactly. At a high level, this is achieved when  $f_{\theta}^{h}$  simply subtracts the target from the state measurement and when using summation and exponential smoothing heads with  $\alpha = 1$ . We write down the specific instance of GPIDE that results in these terms in Appendix A.1. While it is trivial for GPIDE to reconstruct the P, I, and D terms, it is less clear how an LSTM or GRU would achieve this, especially because of the I term. At the same time, GPIDE is much more flexible than the PID-representation since altering  $f_{\theta}^{h}$  results in different representations at each time step and altering the type of head results in different temporal relationships.
A control task may be partially observable for a myriad of reasons including unmeasured state variables [28, 75, 29], sensor noise[47], and unmeasured system parameters [76, 54]. When there are unmeasured system parameters, this is usually framed as a meta-reinforcement learning (MetaRL) [73] problem. This is a specific subclass of POMDPs where there is a collection of MDPs, and each episode, an MDP is sampled from this collection. Although these works do consider system parameters varying between episodes, the primary focus of the experiments usually tends to be on the multi-task setting (i.e. different reward functions instead of transition functions) [77, 18, 60]. We consider not only differing system parameters but also the presence of unmeasured state variables; therefore, the class of POMDPs considered in this paper is broader than the one studied in MetaRL.
Using recurrent networks has long been an approach for tackling POMDPs [29], and is still a common way to do so in a wide variety of settings [19, 73, 70, 50, 47, 75, 66, 11, 2]. Moreover implementations are publicly available both for on-policy [41, 30] and off-policy [50, 75, 11] algorithms, making it an easy pick for those wanting a quick solution. Some works [32, 77, 28, 18, 3] use recurrent networks to estimate the belief state [37], which is a distribution over the agent's true state. However, Ni et al. [50] recently showed that well-implemented, recurrent versions of SAC [26] and TD3 [23] perform competitively with many of these specialized algorithms. In either case, we believe works that estimate the belief state are not in conflict with our own since their architectures can be modified to use GPIDE instead of a recurrent unit.
Beyond recurrent networks, there has been a surge of interest in applying transformers to reinforcement learning [40]. However, we were unable to find many instances of transformers being used as history encoders in the online setting, perhaps because of their difficulty to train. Parisotto et al. [55] introduced a new architecture to remedy these difficulties; however, Melo [46] applied transformers to MetaRL and asserted that careful weight initialization is the only thing needed for stability in training. We note that GPIDE with only attention heads is similar to a single multi-headed self-attention block that appears in many transformer architectures; however, we show that attention is the least important type of head in GPIDE and often hurts performance (see Section 5.3).
Perhaps closest to our proposed architecture is PEARL [60], which does a multiplicative combination of Gaussian distributions corresponding to each state-action-reward tuple. However, their algorithm is designed for the MetaRL setting specifically. Additionally, we note that the idea of summations and averaging has been shown to be powerful in prior works. Specifically, Oliva et al. [52] introduced the Statistical Recurrent Unit, an alternative architecture to LSTMs and GRUs that leverages moving averages and performs competitively across several supervised learning tasks.
There are many facets of RL where improvements can be made to robustness, and many works focus on altering the training procedure. They use techniques such as optimizing the policy's worst-case performance [59, 36] or using variational information bottlenecking (VIB) [4] to limit the information used by the policy [42, 33, 21]. In contrast, our work specifically focuses on how architecture choices of history encoders affect robustness, but we note our developments can be used in conjunctions with
these other directions, possibly resulting in improved robustness. We perform additional experiments that consider VIB in Appendix F.1.
Lastly, we note that there is a plethora of work interested in the intersection of reinforcement learning and PID control [35, 25, 39, 22, 74, 12]. These works focus on using reinforcement learning to tune the coefficients of PID controllers (often in MIMO settings). We view these as important works on how to improve PID control using reinforcement learning; however, we view our own work as how to improve deep reinforcement learning by leveraging ideas from PID control.
In this section, we experimentally compare PIDE and GPIDE against recurrent and transformer encoders. In particular, we explore the following questions:
- How does the performance of a policy using PIDE or GPIDE do on tracking problems? In addition, how well can policies adapt to different system parameters and how robust to modelling error are they on these problems? (Section 5.1)
- Going beyond tracking problems, how well does GPIDE perform on higher dimensional locomotion control tasks (Section 5.2)
- How important is each type of head in GPIDE? (Section 5.3)