SlowGuess/NeurIPS_2023_OCR huggingface dataset:
-  $(\alpha)$  Learning Rate: Cartpole  $= 10^{-2}$ , Ant  $= 5 \cdot 10^{-4}$ , Hopper  $= 5 \cdot 10^{-3}$
-  $(\alpha)$  Batch Size for Actor Update: Cartpole, Ant = 2048, Hopper = 8192
$(\alpha)$  Number of Epochs for Actor Update: 16
$\cdot (\alpha)\alpha_{0}$  : Cartpole,  $\mathrm{Ant} = 5\cdot 10^{-1}$  , Hopper  $= 5\cdot 10^{-3}$
$(\alpha)\max (\alpha)\colon 1.0$
$\cdot (\alpha)\beta$  :1.02
$\cdot (\alpha)\delta_{det}$  ..0.4
$(\alpha)\delta_{o o r r}$  : Cartpole, Hopper  $= 0.75$  , Ant  $= 0.5$
(PPO) Learning Rate: Cartpole  $= 3 \cdot 10^{-4}$ , Ant, Hopper  $= 10^{-4}$
(PPO) Learning Rate Scheduler: Cartpole = Adaptive, Ant, Hopper = Constant
(PPO) Batch Size for Actor Update: 2048
(PPO) Number of Epochs for Actor Update: 5
(PPO)  $\epsilon_{clip}$ : 0.2
${}^{2}$  Learning rate is adaptively controlled,so that the KL divergence between the updated policy and the original policy is maintained at certain value, 0.008 in this case.
Traffic Problems (Section 5.3) For these problems, common settings are as follows.
- Actor Network: MLP with [512, 64, 64] layers and ELU activation function
- Critic Network: MLP with [64, 64] layers and ELU activation function
- Critic Hyperparameters: Learning rate  $= 10^{-3}$ , Iterations  $= 16$ , Batch Size  $= 4$
Number of parallel environments: 64
Horizon Length: 32
-  $\gamma$  (Discount factor): 0.99
$\tau$  (GAE): 0.95
- Learning Rate:  $3 \cdot 10^{-4}$
- Learning Rate Scheduler: Linear
- Learning Rate:  $10^{-3}$
- Learning Rate Scheduler: Linear
- Learning Rate:  $3 \cdot 10^{-4}$
- Learning Rate Scheduler: Constant
- Batch Size for Actor Update: 2048
Number of Epochs for Actor Update: 5
$\epsilon_{clip}$  : 0.2
$(\alpha)$  Learning Rate:  $10^{-5}$
$\bullet$ $(\alpha)$  Batch Size for Actor Update: 2048
$\bullet$ $(\alpha)$  Number of Epochs for Actor Update: 16
$(\alpha)\alpha_{0}\colon 10^{-1}$
$(\alpha)\max (\alpha)\colon 1.0$
$(\alpha)\beta :1.1$
$(\alpha)\delta_{det}$  .. 0.4
$\bullet (\alpha)\delta_{o o r r}:$  0.5
(PPO) Learning Rate:  $3 \cdot 10^{-4}$
(PPO) Learning Rate Scheduler: Constant
(PPO) Batch Size for Actor Update: 2048
(PPO) Number of Epochs for Actor Update: 5
(PPO)  $\epsilon_{clip}$ : 0.2
7.5 Problem Definitions
Here we present details about the problems we suggested in Section 5.
(a) De Jong's Function
Figure 8: Landscape of target functions in 2 dimensions.
(b) Ackley's Function
7.5.1 Function Optimization Problems (Section 5.1)
(N-dimensional) De Jong's function  $(F_{D})$  and Ackley's function  $(F_{A})$  are defined for  $n$ -dimensional vector  $x$  and are formulated as follows [Molga and Smutnicki, 2005]:
$$
\begin{array}{l} F _ {D} (x) = \sum_ {i = 1} ^ {n} x _ {i} ^ {2}, \\ F _ {A} (x) = - 2 0 \cdot e x p (- 0. 2 \cdot \sqrt {\frac {1}{n} \sum_ {i = 1} ^ {n} x _ {i} ^ {2}}) - \\ e x p \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \cos (2 \pi x _ {i})\right) + 2 0 + e x p (1). \\ \end{array}
$$
As we mentioned in Section 5.1, we multiply -1 to these functions to make these problems maximization problems. Also, even though these two functions have their optimum at  $x = 0$ , they exhibit very different landscape as shown in Figure 8.
When it comes to the formal definition as RL environments, we can define these problems as follows.
- Episode Length: 1
- Observation: [0]
- Action:  $n$ -dimensional vector  $\mathbf{x}$ , all of which element is in  $[-1, 1]$ .
- Reward: De Jong's Function  $= F_{D}(5.12\mathbf{x})$ , Ackley's Function  $= F_{A}(32.768\mathbf{x})$
7.5.2 Traffic Problems (Section 5.3)
Before introducing pace car problem, which we specifically discussed in this paper, we'd like to briefly point out the traffic model that we used to simulate motions of individual vehicles in the traffic environment.
Traffic Model In our traffic simulation, the state  $s \in \mathbb{R}^{2N}$  is defined as a concatenation of all vehicle states,  $q_{n} \in \mathbb{R}^{2}$ , where  $1 \leq n \leq N$  stands for vehicle ID.  $q_{n}$  can be represented simply with the vehicle's position  $(x_{n})$  and velocity  $(v_{n})$ .
$$
q _ {n} = \left[ \begin{array}{c} x _ {n} \\ v _ {n} \end{array} \right]
$$
$$
s = \left[ q _ {1} ^ {T}, \dots , q _ {N} ^ {T} \right] ^ {T},
$$
where  $n$  is the vehicle ID. Since this traffic state  $s$  changes over time, we represent the traffic state at timestep  $t$  as  $s_t$ . Then our differentiable traffic simulator is capable of providing the gradient of the simulation state at the next timestep  $t + 1$  with respect to the state at the current timestep  $t$ , or  $\frac{ds_{t+1}}{ds_t}$ .
This is possible because we use the Intelligent Driver Model (IDM) to model car-following behavior in our simulator [Treiber et al., 2000], which is differentiable. IDM describes vehicle speed  $\dot{x}$  and acceleration  $\dot{v}$  as a function of desired velocity  $v_{0}$ , safe time headway in meters  $T$ , maximum acceleration  $a$ , comfortable deceleration  $b$ , the minimum distance between vehicles in meters  $\delta$ , vehicle length  $l$ , and difference in speed with the vehicle in front  $\Delta v_{\alpha}$  as follows:
$$
\dot {x} _ {\alpha} = \frac {\mathrm {d} x _ {\alpha}}{\mathrm {d} t} = v _ {\alpha},
$$
$$
\dot {v} _ {\alpha} = \frac {\mathrm {d} v _ {\alpha}}{\mathrm {d} t} = a \left(1 - \left(\frac {v _ {\alpha}}{v _ {0}}\right) ^ {\delta} - \left(\frac {s ^ {*} (v _ {\alpha} , \Delta v _ {\alpha})}{s _ {\alpha}}\right) ^ {2}\right),
$$
$$
s _ {\alpha} = x _ {\alpha - 1} - x _ {\alpha} - l _ {\alpha - 1},
$$
$$
s ^ {*} (v _ {\alpha}, \Delta v _ {\alpha}) = s _ {0} + v _ {\alpha} T + \frac {v _ {\alpha} \Delta v _ {\alpha}}{2 \sqrt {a b}},
$$
$$
\Delta v _ {\alpha} = v _ {\alpha} - v _ {\alpha - 1},
$$
where  $\alpha$  means the order of the vehicle in the lane. Therefore,  $(\alpha - 1)$ -th vehicle runs right in front of  $\alpha$ -th vehicle, and this relationship plays an important role in IDM. Also note that the computed acceleration term  $(\dot{v})$  is differentiable with respect to the other IDM variables.
IDM is just one of many car-following models used in traffic simulation literature. We choose IDM for its prevalence in previous mixed autonomy literature, however, any ODE-based car-following model will also work in our simulator as far as it is differentiable. In our simulator, automatic differentiation governs the gradient computation.
Lane Under IDM, lane membership is one of the most vital variables for simulation. This is because IDM builds on the leader-follower relationship. When a vehicle changes its lane with lateral movement, as shown as red arrows in Figure 9, such relationships could change, and our gradients do not convey any information about it, because lane membership is a discrete variable in nature. Note that this lateral movement also affects the longitudinal movement, rendered in green arrows in Figure 9, of a vehicle, and gradient from it is valid as far as the lane membership does not change. However, when the lane membership changes, even if our simulator gives gradient, it tells us only "partial" information in the sense that it only gives information about longitudinal behavior. Therefore, we could say that analytical gradients we get from this environment is "incomplete" (not useless at all, because it still gives us information about longitudinal movement), and thus biased.
Pace Car Problem In this problem, there is a single autonomous vehicle that we have to control to regulate other human driven vehicles, which follow IDM, to run at the given target speed. Since human driven vehicles control their speeds based on the relationship to their leading vehicles, our autonomous vehicle has to control itself to run in front of those human driven vehicles and adjust their speeds to the target speed. The number of lanes and human driven vehicles varies across different environments as follows.
- Single Lane: Number of lanes  $= 1$  ,Number of vehicles per lane  $= 1$
- 2 Lanes: Number of lanes  $= 2$ , Number of vehicles per lane  $= 2$
- 4 Lanes: Number of lanes  $= 4$ , Number of vehicles per lane  $= 4$
- 10 Lanes: Number of lanes  $= 10$ , Number of vehicles per lane  $= 1$
See Figure 10 for 10-lane environment.
Then we can formally define RL environments as follows, where  $N$  is the number of human driven vehicles in total, and  $v_{tgt}$  is the target speed.
- Episode Length: 1000
- Observation:  $s \in  {\mathbb{R}}^{2N}$
- Action:  $\mathbf{a} \in \mathbb{R}^2$  ( $\mathbf{a}_0 =$  Acceleration,  $\mathbf{a}_1 =$  Steering)
- Reward:  $1 - \frac{1}{N}\sum_{i = 1}^{N}\min (\frac{|v_i - v_{gt}|}{v_{gt}},1)$
However, to facilitate learning, we additionally adopted following termination conditions to finish unpromising trials early.
- Terminate when autonomous vehicle collides with human driven vehicle
- Terminate when autonomous vehicle goes out of lane
- Terminate when autonomous vehicle runs too far away from the other vehicles
- Terminate when autonomous vehicle falls behind the other vehicles
When one of these conditions is met, it gets reward of  $-1$  and the trial terminates.
7.6 Renderings
7.6.1 Traffic Simulation
Gradient flow As described in Appendix 7.5.2, analytical gradients are only valid along longitudinal direction, as illustrated with green arrows in Figure 9. When a vehicle changes lane with lateral movement, as shown with red arrows, even though such change would incur different behaviors of following vehicles in multiple lanes, analytical gradients cannot convey such information. However, they still tell us information about longitudinal behavior - which is one of the motivations of our research, to use analytical gradients even in these biased environments.
Figure 9: Traffic Environment. In traffic flows, only partial gradients of forward simulation are known; acceleration along a (longitudinal) traffic lane is continuous and admits gradient flow (green), while (lateral) lane-changing is a discrete event and thus prohibits gradient flow (red). Analytical gradients convey only partial information about traffic dynamics.
Pace Car Problem We additionally provide renderings of the 10-lane pace car problem defined in Appendix 7.5.2. In Figure 10, we can observe 10 parallel lanes, 10 human driven vehicles rendered in yellow, and the autonomous vehicle rendered in blue that we have to control. As shown there, human driven vehicles adjust their speeds based on IDM when the autonomous vehicle blocks their way. Therefore, the autonomous vehicle has to learn how to change their lanes, and also how to run in appropriate speed to regulate the following vehicles. Our experimental results in Section 5.3 show that our method achieves better score than the baseline methods by adopting biased analytical gradients in PPO.
(a) 0-th frame
(c) 300-th frame
(b) 150-th frame
(d) 450-th frame
Figure 10: 10-Lane pace car environment. The autonomous vehicle (Blue) has to interfere paths of the other human driven vehicles (Yellow) to limit their speeds to  $10m / s$ . Even though it is a hard problem to achieve high score, our method achieved far better score than the baseline methods.
7.7 Computational Costs
We provide wall clock training times for each method in traffic environments to finish all the training epochs in Table 2. All of the trainings were done based on the settings in Appendix 7.5.2. Note that the longer training time does not always mean that the method is worse than the others – the method could have achieved better results in shorter time, while requiring much longer time to complete all the training epochs.
In Table 2, we can observe that GI-PPO requires a little more time than RP, as it computes analytical gradients as RP does, but requires an additional time for controlling  $\alpha$  and doing PPO update. However, the additional cost is not so significant. In contrast, LR+RP consumes much more time than the other methods, even though it is based on the same spirit as ours. This is because it has to backpropagate through each trajectory in the experience buffer to estimate the sample variance of the gradients. Therefore, we can see that our method attains better results than LR+RP (Figure 4) with lower computational cost.
Table 2: Average (wall clock) training time (sec) for traffic problems.
Problem	LR	RP	PPO	LR+RP	GI-PPO
Single Lane	120	282	181	1335	332
2 Lanes	142	330	218	1513	411
4 Lanes	304	646	366	2093	813
10 Lanes	244	496	294	2103	533