SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left\| \Pi_ {\mathbf {W}} \bar {T} ^ {\bar {\pi}} \mathbf {V} _ {\text {p r o j .}} ^ {1} - \Pi_ {\mathbf {W}} \bar {T} ^ {\bar {\pi}} \mathbf {V} _ {\text {p r o j .}} ^ {2} \right\| _ {\mathbf {W}} ^ {2} (28) \\ = \left\| \Pi_ {\mathbf {W}} \left(\bar {T} ^ {\bar {\pi}} \mathbf {V} _ {\text {p r o j .}} ^ {1} - \bar {T} ^ {\bar {\pi}} \mathbf {V} _ {\text {p r o j .}} ^ {2}\right) \right\| _ {\mathbf {W}} ^ {2} (29) \\ \leq \left\| \Pi_ {\mathbf {W}} \left(\bar {T} ^ {\bar {\pi}} \mathbf {V} _ {\text {p r o j .}} ^ {1} - \bar {T} ^ {\bar {\pi}} \mathbf {V} _ {\text {p r o j .}} ^ {2}\right) \right\| _ {\mathbf {W}} ^ {2} + \left\| (\mathbf {I} - \Pi_ {\mathbf {W}}) \left(\bar {T} ^ {\bar {\pi}} \mathbf {V} _ {\text {p r o j .}} ^ {1} - \bar {T} ^ {\bar {\pi}} \mathbf {V} _ {\text {p r o j .}} ^ {2}\right) \right\| _ {\mathbf {W}} ^ {2} (30) \\ = \left\| \bar {T} ^ {\bar {\pi}} \mathbf {V} _ {\text {p r o j .}} ^ {1} - \bar {T} ^ {\bar {\pi}} \mathbf {V} _ {\text {p r o j .}} ^ {2} \right\| _ {\mathbf {W}} ^ {2} (31) \\ = \left\| \bar {\mathbf {R}} + \gamma \bar {\mathbf {P}} \mathbf {V} _ {\text {p r o j .}} ^ {1} - \left(\bar {\mathbf {R}} + \gamma \bar {\mathbf {P}} \mathbf {V} _ {\text {p r o j .}} ^ {2}\right) \right\| _ {\mathbf {W}} ^ {2} (32) \\ = \gamma \Sigma_ {k = 1} ^ {j} \rho^ {\bar {\pi}} (\bar {s} _ {k}) \left(\Sigma_ {l = 1} ^ {j} \bar {P} ^ {\bar {\pi}} (\bar {s} _ {l} | \bar {s} _ {k}) \left(V _ {l} ^ {1, \text {p r o j .}} - V _ {l} ^ {2, \text {p r o j .}}\right)\right) ^ {2} (33) \\ \leq \gamma \Sigma_ {k = 1} ^ {j} \rho^ {\bar {\pi}} (\bar {s} _ {k}) \Sigma_ {l = 1} ^ {j} \bar {P} ^ {\bar {\pi}} (\bar {s} _ {l} | \bar {s} _ {k}) \left(V _ {l} ^ {1, \text {p r o j .}} - V _ {l} ^ {2, \text {p r o j .}}\right) ^ {2} (34) \\ = \gamma \Sigma_ {l = 1} ^ {j} \Sigma_ {k = 1} ^ {j} \rho^ {\bar {\pi}} (\bar {s} _ {k}) \bar {P} ^ {\bar {\pi}} (\bar {s} _ {l} | \bar {s} _ {k}) \left(V _ {l} ^ {1, \text {p r o j .}} - V _ {l} ^ {2, \text {p r o j .}}\right) ^ {2} (35) \\ = \gamma \Sigma_ {l = 1} ^ {j} \rho^ {\bar {\pi}} (\bar {s} _ {l}) \left(V _ {l} ^ {1, \text {p r o j .}} - V _ {l} ^ {2, \text {p r o j .}}\right) ^ {2} (36) \\ \because \Sigma_ {k = 1} ^ {j} \rho^ {\bar {\pi}} (\bar {s} _ {k}) \bar {P} ^ {\bar {\pi}} (\bar {s} _ {l} | \bar {s} _ {k}) = \rho^ {\bar {\pi}} (\bar {s} _ {l}) \text {b y} \\ = \gamma \| \mathbf {V} _ {\text {p r o j .}} ^ {1} - \mathbf {V} _ {\text {p r o j .}} ^ {2} \| _ {\mathbf {W}} ^ {2}, (37) \\ \end{array}
$$
where Equality (31) holds by the Pythagorean Theorem, and Inequality (34) follows from the Jensen's inequality.
C Environment Details
(b)
Figure 5: MuJoCo continuous control environments in the experiment: (a) HalfCheetah-v3, (b) Walker2d-v3, (c) Hopper-v3, (d) Swimmer-v3, (e) InvertedPendulum-v2, and (f) Reacher-v2.
Table 2: Details of the MuJoCo environment
Action dimension	State dimension	Timestep (s)
HalfCheetah-v3	6	17	0.05
Walker2d-v3	6	17	0.008
Hopper-v3 (s)	3	11	0.008
Swimmer-v3	2	6	0.04
InvertedPendulum-v2	1	4	0.04
Reacher-v2	2	11	0.02
D Implementation Details
In this section, we provide the implementation details of the algorithms used in this study. All methods presented in the experiment used an SAC as their base learning algorithm with the following characteristics:
- Stochastic Gaussian policy approaches.
- Automated entropy adjustment [14].
- Clipped-double  $Q$ -learning, which was introduced in the TD3 algorithm to prevent overestimating the  $Q$ -value [12].
- Soft target update, which changes the target values slowly and improves the learning stability[21].
- Adam optimizer, a variant of the stochastic gradient descent method [19].
The details of the hyperparameters are presented in Table 3.
Table 3: Hyperparameters for BPQL and the baselines.
Hyperparameters	Values
Critic network	256, 256
Policy network	256, 256
Discount factor	0.99
Replay memory size	1 M
Minibatch size	256
Learning rate	0.0003
Target entropy	-dim|A|
Target smoothing coefficient	0.995
Optimizer	Adam
E Plots of Performance Comparison
In this section, we present several plots of the performance curves of BPQL and other baselines in the environments with delayed timesteps of 3, 6 and 9.
Delayed timesteps=3
Delayed timesteps=6
Delayed timesteps=9
Figure 6: Performance curves for each algorithm in continuous tasks with three, six and nine delayed timesteps.
F.1 Extremely Long Delayed Environment
We conducted additional experiments to determine how well BPQL could solve the control problem even in very long delayed environments. Figure 7 shows that in BPQL, the policy is improved through interaction with the environment in a very long delayed environment, but the conventional methods find it difficult improve their policy.
Figure 7: HalfCheetah-v3 environment with 20 delayed timesteps (left) and InvertedPendulum-v2 environment with 25 delayed timesteps (right). Each timestep is  $0.05\mathrm{s}$  and  $0.04\mathrm{s}$  for HalfCheetah-v3 and InvertedPendulum-v2 environments, respectively. In the InvertedPendulum-v2 environment, to prevent early failure, the agent uses a pretrained policy to determine the actions of the first 25 (=number of delayed timesteps) timesteps. We repeated the test for the tasks five times with different random seeds.
F.2 Action delay
We also evaluated BPQL and other baselines in action delayed environments. Figure 8 illustrates the interaction between an agent and action delayed environment.
Figure 8: In an action delayed environment, the environment takes the delayed action  $a_{t - d}$  instead of the current action  $a_t$ , where  $d$  is the number of delayed timesteps.
F.2.1 Augmented State in an Action Delayed Environment
In an action delayed environment, the augmented state at time  $t + d$  is defined as:
$$
\bar {s} _ {t + d} = \left(s _ {t}, a _ {t - d}, a _ {t - d + 1}, \dots , a _ {t - 1}\right), \tag {38}
$$
where the  $d$  is the number of delayed timesteps for an action. The CDMDP for an action delayed environment is defined as a tuple  $(\mathcal{X},\mathcal{A},R,P,\gamma ,d_a)$ , where  $\mathcal{X}$  is the original state space,  $\mathcal{A}$  is the action space,  $R:\mathcal{X}\times \mathcal{A}\mapsto \mathbb{R}$  is the reward function,  $P:\mathcal{X}\times \mathcal{A}\times \mathcal{X}\mapsto [0,1]$  is the transition kernel,  $\gamma \in (0,1)$  is a discount factor, and  $d_{a}$  is the number of delayed timesteps for an action.
This CDMDP can be reduced to a MDP  $(S, \mathcal{A}, \bar{\mathcal{R}}, \bar{P}, \gamma)$ , where  $\mathcal{S} = \mathcal{X} \times \mathcal{A}^{d_a}$  is an augmented state space,  $\bar{\mathcal{R}}: S \times \mathcal{A} \mapsto \mathbb{R}$  is the reward mapping, and  $\bar{P}: S \times \mathcal{A} \times S \mapsto [0,1]$  is the transition kernel as in the case of observation delayed environments [18]. BPQL for action delayed environments uses the augmented state defined in Equation (38) instead of the one defined in Equation (6).
F.2.2 Results
Figure 9: Performance curves for each algorithm in continuous tasks with 3 action delayed timesteps. We repeated the test for the tasks five times with different random seeds.
Figure 10: Performance curves for each algorithm in continuous tasks with 6 action delayed timesteps. We repeated the test for the tasks five times with different random seeds.
Figure 11: Performance curves for each algorithm in continuous tasks with 9 action delayed timesteps. We repeated the test for the tasks five times with different random seeds.
F.3 Discrete Domain
We conducted additional experiments on the classical discrete control OpenAI gym [7] tasks: CartPole-v1 and LunarLander-v2. In the CartPole-v1 environment, there are two available actions, while in the LunarLander-v2 environment, four actions are available. We compared the performance of BPQL with the Delayed-Q algorithm [10] and Delayed-SAC in various action delay settings. To adapt BPQL and Delayed-SAC to the discrete setting, we implemented a discrete version of SAC [9] for both of them, instead of the original continuous version. We trained the agent for 1 million timesteps in all environments (except for Delayed-SAC and BPQL in the CartPole-v1 task, which was trained for 0.5 million timesteps), and obtained results using 10 random seeds.
The results showed that all algorithms performed similarly in the CartPole-v1 environment, which is considered relatively easier. However, in the LunarLander-v2 environment, as the delay increased, both Delayed-Q and Delayed-SAC showed a notable deterioration in performance. They even failed to control the agent in many episodes under the 10 timesteps delay setting. In contrast, the proposed BPQL algorithm consistently maintained high performance across all tested delay settings (3,5, and 10).
Table 4: Results of OpenAI Gym's discrete tasks. All results are shown with the standard deviation over the ten trials denoted by  $\pm$ . The scores of Delayed-Q in CartPole-v1 are sourced from the original paper [10] and marked with *
Environment	Delayed-Q [10]	Delayed-SAC (discrete ver.)	BPQL (discrete ver.)
CartPole-v1 (d=15)	414±14*	411±75	464±60
CartPole-v1 (d=25)	324±7*	396±67	367±94
LunarLander-v2 (d=3)	247±23	285±3	275±10
LunarLander-v2 (d=5)	173±93	282±4	256±53
LunarLander-v2 (d=10)	-84±67	-72±178	245±59