SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[83] Omer Gottesman, Joseph D. Futoma, Yao Liu, Soanli Parbhoo, Leo Anthony Celi, Emma Brunskill, and Finale Doshi-Velez. Interpretable off-policy evaluation in reinforcement learning by highlighting influential transitions. In International Conference on Machine Learning, 2020.
[84] Nicholay Topin and Manuela Veloso. Generation of policy-level explanations for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 2514-2521, 2019.
[85] Sarath Sreedharan, Siddharth Srivastava, and Subbarao Kambhampati. Tldr: Policy summarization for factored ssp problems using temporal abstractions. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 30, pages 272-280, 2020.
[86] Anton Matsson and Fredrik D Johansson. Case-based off-policy evaluation using prototype learning. In Uncertainty in Artificial Intelligence, pages 1339-1349. PMLR, 2022.
[87] Claire Glanois, Paul Weng, Matthieu Zimmer, Dong Li, Tianpei Yang, Jianye Hao, and Wulong Liu. A survey on interpretable reinforcement learning. arXiv preprint arXiv:2112.13112, 2021.
[88] George A Vouros. Explainable deep reinforcement learning: state of the art and challenges. ACM Computing Surveys, 55(5):1-39, 2022.
[89] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014.
[90] Grady Williams, Nolan Wagener, Brian Goldfain, Paul Drews, James M Rehg, Byron Boots, and Evangelos A Theodorou. Information theoretic mpc for model-based reinforcement learning. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 1714-1721. IEEE, 2017.
[91] Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, and Alexander J. Smola. Meta-q-learning. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=SJeD3CEFPH.
[92] Hao Sun, Ziping Xu, Meng Fang, Zhenghao Peng, Jiadong Guo, Bo Dai, and Bolei Zhou. Safe exploration by solving early terminated mdp. arXiv preprint arXiv:2107.04200, 2021.
[93] Tianwei Ni, Benjamin Eysenbach, and Ruslan Salakhutdinov. Recurrent model-free RL can be a strong baseline for many POMDPs. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 16691-16723. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/v162/ni22a.html.
[94] Hao Sun, Ziping Xu, Zhenghao Peng, Meng Fang, Taiyi Wang, Bo Dai, and Bolei Zhou. Constrained mdps can be solved by early-termination with recurrent models. In NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022.
[95] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
[96] Hao Sun, Boris van Breugel, Jonathan Crabbe, Nabeel Seedat, and Mihaela van der Schaar. Daux: a density-based approach for uncertainty explanations. arXiv preprint arXiv:2207.05161, 2022.
Appendix: Table of Contents
A Missing Proofs 18
A.1 Proof of the Estimation Error Bound for  $v(a_{t})$  18
A.2 Proof of the Existence and Uniqueness 18
B Extended Related Work 19
B.1 Offline RL 19
B.2 Episodic Control and Nearest Neighbor Control 20
B.3 Explainable RL 21
C The Strictly Batch(Offline) Imitation Setting 21
D Further Implementation and Experiment Details 22
D.1 Reproducibility: Code 22
D.2 Learning the Belief Space 22
D.3 Addressing the Scalability Issue: Finding the Minimal Convex Hull Efficiently 22
D.4 Data Generation Process: Heterogeneous Pendulum 23
D.5 Hardware and Running Time 23
D.6 Baseline Implementations 23
D.7 The Model-Free RL Baseline and Its Stability Issue 24
E Additional Qualitative and Quantitative Results 24
E.1 Adaptivity: Visualizing the Decisions of AOC and Quantitative Performance 24
E.2 More Visualization Results and Extended Discussion on the Healthcare Dataset 24
F Additional Experiments 26
F.1 Trade-Off between Accountability and Performance 26
F.2 Visualizing Conservation of AOC 27
F.3 Additional Environment: LunarLanderContinuous 27
F.4 Black-Box Policy as More Efficient Sampler 29
F.5 Identify Control Time OOD Examples with AOC 29
A Missing Proofs
A.1 Proof of Proposition 3.8 (See page 4)
where  $||\cdot ||_{\mathcal{V}}$  is a norm on  $\mathcal{V}$  and  $||\pmb {l}||_{\mathrm{op}} = \inf \left\{\lambda \in \mathbb{R}^{+}:||\pmb {l}(b)||_{\mathcal{V}}\leq \lambda ||\pmb {b}||_{\mathcal{B}}\right\}$  is the operator norm for the linear mapping.
Proof. Leveraging the linearity of operator  $l$ , the definition of the operator norm  $||\cdot||_{\mathrm{op}}$ , and Definition 3.7, we have:
$$
\begin{array}{l} \left| \left| l (\hat {b}) - l \left(b _ {t}\right) \right| \right| _ {\mathcal {V}} = \left| \left| l (\hat {b} - b _ {t}) \right| \right| _ {\mathcal {V}} \\ \leq \left\| \boldsymbol {l} \right\| _ {\mathrm {o p}} \cdot \left\| \hat {\boldsymbol {b}} - b _ {t} \right\| _ {\mathcal {V}} \tag {12} \\ = \left\| \boldsymbol {l} \right\| _ {\mathrm {o p}} \cdot r _ {\mathcal {C}} \left(b _ {t}\right) \\ \end{array}
$$
A.2 Proof of Proposition 3.10 (See page 5)
Lemma A.1 (Affine Independence of  $\tilde{\mathcal{C}}(b_t)$ ). The elements in the belief corpus built on top of  $\tilde{\mathcal{C}}(b_t)$ , as the corpus subset:  $b^c \in \pmb{b}(\tilde{\mathcal{C}}(b_t)) \subset \mathcal{B}$  must be affinely independent, that is
$$
\sum_ {c = 1} ^ {C} \lambda^ {c} b ^ {c} = 0 \wedge \sum_ {c = 1} ^ {C} \lambda^ {c} = 0 \Longrightarrow \lambda^ {c} = 0, \forall c \in [ C ] \tag {13}
$$
Proof. The proof is based on contradiction. Note that by definition, there are  $d_b + 1$  elements in the belief corpus set. If those elements are not affinely independent, it basically means that  $b_t$  can be expressed in a lower dimensional space. In this case, the composition of the convex set is redundant. Without loss of generality, we use  $b^1$  to denote the redundant element
$$
b ^ {1} = - \sum_ {c = 2} ^ {C} \frac {\lambda^ {c}}{\lambda^ {1}} b ^ {c}, \tag {14}
$$
indicating that without  $b^1$ , we still have  $b_t \in \mathcal{CB}(\mathcal{C}')$ , where  $\mathcal{C}' = \tilde{\mathcal{C}}(b_t) \setminus (o^1, a^1, h^1)$ . This contradicts that  $\tilde{\mathcal{C}}(b_t)$  is the minimal hull that contains  $b_t$ .
Proof. Following the assumption, there is at least one trivial corpus subset that exists for  $b_{t}$  — the offline dataset  $\mathcal{D}$  as the corpus subset — such that a zero belief corpus residual can be achieved. The existence of  $\tilde{\mathcal{C}}(b_{t})$  follows the fact the cardinality of the  $\mathcal{D}$  is a finite number. The existence of decomposition on  $\tilde{\mathcal{C}}(b_{t})$  is then a consequence of the definition of the minimal hull.
Based on Lemma A.1, the elements in  $\tilde{\mathcal{C}}(b_t)$  construct the minimal hull and are affinely independent. The uniqueness of the decomposition can be shown by contradiction:
Assume that there are two different decompositions for composing the belief state  $b_{t}$ :
$$
b _ {t} = \sum_ {c = 1} ^ {C} \omega^ {c} b ^ {c} = \sum_ {c = 1} ^ {C} \tilde {\omega} ^ {c} b ^ {c}, \tag {15}
$$
2otherwise, it falls into an out-of-distribution prediction problem and is beyond the scope of this work. Nonetheless, we note that AOC is able to perform OOD detection. See section F.5
where  $\omega^c, \tilde{\omega}^c \in [0,1], \sum_{c=1}^{C} \omega^c = 1, \sum_{c=1}^{C} \tilde{\omega}^c = 1$ . In this case, we have
$$
\begin{array}{l} 0 = \sum_ {c = 1} ^ {C} \omega^ {c} b ^ {c} - \sum_ {c = 1} ^ {C} \tilde {\omega} ^ {c} b ^ {c} \\ = \sum_ {c = 1} ^ {C} \left(\omega^ {c} - \tilde {\omega} ^ {c}\right) b ^ {c} \tag {16} \\ = \sum_ {c = 1} ^ {C} \lambda^ {c} b ^ {c}, \text {w h e r e} \lambda^ {c} \equiv \omega^ {c} - \tilde {\omega} ^ {c}. \\ \end{array}
$$
However,  $\sum_{c=1}^{C} \lambda^c = \sum_{c=1}^{C} (\omega^c - \tilde{\omega}^c) = \sum_{c=1}^{C} \omega^c - \sum_{c=1}^{C} \tilde{\omega}^c = 1 - 1 = 0$  contradicts with the fact that the elements are affinely independent. Hence the proof is completed.
B Extended Related Work
B.1 Offline RL
Offline RL [8-14, 29, 30, 15, 16] has gained increasing attention in recent years due to its potential for solving practical problems, such as robotics control and game playing, where collecting new data can be expensive or time-consuming. However, it also presents several challenges, such as distribution shift and overfitting, which can lead to poor performance when deploying the learned policy to the actual environment. There are several model-free approaches to addressing these challenges in Offline RL. Such as distributional matching [8, 31], regularization techniques to prevent overfitting [32], conservation [12, 33] or adding noise [15] to the policy or using adversarial training [34]. A third approach is to incorporate uncertainty estimation to evaluate the performance of the learned policy on unseen data [13].
On the other hand, model-based RL tackles the problem by first learning world models and then performing planning algorithms on the learned model [22, 23, 35, 36]. In either model-based or model-free approaches, black-box approximators are used; hence, the decision is not transparent.
In offline-RL, both model-based and model-free approaches leverage black-box approximators. As a consequence, the pursuance of accountability can not be achieved through those conventional algorithms.
We would like to note that, although AOC also studies the control problems under the offline setting, its focus goes beyond the conservative efficient learning objective in offline-RL literature. As we have demonstrated in the Table 1, AOC has five distinct properties that are all crucial for accountable offline control tasks:
(P1). Conservation.
(P2). Accountability.
(P3). Low-Data Requirement.
(P4).Adaptivity.
(P5). Reward-Free.
Below, we further explain those properties and corresponding methods in turn. For each of the properties, we start with introducing the definitions, followed by comparisons among AOC and MFRL, MBRL, and BC. The discussion on MFRL and MBRL includes the offline-RL algorithms.
(1) Accountability: the decision-making process is traceable, and the decisions can be supported by concrete examples in the offline dataset.
- AOC: The decision-making process of AOC is supported by a corpus subset from the offline dataset, hence all the decisions are transparent and traceable.
- MFRL: In MFRL, a black-box value network and black-box policy network are learned with the offline dataset. There is no decision support for the black-box policies.
- MBRL: In MBRL, a black-box world model optimized with the offline dataset is used as a proxy of the actual dynamics, and planning algorithms are then applied to such a black-box model to make decisions. Those decisions are not supported by explicit references.
- BC: In BC, a black-box policy is learned through supervised learning. The output of such a policy is hard to be linked with specific training examples.
(2) Conservation: estimations of decision outcomes are interpolated, avoiding aggressive extrapolation that is harmful in offline control.
- AOC: AOC performs conservative decision-making by using decision supports within a minimal convex hull. How such a decomposition in the convex hull improves conservation is justified theoretically by Proposition 3.8 (Estimation Bound) and Proposition 3.10 (Existence and Uniqueness).
- MFRL: In MFRL like CQL and TD3-BC, the conservation is explicitly given as constraints or distribution matching. We would note that conventional MFRL algorithms are not designed for those tasks and suffer from aggressive extrapolation.
- MBRL: Similar to MFRL, external efforts should add conservation to MBRL. Because the conventional design of model-based learning does not address such an issue.
- BC: In BC, the learning objective is to minimize the prediction difference. There is little we can do to aid conservation.
(3) Low-Data: whether a method works in the low-data regime.
- AOC: The decision process of AOC only relies on a few examples forming the minimal convex hull, hence the algorithm performs well under the low-data regime, making it generally applicable to many real-world data-scarce tasks.
- MFRL: In MFRL, the black-box value network and policy network can be designed to be sample-efficient.
- MBRL: In MBRL, sufficient data is always required to learn an accurate world model.
- BC: the performance of BC is highly dependent on the data quality. It is not designed for the low-data regime.
(4) Adaptive: whether the control behavior of a method can be adjusted according to additional constraints as clinical guidelines without modification or re-training.
- AOC: by changing reference examples, i.e., the decision corpus, during test time inference, AOC can seamlessly perform different types of decision-making according to user specifications.
- MFRL: In MFRL, when new data is used, a new value network and policy network need to be re-trained.
- MBRL: In MBRL, the world model construction is independent of the data, hence the decisions can be adaptive by changing a new planning algorithm on top of the world model. No model re-training is needed.
- BC: In BC, a new model needs to be trained with a specified type of decision corpus.
(5) Reward-Free: availability of extension to the strictly batched imitation setting where rewards are unavailable.
- AOC: AOC can be extended to the strictly batched imitation setting where rewards are unavailable.
- MFRL: In MFRL, the Q-values can not be calculated without the reward function.
- MBRL: In MBRL, the planning algorithms do not have a clear objective to optimize without reward signals.
- BC: BC is not affected by the absence of reward signals, because it does not need the reward to learn its policy.
B.2 Episodic Control and Nearest Neighbor Control
Episodic Memory and Episodic Control [17-19], inspired by biological learning mechanism, are studied as an alternative way of policy learning [37]. Follow-up works introduce various modifications and extend episodic control to the continuous domains. e.g., Hu et al. [38] introduces Generalized Episodic Memory (GEM) which effectively organizes the state-action values of episodic memory in a generalizable manner and supports implicit planning on memorized trajectories. Ma et al. [39] leverages episodic memory in offline RL setting with a pessimistically estimated value function. Li et al. [40] proposing a novel state-abstractor framework for episodic control and improving the learning efficiency in continuous control benchmarks. In all those works, a value function
resembling the hippocampus episodic memorization mechanism is introduced as an alternative to Q-learning [41, 42] that updates the state-action values with temporal difference learning or Monte-Carlo estimation [43-45].
In the continuous control domains, policy gradient methods [46, 47] or supervised learning-based methods [48, 49, 10, 50] are then applied for the policy improvement. All of those approaches are limited to black-box value-based learning and require additional black-box policy networks in the continuous control domain, whereas our proposed method performs a transparent decision-making process without explicit policy learning.
Explicit nearest neighbor methods that perform decision-making according to training-time similar trajectories have been studied theoretically [20] and empirically [21]. Although those methods also enjoy transparency, they suffer from the problem of the curse of dimensionality. Moreover, defining the nearest neighbor with a heuristically determined Euclidean metric also suffers the problem of aggressive extrapolation and thus is not suitable for the offline setting [8, 12].
B.3 Explainable RL
Understanding the decisions made by RL agents is a key issue in many high-stake real-world domains such as autonomous driving [51], finance [52, 53] and healthcare [54, 28, 55-59]. Previous Explainable-RL (XRL) literature can be broadly classified with a taxonomy of three classes [60]: (1) Feature importance, that includes learning policy through an explainable policy class [61-63], converting black-box models into an interpretable format [64-68], and natural language [69-71] or saliency map based explanations [72-74]; (2) Transparent Learning Process that reveals the influences of MDP ingredients during the learning process, including methods that learn to predict the counterfactual outcomes for decision-making [75-77], decompose the learning objective [78-81], and identifying the crucial training datum [82, 83]; (3) Policy Level, which illustrates the long-term behavior of the agent [84-86]. For more extensive discussions on XRL literature, we refer the readers also to [87, 88]. Different from previous approaches, our work introduces the first example-based explanation for policy learning. Supported by training dataset examples, the execution of our control algorithm is accountable.
C The Strictly Batch(Offline) Imitation Setting
The instant reward may not be contained in the offline dataset in strictly batch imitation (SBI) [31] learning settings such as clinical treatment and healthcare scenarios. In such cases, the value of actions can not be estimated through Monte Carlo, and it is generally impossible to learn a belief state based on the value prediction. In such a case, we need to adapt the Accountable Offline Controller accordingly.
In such a setting, the dataset  $\mathcal{D}_{\mathrm{SBI}} = \{o_t^i,a_t^i,o_{t + 1}^i\}_{t = 1,\dots ,T}^{i = 1,\dots ,N}$  contains only sequential observations  $o_{t}^{i},o_{t + 1}^{i},i\in [N],t\in [T]$ , actions  $a_{t}^{i},i\in [N],t\in [T]$  performed by behavior policies, which is always an expert or near-expert controller, and transition histories  $h_t^i,i\in [N],t\in [T]$  that can be composed of the former quantities.
The Accountable Offline Controller should be adapted to handle this setting. Specifically, we can still define the corpus subset as
Definition C.1 (SBI Corpus Subset). A SBI Corpus Subset  $\mathcal{C}_{\mathrm{SBI}}$  is defined as a subset of an offline dataset  $\mathcal{D}_{\mathrm{SBI}}$ , indexed by  $[C] := \{1, 2, \dots, C\}$  — the natural numbers between 1 and  $C$ .
$$
\mathcal {C} _ {\mathrm {S B I}} = \left\{\left(o ^ {c}, a ^ {c}, h ^ {c}\right) \in \mathcal {D} _ {\mathrm {S B I}} \mid c \in [ C ] \right\}. \tag {17}
$$
Property C.2. (SBI Linear Restricted Belief) The policy function  $\pi : \mathcal{O} \times \mathcal{H} \mapsto \mathcal{A}$  can be decomposed as  $\pi = l \circ b$ , where  $b : \mathcal{O} \times \mathcal{H} \mapsto \mathcal{B} \subseteq \mathbb{R}^{d_b}$  maps the joint observation-history space to a  $d_b$  dimensional belief variable  $b_t = b(o_t, h_t)$  and  $l : \mathcal{B} \mapsto \mathcal{A}$  is a linear function that maps the belief  $b_t$  to an output  $l(b_t) \in \mathcal{A}$ .
Then any control time behavior generated by policy  $\pi$  is accountable in the sense that it can be decomposed by the belief corpus, defined as
Definition C.3. (SBI Belief Corpus) A SBI Belief Corpus  $\pmb{b}(\mathcal{C}_{\mathrm{SBI}})$  is defined by applying the belief function  $\pmb{b}$  to the corpus subset  $\mathcal{C}_{\mathrm{SBI}}$ ,
$$
\boldsymbol {b} \left(\mathcal {C} _ {\mathrm {S B I}}\right) = \left\{b ^ {c} = \boldsymbol {b} \left(o ^ {c}, h ^ {c}\right) \mid \left(o ^ {c}, a ^ {c}, h ^ {c}\right) \in \mathcal {C} _ {\mathrm {S B I}} \right\} \subset \mathcal {B}, \tag {18}
$$
on top of which we can define the Belief Corpus Hull in the SBI setting:
Definition C.4. (SBI Belief Corpus Hull) The SBI Belief Corpus Convex Hull spanned by a corpus subset  $\mathcal{C}_{\mathrm{SBI}}$  with the belief corpus  $b(\mathcal{C}_{\mathrm{SBI}})$  is the convex set
$$
\mathcal {C B} \left(\mathcal {C} _ {\mathrm {S B I}}\right) = \left\{\sum_ {c = 1} ^ {C} w ^ {c} b ^ {c} \mid w ^ {c} \in [ 0, 1 ], b ^ {c} \in \boldsymbol {b} \left(\mathcal {C} _ {\mathrm {S B I}}\right), \forall c \in [ C ], \sum_ {c = 1} ^ {C} w ^ {c} = 1 \right\}, \tag {19}
$$
followed by the concept of Minimal Hull in the SBI setting defined as
Definition C.5 (SBI Minimal Hull). Denoting the decision corpora whose belief convex hull contain a belief variable  $b_{t}$  by
$$
\mathcal {C} _ {\mathrm {S B I}} \left(b _ {t}\right) = \left\{\mathcal {C} _ {\mathrm {S B I}} \subseteq \mathcal {D} _ {\mathrm {S B I}} \Bigg | b _ {t} \in \mathcal {C B} \left(\mathcal {C} _ {\mathrm {S B I}}\right) \right\}
$$
Among those subsets, the one that contains  $d_b + 1$  decision corpora and has the smallest hyper-volume forms the minimal hull, denoted by  $\tilde{\mathcal{C}}_{\mathrm{SBI}}(b_t)$ :
$$
\tilde {\mathcal {C}} _ {\mathrm {S B I}} (b _ {t}) := \min _ {\mathcal {C} _ {\mathrm {S B I}}} \int_ {\mathcal {C B} (\mathcal {C} _ {\mathrm {S B I}})} d V
$$
Similar to the property in AOC, in the SBI setting, the control time decisions can be decomposed with examples in the offline dataset that constructs such an SBI Minimal Hull,
$$
\pi \left(a _ {t} \mid o _ {t}, h _ {t}\right) = \boldsymbol {l} \circ \boldsymbol {b} \left(o _ {t}, h _ {t}\right) = \boldsymbol {l} \circ \sum_ {c = 1} ^ {C} w ^ {c} \cdot \boldsymbol {b} \left(o ^ {c}, h ^ {c}\right) = \sum_ {c = 1} ^ {C} w ^ {c} \cdot \boldsymbol {l} \circ \boldsymbol {b} \left(o ^ {c}, h ^ {c}\right) = \sum_ {c = 1} ^ {C} w ^ {c} \cdot a ^ {c},
$$
where  $(o^c, a^c, h^c) \in \tilde{\mathcal{C}}_{\mathrm{SBI}}(b_t)$ .
D Further Implementation and Experiment Details
D.1 Reproducibility: Code
We elaborate on our implementation details in this section. Our code is available at https://github.com/orgs/vanderschaarlab/repository/accountableofflinerl.
D.2 Learning the Belief Space
To efficiently encode the information in historical transition, in our work, we employ the Gated Recurrent Units (GRU) [89] to map fixed-length transition histories into embedding vector variables, followed by 3 fully connected layers as the belief function  $\pmb{b}$ . In principle, any other recurrent networks should also be able to process such context information. Table 4 presents the hyper-parameters we use in the belief learning process.
D.3 Addressing the Scalability Issue: Finding the Minimal Convex Hull Efficiently
Rigorously searching for the minimal convex hull in the belief space is a combinatorial optimization problem. In our implementation, we leverage a heuristic search method that first reduces the search space by looking for  $k$ -nearest neighbors in the belief space, and then build the approximate minimal convex hull on top of those  $k$ -nearest neighbors of the control time belief  $b_{t}$ . With  $d_{b}$ -dimensional belief space, the minimal convex hull will contain at most  $d_{b} + 1$  examples, hence we can set  $k = 2(d_{b} + 1)$ , and conduct the combinatorial optimization on those  $k$  examples, which is much easier than the original problem (reduce combinatorial problem Select  $d_{b} + 1$  out of  $N$  into Select  $d_{b} + 1$  out of  $k$ ).
Table 4: Hyperparameters in learning the belief function.
Hyper-Param	Choice
Context Model	GRU
Hidden Unit Number	128
Hidden Recurrent Layer	1
Batch Size	500
Epochs	4000
Optimizer	Adam
Learning Rate	0.001
Memory Length	4
Embedding Dimension	20
D.4 Data Generation Process: Heterogeneous Pendulum
The classical control task of Pendulum has the goal to swing up and balance a pendulum using a control input. In the control task, a policy can apply torque to the joint in order to swing the pendulum up and then maintain its upright position. The state of the system is defined by the pendulum's angle and angular velocity, and the action is the torque applied to the joint. The reward function typically provides a positive reward for keeping the pendulum upright and a negative reward for large torques or deviations from the upright position.