## 2 Problem Statement  
### Reinforcement Learning Framework for Control Systems  
Reinforcement Learning (RL) is a mathematical framework used to solve sequential decision-making problems, where an agent interacts with an environment to achieve a specific goal by maximizing a cumulative reward signal. These problems are often formulated as a Markov Decision Process (MDP), which is characterized by the 5-tuple: \[\mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{P},r,\gamma),\] (1) where * \(\mathcal{S}\): The _state space_, representing all possible states of the environment. A state \(s_{t}\in\mathcal{S}\) encodes all relevant information about the environment at a given time \(t\). * \(\mathcal{A}\): The _action space_, representing all possible actions that the agent can take. An action \(a_{t}\in\mathcal{A}\) represents a decision or control input applied by the agent at time \(t\). * \(\mathcal{P}(s^{\prime}|s,a)\): The _state transition probability distribution_, which defines the probability of transitioning from a current state \(s\in\mathcal{S}\) to a next state \(s^{\prime}\in\mathcal{S}\) as a result of taking action \(a\in\mathcal{A}\). This function captures the dynamics of the environment and may be deterministic or stochastic. * \(r(s,a)\): The _reward function_, which assigns a scalar reward to each state-action pair \((s,a)\). This reward quantifies the immediate desirability of taking action \(a\) in state \(s\). For example, it might reflect the closeness to a goal or penalties for undesirable events such as collisions. * \(\gamma\in[0,1)\): The _discount factor_, which determines the relative importance of immediate versus future rewards. A smaller \(\gamma\) places more emphasis on immediate rewards, while a value closer to 1 encourages long-term planning. The goal of the RL agent is to find an optimal policy \(\pi(a|s)\), which defines the probability of taking an action \(a\) given a state \(s\), to maximize the expected cumulative discounted reward: \[J(\pi)=\mathbb{E}_{\pi}\Bigg{[}\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t}) \Bigg{]},\] (2) where \(s_{t}\) and \(a_{t}\) represent the state and action at time step \(t\), respectively. The quality of a policy \(\pi\) is often evaluated using the _state-value function_\(V^{\pi}(s)\) or the _action-value function_\(Q^{\pi}(s,a)\), defined as follows: \[V^{\pi}(s)=\mathbb{E}_{\pi}\Bigg{[}\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t}) \,|\,s_{0}=s\Bigg{]},\] (3) \[Q^{\pi}(s,a)=\mathbb{E}_{\pi}\Bigg{[}\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t}) \,\big{|}\,s_{0}=s,a_{0}=a\Bigg{]}.\] (4) The optimal policy \(\pi^{*}\) maximizes these functions and satisfies the Bellman optimality equation: \[Q^{*}(s,a)=r(s,a)+\gamma\mathbb{E}_{s^{\prime}\sim P}\bigg{[}\max_{a^{\prime} \in\mathcal{A}}Q^{*}(s^{\prime},a^{\prime})\bigg{]}.\] (5) The Bellman optimality equation is foundational for reinforcement learning, serving as the theoretical basis for many algorithms aimed at finding optimal policies in sequential decision-making problems. It is rooted in the principle of optimality, which states the following: _An optimal policy has the property that, regardless of the initial state and initial decision, the subsequent decisions must constitute an optimal policy with regard to the state resulting from the first decision_[1]. This principle allows us to decompose the long-term optimization problem into smaller subproblems, leveraging the recursive nature of the problem. For a given state-action pair \((s,a)\), the equation defines the optimal action-value function \(Q^{*}(s,a)\)as the immediate reward plus the maximum possible discounted future reward from subsequent states. The intuition behind the equation lies in the agent's sequential decision-making process. At each step, the agent selects an action \(a\) in state \(s\) to maximize the sum of immediate and future rewards. The equation's recursive nature means the value of the current state-action pair depends on the values of future states, tying local decisions to global optimization.