## 2 Materials and Methods
### Background
#### 2.1.1 Reinforcement Learning and Inverse Reinforcement Learning  
Reinforcement Learning (RL) is a type of machine learning where an agent learns to make decisions by interacting with an environment. The problem can be formulated as a Markov Decision Process (MDP), defined as the tuple \(\mathcal{M}=\{\mathcal{S},\mathcal{A},p(s,a),\rho_{0},r\}\). Here, \(\mathcal{S}\) represents the state space, which encompasses all possible states the environment can be in. The action space, denoted by \(\mathcal{A}\), consists of all possible actions the agent can take. The dynamics function \(p(s,a)\) describes the transition probabilities, specifying how the environment responds to the agent's actions. The initial state distribution \(\rho_{0}\) defines the probability distribution over the initial states of the environment, while the reward function \(r\) assigns a numerical reward to each state-action pair, providing feedback to the agent regarding the effectiveness of its actions.  
The primary objective of the agent is to derive a policy \(\pi(a|s)\), which is a strategy that outlines the probability of taking action \(a\) given the current state \(s\). To achieve this, the agent seeks to maximize the expected sum of rewards, denoted as the return \(R\) and represented by the equation  
\[R^{\pi}=\mathbb{E}_{s_{t},a_{t}\sim\pi}\left[\sum_{t=0}^{T}\gamma^{t}r(s_{t},a _{t})\right] \tag{1}\]  
In this equation, \(R^{\pi}\) denotes the return or expected cumulative reward following policy \(\pi\). The variable \(T\) signifies the time horizon, defining the length of the episode, while \(\gamma\) is the discount factor (where \(0\leq\gamma<1\)), which balances the significance of immediate rewards against future rewards. The term \(r(s_{t},a_{t})\) represents the reward received after taking action \(a_{t}\) in state \(s_{t}\) at time \(t\). A policy that maximizes the expected total return over all states is called an optimal policy \(\pi^{*}\), and RL algorithms aim to optimize policy \(\pi\) by maximizing its cumulative reward over time.  
In this context, the value functions are fundamental equations for modeling a policy:  
* The state value function \(V(s)\) provides an estimate of the expected cumulative reward that can be obtained starting from state \(s\) and following policy \(\pi\). It is defined as \[V^{\pi}(s)=\mathbb{E}_{s_{t},a_{t}\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}r (s_{t},a_{t})\mid s_{0}=s\right]\] (2)
* The State-Action Value Function \(Q(s,a)\) estimates the expected cumulative reward starting from state \(s\), taking action \(a\), and following policy \(\pi\). It is defined as \[Q^{\pi}(s,a)=\mathbb{E}_{s_{t},a_{t}\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^{t }r(s_{t},a_{t})\mid s_{0}=s,a_{0}=a\right]\] (3)  
Numerous algorithms are based on these equations to optimize policies in RL, enabling agents to learn effective strategies for decision-making. Among these, Deep Reinforcement Learning (DRL) [35] represents a class of algorithms that integrate deep learning techniques with traditional reinforcement learning methods. DRL utilizes neural networks to approximate value functions and policies, allowing agents to handle high-dimensional state spaces and complex environments more efficiently.  
On the other hand, Inverse Reinforcement Learning (IRL) is a framework within RL that focuses on inferring a reward function for the agent based on observed behavior, typically through expert demonstrations [6]. Then, we denote MDP\(\backslash\)R as an MDP without a reward function with the tuple \(\mathcal{M}\backslash\mathcal{R}=\{\mathcal{S},\mathcal{A},p(s,a),\rho_{0}, \pi_{E}\}\), where \(\pi_{E}\) is the expert policy. IRL algorithms aim to optimize a reward function \(r\) that aligns with the expert policy \(\pi_{E}\). However, this expert policy is not typically known directly; instead, it is inferred from demonstration trajectories provided by the expert.