## 2 Materials and Methods
### Background
#### 2.1.2. Apprenticeship Learning via Inverse Reinforcement Learning  
Apprenticeship Learning via IRL (AL-IRL) [9] is the foundation of this work, aimed at optimizing a reward function based on samples from expert trajectories. In the IRL framework without a reward function (MDP\(\backslash\)R), we assume that a vector of features \(\phi:\mathcal{S}\rightarrow[0,1]^{k}\) can be defined. The optimal reward function can then be expressed as \(r^{*}=w^{*}\cdot\phi\), where \(w^{*}\in\mathbb{R}^{k}\) represents the optimal weights. This formulation ensures that the reward function remains bounded between 0 and 1, requiring that the weights satisfy the constraint \(||w^{*}||_{1}\leq 1\). Next, we define the expected discounted accumulated feature vector, or feature expectation \(\mu(\pi)\), as follows:  
\[\mu(\pi)=\mathbb{E}_{s_{t}\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}\phi(s_{t })\right]\in\mathbb{R}^{k} \tag{4}\]  
Conversely, when we have access to expert demonstrations under the policy \(\pi_{E}\) through trajectories represented by state sequences \(\mathcal{T}_{E}=\{\mathcal{T}^{1},\ldots,\mathcal{T}^{m}\}\), we can estimate the expert feature expectation \(\hat{\mu}_{E}\) using the following expression:  
\[\hat{\mu}_{E}=\frac{1}{m}\sum_{i=1}^{m}\sum_{t=0}^{T}\gamma^{t}\phi_{t}^{(i)} \tag{5}\]  
In this context, given an MDP\(\backslash\)R, a feature mapping \(\phi\), and the expert feature expectation \(\mu_{E}\), AL-IRL aims to find a policy \(\hat{\pi}\) that closely matches the expert's performance with the unknown reward function \(r^{*}=w^{*}\cdot\phi\). The objective is to ensure that the feature expectations of the learned policy approximate those of the expert:  
\[||\mu(\hat{\pi})-\mu_{E}||_{2}\leq\epsilon \tag{6}\]  
Equation (6) is referred to as the apprenticeship loss. To optimize it, a method based on projections is employed, where the updated feature expectation \(\mu(\hat{\pi})\approx\mu^{(i-1)}\) is estimated using the following equations:  
\[\mu_{line}^{(i-1)}=(\mu^{(i-1)}-\overline{\mu}^{(i-2)}) \tag{7}\] \[\overline{\mu}^{(i-1)}=\overline{\mu}^{(i-2)}+\frac{(\mu_{line}^ {(i-1)})^{T}(\mu_{E}-\overline{\mu}^{(i-2)})}{(\mu_{line}^{(i-1)})^{T}(\mu_{ line}^{(i-1)})}(\mu_{line}^{(i-1)}) \tag{8}\]  
Equation (8) computes the orthogonal projection of \(\mu_{E}\) onto the line through \(\overline{\mu}^{(i-2)}\) and \(\mu^{(i-1)}\). The complete optimization procedure is outlined in Algorithm 1. ```
1:Randomly pick \(\pi^{(0)}\). Compute \(\mu^{(0)}=\mu(\pi^{(0)})\) with Equation (4). 2:Compute \(\mu_{E}\) with Equation (5). 3:Set \(i=0\). 4:Set \(\overline{\mu}^{(0)}=\mu^{(0)}\) and \(w^{(1)}=\mu_{E}-\mu^{(0)}\)
5:repeat
6:Compute optimal policy \(\pi^{(i)}\) using an RL algorithm with \(r=(w^{(i)})^{T}\phi\). 7:Compute \(\mu^{(i)}=\mu(\pi^{(i)})\) with Equation (4). 8:Set \(i=i+1\). 9:Compute \(\overline{\mu}^{(i-1)}\) with Equations (7) and (8). 10:Set \(w^{(i)}=\mu_{E}-\overline{\mu}^{(i-1)}\). 11:Set \(t^{(i)}=||w^{(i)}||_{2}\). 12:until\(t^{(i)}<\epsilon\)
```  
**Algorithm 1** Apprenticeship Learning via IRL using the projection method. When it comes to RL algorithms for computing an optimal policy (step 6 in Algorithm 1), various approaches have been proposed depending on the task and environment. Foundational works [6; 9; 24] often relied on exact value iteration met- hods [3], as the environments were fully known, allowing the focus to remain on the AP-IRL algorithm itself. Similarly, early implementations in real-world tasks [10; 11] employed value iteration, utilizing a known or estimated model of the environment. More recent works have shifted toward leveraging modern DRL methods, such as Deep Q-Learning (DQN) [36], which is well suited for discretized state and action spaces, as demonstrated in quadcopter navigation [13], or Deep Deterministic Policy Gradient (DDPG) [37], which is designed for continuous state and action spaces, as applied to swarm guiding [12].