## 2 Materials and Methods
### Expert-Trajectory-Based Features for AL-IRL for Robotic Manipulation
#### 2.2.2 Reverse Discount in Feature Expectation  
When computing the feature expectation according to expressions (4) and (5), we use the parameter \(\gamma\) as a discount factor. Its purpose is to prioritize information from earlier states and decisions, placing less emphasis on future states. In an environment with a wide initial state distribution \(\rho\), the batch of expert trajectories starts in different initial states according to that distribution and ends in states corresponding to the goal. The same will happen with the features: experts will begin with different initial features and will end with the features corresponding to goal achievement. Assuming that the feature function is a subjective linear function over the state occupancy measure, this implies that each \(\phi(s_{0})\in\phi_{0}\) is a one-hot vector, as in the case of discretization in related works. The features of initial states and final states can be defined as  
\[\phi_{0}=\Big{\{}\phi(s)\in[0,1]^{k}\mid s\sim\rho_{0}\Big{\}}, \quad\text{with }\rho_{0}\subset S \tag{13}\] \[\phi_{T}=\Big{\{}\phi(s)\in[0,1]^{k}\mid s\sim\rho_{T}\Big{\}}, \quad\text{with }\rho_{T}\subset S \tag{14}\]  
where \(\rho_{0}\) is the initial state distribution, and \(\rho_{T}\) is the final state distribution. We can then define the feature expectation over a trajectory as  
Figure 3: An example of the centers obtained to represent the feature space based on expert trajectory information in a 2D state space. The expert trajectory is shown in light red, with \(C=2\) clustered centers highlighted in red. Random points within the \(L=2\) layers are displayed in light blue and light cyan, while the \(C\cdot K\cdot L=8\) centers of the layers are marked in blue and cyan. Figure 2: A flow diagram illustrating the process of generating centers that represent the feature space based on expert trajectories. Below each step, a 2D example illustrates the process: expert trajectory (light red), clustered centers (red), random points within layers (light blue and light cyan), and layer centers (blue and cyan). \[\mu(\pi)=\mathbb{E}_{s_{t}\sim\pi}\Bigg{[}\phi_{0}+\gamma^{T}\phi_{T}+\sum_{t=1}^{T -1}\gamma^{t}\phi(s_{t})\Bigg{]}, \tag{15}\]  
where \(\phi(s_{t})\) is the feature vector at time \(t\), and \(\gamma\) is the discount factor. The feature expectation is computed as a weighted sum over time, from initial to final states. We can split the computation of the feature expectation over a set of \(m\) trajectories as follows:  
\[\hat{\mu}_{\{\mathcal{T}^{(0)},\ldots,\mathcal{T}^{(m)}\}} =\frac{1}{m}\sum_{i=1}^{m}\Big{[}\phi(s_{0}^{(i)})+\gamma^{T} \phi(s_{T}^{(i)})+\sum_{i=1}^{T-1}\gamma^{t}\phi(s_{t}^{(i)})\Big{]} \tag{16}\] \[=\Big{[}\frac{1}{m}\sum_{i=1}^{m}\phi(s_{0}^{(i)})\Big{]}+\Big{[} \frac{1}{m}\sum_{i=1}^{m}\gamma^{T}\phi(s_{T}^{(i)})\Big{]}+\Big{[}\frac{1}{m }\sum_{i=1}^{m}\sum_{t=1}^{T-1}\gamma^{t}\phi(s_{t}^{(i)})\Big{]}\] (17) \[=\hat{\mu}_{0}+\hat{\mu}_{T}+\hat{\mu}_{H} \tag{18}\]  
Here, the estimation is split into expectations of initial features (\(\hat{\mu}_{0}\)), final features (\(\hat{\mu}_{T}\)), and intermediate features along the trajectory (\(\hat{\mu}_{H}\)). Considering the assumptions we have expressed, and denoting the sizes of these feature subsets by \(|\phi_{0}|\) and \(|\phi_{T}|\), the maximum values of the initial and final feature expectations are  
\[\max\hat{\mu}_{0} =\frac{1}{|\phi_{0}|} \tag{19}\] \[\max\hat{\mu}_{T} =\frac{\gamma^{T}}{|\phi_{T}|} \tag{20}\]  
Thus, in environments where \(|\phi_{0}|\) is large and \(|\phi_{T}|\) is small, the contributions of initial and final features to the overall expectation may be balanced, preventing the vanishing of feature expectations for late states. However, when both \(|\phi_{0}|\) and \(|\phi_{T}|\) are small, the discount factor may cause the expectation of later states to vanish compared to the initial states. For example, this situation often occurs in robotic manipulator environments.