## 3 Proposed Methods
### PSGS and MDP Formulation
In Equation (4c), the completion set \(C_{v+1}\) of \(S_{v+1}\) is set to the union of \(C_{v}\) and the set containing jobs whose finish times are less than equal to \(t_{v+1}\). The balance resources for state \(S_{v+1}\) are calculated using Equation (4d) or (4j). For each resource, the remaining resource is the difference of the total resource and the sum of the resources currently being used by active jobs at state \(A_{v}\). Finally, an active set \(A_{v+1}\) is updated by adding the decision set to active set \(A_{v}\) whose finish times are greater than \(t_{v+1}\). In Equations (4d) and (4j), the value of \(|\mathbf{q}|=\left|\mathbf{q}_{j}\right|,\ \forall j\in\{1,..,J\}\), as this indicates the different types of resources required by an activity \(j\). If \(E_{v}\
eq\mathcal{O}\) in Equation (4a), then Equation (4e) is executed before Equation (4d) by setting \(x_{v}=x_{v}\); if \(E_{v}=\mathcal{O}\), a new \(E_{v+1}\) is determined directly after Equation (4d), and we set \(x_{v}=x_{v+1}\). The set \(x_{v}\) is determined using Equation (6). If a statement given by Equation (4f) is true, then Equations (4g)-(4k) are executed. That is, if there is no active job being processed or there exists at least one job in the eligible set \(E_{v}\) in which the resources required are less than the balance resources checked by Equation (4f), then the time \(t_{v+1}\) and completion set \(C_{v+1}\) of \(S_{v+1}\) are the copies from the previous state. Equation (4i) determines the active set \(A_{v+1}\) by including the decision set. Finally, Equation (4k) determines the completion times of the activities in the active set \(A_{v}\). We see that the state \(S_{v+1}\) depends only on state \(S_{v}\), which indicates that the process is a Markov process [56]. 5. _Benefit function_: Let \(\pi=\left\{\omega_{0}^{\pi}(S_{0}),\ldots,\omega_{\mathrm{F}}^{\pi}(S_{F})\right\}\) be the policy containing a sequence of decisions, where \(\omega_{v}^{\pi}(S_{v}):S_{i}\to x_{v}^{\pi}\). Let \(\psi(S_{v},x_{v}^{\pi},S_{v+1})\) be the change in the NPV of the project as a result of decision \(x_{v}^{\pi}\) at \(S_{v}\) that leads to state \(S_{v+1}\). We define the benefit-to-go function with the policy \(\pi\) starting from stage \(v\) with state \(S_{v}\) as follows: \[ \begin{split}& B_{v}(S_{v})=\mathbb{E}\left[\sum_{j=v}^{F}\psi(S_{v},x_{v}^ {\pi},S_{v+1})\right]\\ &=\mathbb{E}[\psi(S_{v},x_{v}^{\pi},S_{v+1})+B_{v+1}(S_{v+1})] \end{split} \] (5) The benefit-to-go function \(B_{v}(S_{v})\) is the expectation of the sum of \(\psi(\cdot)\) from states \(v\) to \(F\). \(B_{v}(S_{v})\) can then be re-written as the expectation of \(\psi(S_{v},x_{v}^{\pi},S_{v+1})\) and \(B_{v+1}(S_{v+1})\). To maximize the objective function given by Equation (3), we employ the Bellman equation [57; 58] to determine an optimal policy:  
\[x_{v}^{\pi^{*}}=\underset{x_{v}\subset\mathcal{X}(S_{v})}{\operatorname{argmax} }\mathbb{E}[\psi(S_{v},x_{v},S_{v+1})+B_{v+1}(S_{v+1})] \tag{6}\]  
Solving Equation (6) is computationally difficult due to the well-known curse of dimensionality. Given the permutation nature of our problem, we use a rollout-policy-based [22; 59] ADP to approximate \(x_{v}^{\pi^{*}}\). In this scheme, we approximate \(B_{v+1}(S_{v+1})\) using a certain heuristic \(h_{v+1}\) from a set of heuristics \(\mathcal{H}=\{h_{1},\ldots,h_{k}\}\). Because there are a large number of states and trajectories, we employ forward iteration with MC [45] to select the \(h_{v+1}\). In this study, we use two-stage look-ahead to guide the selection of the one-stage look-ahead heuristic and approximate the benefit-to-go value. That is, to select \(h_{v+1}\) at state \(S_{v+1}\), we run multiple trajectories using \(h_{i}\in\mathcal{H}^{\prime}\subset\mathcal{H}\) and keep track of heuristics \(h_{j}\in\mathcal{H}\) at \(S_{v+2}^{\prime}\). The state \(S_{v+2}^{\prime}\) is the state in each trajectory of the MC simulation (we call it the simulation stage) and is different from the actual \(S_{v+2}\).