# Balancing Project Schedule, Cost, and Value under Uncertainty: A Reinforcement Learning Approach
## Appendix A Functions Used in the Main RL Procedures  
### LPM  
Table 14 summarizes our notation for RL, in addition to the notation utilized in the quantitative models. Subsequently, we present the pseudocode and provide a comprehensive explanation of our MCC method. In Algorithm A1, we illustrate the initial phase of our approach, where we initialize the action-value table with intentionally overestimated values, employing a technique referred to as optimistic initial values [100]. The purpose of this strategy is to promote exploration in the early stages. Initially, the modes with the highest action-values are chosen, resulting in the agent receiving a reward that may be lower than anticipated. This, in turn, encourages the agent to select other modes with optimistically high action-values in subsequent iterations. ```
definitialize_action_values\(\left(J,M_{j},\forall j=1,\ldots,J\right)\): foractivity\(j=1,\ldots,J\): formode\(m=1,\ldots,M_{j}\): \(q(j,m)=\) large number return\(q(j,m),\forall j=1,\ldots,J,\forall m=1,\ldots,M_{j}\)
```  
**Algorithm A1** Initialization of the action-value list. When performing the policy calculation (Algorithm A2), we address a well-known challenge in RL and other search methodologies, namely, the tradeoff between exploration and exploitation. Opting for a purely greedy policy, where we select the mode with the highest action-value for each activity, may lead us to quickly find a solution but potentially overlook a superior solution achievable through a different combination of modes. Conversely, employing a random policy would result in pure exploration without any learning. To strike a balance, we utilize a technique called \(\varepsilon\)-greedy policies, described in Section 5. Moving forward, we use the policy to select modes and then compute the reward for this action (Algorithm A3). The function's main concept is that a feasible project plan gets the reward of the project value, which is our optimization goal; otherwise, it gets a zero reward as a penalty. The plan is feasible only if it meets or exceeds the decision-makers' requirements for finishing on time and on budget with certain probabilities. We utilize two distinct approaches for updating the action-values, as described by [100]. The first method, known as average rewards (RL\({}_{1}\)), involves calculating the action-values by taking the average of the rewards obtained each time a particular mode is selected for an activity. To prevent the accumulation of large lists and the associated increase in memory usage and runtime, we adopt an incremental approach for computing the averages. The variable \(N_{j}\) in Algorithm A4 represents the number of times that mode was selected for the activity. ```
defcalculate_policy\(\left(J,M_{j},q(j,m),\forall j=1,\ldots,J,\forall m=1,\ldots,M_{j}\right)\): foractivity\(j=1,\ldots,J\): \(q^{*}=\underset{m}{\max}q(j,m)\) \(x\)= number of modes \(m\) for which \(q(j,m)=q^{*}\) \(\pi(j,m)=\left\{ \begin{array}{l}\left(\frac{1}{x}\right)\left(1-\frac{ \varepsilon}{M_{j}}\left(M_{j}-x\right)\right)\forall q(j,m)=q^{*}\\ \\ \frac{\varepsilon}{M_{j}}\forall q(j,m)\
eq q^{*}\end{array} \right.\) return\(\pi(j,m),\forall j=1,\ldots,J,\forall m=1,\ldots,M_{j}\)
```  
**Algorithm A2** Policy calculation. ```
def calculate_reward \(\left(\hat{m}_{j},\forall j=1,\ldots,J\right):\)  simulate large number of project runs if proportion on time \(\geq\beta\)and proportion on budget \(\geq\hat{\beta}:\) return\(\mathcal{R}\left(j,\hat{m}_{j}\right)=V^{\ast}\) else: return\(\mathcal{R}\left(j,\hat{m}_{j}\right)=0\)
```  
**Algorithm A4**Action-value update using average rewards (RL\({}_{1}\)).