# Balancing Project Schedule, Cost, and Value under Uncertainty: A Reinforcement Learning Approach
## Appendix A Functions Used in the Main RL Procedures
### Cbm  
In addition to the notation presented in Table A1, the RL algorithm for CCBM incorporates the notation provided in Table A2. As in LPM, we begin by initializing the list of action-values (Algorithm A6). Each action consists of selecting an activity mode and start time. We divide the time interval from zero to the latest possible start of the activity into 10 equal segments and use them as the start times. We then use the action-value list to compute the policy (Algorithm A7). ```
def initialize_action_values \(\left(J,M_{j},LS_{j},\forall j=1,\ldots,J\right)\): for activity \(j=1,\ldots,J:\) for mode \(m=1,\ldots,M_{j}:\) for start time \(t=0,\frac{LS_{j}}{9},\frac{2 LS_{j}}{9},\ldots,LS_{j}:\) \(q(j,m,t)=\) large number return\(q(j,m,t),\forall j=1,\ldots,J,\forall m=1,\ldots,M_{j},\forall t=0,\frac{LS_{j}}{9}, \frac{2 LS_{j}}{9},\ldots,LS_{j}\)
```  
**Algorithm A7**Initialization of the action-value list. ```
def initialize_action_values \(\left(J,M_{j},LS_{j},\forall j=1,\ldots,J\right):\) for activity \(j=1,\ldots,J:\) for mode \(m=1,\ldots,M_{j}:\) for start time \(t=0,\frac{LS_{j}}{9},\frac{2 LS_{j}}{9},\ldots,LS_{j}:\) \(q(j,m,t)=\) large number return\(q(j,m,t),\forall j=1,\ldots,J,\forall m=1,\ldots,M_{j},\forall t=0,\frac{LS_{j}}{9}, \frac{2 LS_{j}}{9},\ldots,LS_{j}\)
```  
**Algorithm A8**Initialization of the action-value list. ```
def initialize_action_values \(\left(J,M_{j},LS_{j},\forall j=1,\ldots,J\right):\) for activity \(j=1,\ldots,J:\) for mode \(m=1,\ldots,M_{j}:\) for start time \(t=0,\frac{LS_{j}}{9},\frac{2 LS_{j}}{9},\ldots,LS_{j}:\) \(q(j,m,t)=\) large number return\(q(j,m,t),\forall j=1,\ldots,J,\forall m=1,\ldots,M_{j},\forall t=0,\frac{LS_{j}}{9}, \frac{2 LS_{j}}{9},\ldots,LS_{j}\)
```
```
def calculate_policy \(\left(J,M_{j},q(j,m,t),\forall j=1,\ldots,J\right)\): for activity \(j=1,\ldots,J\) : \(q^{*}=\max_{m,t}(j,m,t)\) \(x=\) number of action \(-\) values for which \(q(j,m,t)=q^{*}\) \(\pi(j,m,t)=\left\{ \begin{array}{l}\left(\frac{1}{x}\right)\left(1-\frac{ \varepsilon}{10M_{j}}\left(10M_{j}-x\right)\right),\forall m,t|q(j,m,t)=q^{*}\\ \frac{\varepsilon}{10M_{j}}\forall m,t|q(j,m,t)\
eq q^{*}\end{array} \right.\) return\(\pi(j,m,t),\forall j=1,\ldots,J,\forall m=1,\ldots,M_{j},\forall t=0,\frac{LS_{j}}{9}, \frac{2 LS_{j}}{9},\ldots,LS_{j}\)
```  
**Algorithm A7** Policy calculation. We then follow the policy to select an action for each activity (Algorithm A8), which involves choosing a mode and a start time based on the probabilities in the policy table. To ensure that the start times respect the precedence relations, we shift each activity to the right by adding the finish time of its immediate predecessor to its chosen start time. This means that if activity \(j\) has a start time of \(\hat{t}_{j}\), we move it to start at \(\hat{t}_{j}\) plus the finish time of its immediate predecessor. We use the most likely duration of each activity in its selected mode to calculate the finish times. Finally, we arrange all activities in ascending order of their adjusted start times, resulting in a precedence-feasible list of activities and their modes. ```
def choose_mode_start \(\left(\pi(j,m,t),d^{ML}_{j\#},\mathcal{P}(j),\forall j=1,\ldots,J\right)\): for activity \(j=1,\ldots,J\) :  choose \(\hat{m}_{j},\hat{t}_{j}\) according to \(\pi(j,m,t)\) if\(\mathcal{P}(j)=:\) \(\hat{t}_{j}^{*}=\hat{t}_{j}\) else: \(\hat{t}_{j}^{*}=\hat{t}_{j}+\max(\hat{t}_{i}^{*}+d^{ML}_{i\#},\forall\,i\in \mathcal{P}(j))\) return sorted \(\left(\hat{m}_{j}\middle|j\in\left\{1,\ldots,J\right\},\hat{m}_{j}\in\left\{1, \ldots M_{j}\right\},\,\text{key}=\hat{t}_{j}^{*}\right)\)
```  
**Algorithm A8** Choose activity mode and start time. We note that choosing different start times for the actions is equivalent to choosing different precedence combinations between the activities. The range of possible start times from zero to the upper bound allows for more flexibility and diversity in starting or delaying each activity, which can lead to a better and richer search space for solutions.