# Balancing Project Schedule, Cost, and Value under Uncertainty: A Reinforcement Learning Approach
## 5 The RL Solution  
RL has demonstrated remarkable achievements in various domains, ranging from mastering backgammon at a level comparable to the world's best players [94] to successfully landing unmanned aerial vehicles (UAVs) [95], defeating top-ranked contestants in Jeopardy! [96], and achieving human-level performance in Atari games [97]. These accomplishments highlight the effectiveness of RL in dealing with uncertain environments. Inspired by this success, we apply RL to the formulations described in Section 4. While RL-based heuristics have been employed in project scheduling [98; 99], to the best of our knowledge, no previous work has addressed multimode problems with chance constraints using RL. The RL framework begins by placing an agent in a state denoted as \(S\). The agent takes an action denoted as \(A\) and transitions to state \(\mathcal{S}^{\prime}\), receiving a reward denoted as \(\mathcal{R}^{\prime}\). Subsequently, the agent performs action \(\mathcal{A}^{\prime}\), moves to state \(\mathcal{S}^{\prime\prime}\), and receives reward \(\mathcal{R}^{\prime\prime}\), and the pattern continues. Hence, the agent's life trajectory can be represented as \(\mathcal{S},\mathcal{A},\mathcal{R}^{\prime},\mathcal{S}^{\prime},\mathcal{A} ^{\prime},\mathcal{R}^{\prime\prime},\mathcal{S}^{\prime\prime},\mathcal{A} ^{\prime\prime},\mathcal{R}^{\prime\prime},\mathcal{S}^{\prime\prime}, \mathcal{A}^{\prime\prime}\), and so on. To guide the agent's behavior in each state, a policy denoted as \(\pi(\mathcal{S},\mathcal{A})\) is followed, instructing the agent which action to take. The objective of the RL problem is to learn a policy that maximizes the agent's cumulative reward. Additionally, we introduce an action-value function denoted as \(q(\mathcal{S},\mathcal{A})\), which estimates the reward for taking action \(A\) in state \(S\) and subsequently following policy \(\pi(\mathcal{S},\mathcal{A})\). By applying the RL model to the formulations outlined in Section 4, we define a state as a project activity denoted by \(j\). The agent takes action by selecting a mode \(\hat{m}_{j}\) and additionally, in CCBM and TVNPV, a start time \(\hat{t}_{j}\) for activity \(j\), and then proceeds to the next activity. After determining modes and start times for all activities \(j=1,\ldots,J\), the agent can calculate its reward \(\mathcal{R}(j,m,t)\). As the agent receives rewards, it learns the action-value function \(q(j,m,t)\) and the corresponding policy \(\pi(j,m,t)\) to be followed. In this study, we utilize Monte Carlo control (MCC), an RL method based on [100]. MCC leverages Monte Carlo simulation to estimate CDFs for the activity durations, which are used in reward calculations. The algorithms for each of the three problems consist of a main procedure in which multiple functions are called. This section presents the main procedures along with a high-level explanation. The pseudocode and detailed explanations for each function can be found in Appendix A. The LPM main procedure is shown in Algorithm 1. The iterative process comprises three key steps:  
* _Policy calculation:_ The policy is a table of the probabilities of the agent taking each action. In our case, this means selecting a mode for each activity. We employ a technique called \(\varepsilon\)-greedy policies, where we examine the action-value table and assign a probability \(\varepsilon\) of picking a random mode for an activity. Otherwise, we pick the mode with the highest action-value. * _Reward computation:_ We use the policy to select the modes and then compute the reward for this action. In LPM, we model the reward as the project value, calculated by the project-specific value function. For an example of a value function, see Section 6. * _Action-value update:_ The reward obtained by this choice of modes is used to update the action-value table. To calculate the new action-values, we either average all the rewards obtained for this specific mode choice (RL\({}_{1}\)) or use a constant-step formula (RL\({}_{2}\)).