## 2 Materials and Methods
### Overview, Design Principles, Details
#### 2.1.7 Submodels  
In RL algorithms, in order to assess policies and find a pathway to improving them, a function is used that allocates a value to each (state, action) pair. In RL literature this function is known as Q [58; 59]. In turn, Q is used to update the policy. Different RL algorithms are distinguished in their timing and method of updating Q. We used a class of RL algorithms known as Temporal Difference (TD) learning algorithms. In TD algorithms, learning occurs at each time step. That is, with every action that is taken, its reward and its subsequent state are used to update Q and policy, so that the next action is prescribed with improved knowledge of the behavior of the system [59]. Below, we describe six different TD algorithms which we used. Each of these algorithms defined a submodel in our work. In these descriptions we assume that in state S, the algorithm's policy p prescribes action A. Taking this action results in reward R and subsequent state S'. The next action will be A'. In all cases, the present value of a future earning is calculated using a future discounting rate, \(\gamma\). Moreover, a learning rate, \(\alpha\), is applied to the correction term before updating Q. All the descriptions and formulas are from [PERSON] and [PERSON] [59]. For flowcharts of these algorithms, see the Data Availability section. SARSA  
In SARSA, the next action A\({}^{\prime}\) is identified as p(S\({}^{\prime}\)). Then, assuming that Q leads the system from pair (S, A) to (S\({}^{\prime}\), A\({}^{\prime}\)), the previous assessment of Q is corrected. This correction accounts for the reward R as well as the value of the future pair (S\({}^{\prime}\), A\({}^{\prime}\)). Future discounting rate \(\gamma\) is used to calculate the present value of that future pair. Equation (1) summarizes this description:  
\[\text{Q(S,A)}=\text{Q(S,A)}+\ \alpha\big{[}\text{R}+\ \gamma\times\text{Q (S${}^{\prime}$, A${}^{\prime}$)}-\text{Q(S,A)}\big{]} \tag{1}\]Q-Learning  
Another TD algorithm, known as Q-Learning, looks at Q after taking action A and identifying subsequent state S\({}^{\prime}\). Then, among all pairs (S\({}^{\prime}\), a) that are registered in Q for the new state S\({}^{\prime}\) and all possible actions, the algorithm selects the one with the maximum value, and uses it to correct Q. These operations are summarized in Equation (2):  
\[\text{Q}(\text{S},\text{A})=\text{Q}(\text{S},\text{A})+\ \alpha\big{[}\text{R}+\ \gamma\times\max_{\text{a}}\big{\{}\text{Q}(\text{S}^{ \prime},\text{a})\big{\}}-\text{Q}(\text{S},\text{A})\big{]} \tag{2}\]  
Another TD algorithm, known as Expected SARSA, proceeds similar to Q-Learning up to the correction of Q. In that stage, Expected SARSA considers all (S\({}^{\prime}\), a) pairs and calculates their average value. Equation (3) describes this update process:  
\[\text{Q}(\text{S},\text{A})=\text{Q}(\text{S},\text{A})+\ \alpha\big{[}\text{R}+\ \gamma \times\sum_{\text{a}}\text{P}\big{(}\text{a}|\text{S}^{\prime})\times\text{Q} \big{(}\text{S}^{\prime},\text{a}\big{)}-\text{Q}(\text{S},\text{A})\big{]} \tag{3}\]  
where the summation is performed over all actions a. Corresponding to the above three methods, there are more complicated methods that are called Double Learning algorithms, because they involve two Q tables. In each time-step, one of the Q tables is selected randomly and updated using the other one. The following formulas describe this concept. In each set, only one of the two formulas is performed in each time-step. The formulas for update of Q tables of the Double SARSA, Double Q-Learning, and Double Expected SARSA algorithms are as shown in Equation pairs (4) and (5), (6) and (7), (8) and (9), respectively.