## 4 Methodology
### Schedule Optimization Environment  
The second component of our framework is the simulation environment. The environment is based on the memory model trained in Section 4.2 or other baselines such as EFC and HLR. Unlike previous works [14; 17], the environment proposed in this paper contains two phases, inter-day and intra-day. In the intra-day phase, the environment simulates the learner to respond for items due today one by one, until the daily time limit \(cost_{limit}\) is reached. Successful recall and unsuccessful recall cost \(cost_{r_{t}=1}\) and \(cost_{r_{t}=0}\), respectively. For a new item (never reviewed before), the item will be arranged for review the next day and the time cost is \(cost_{new}\). When the daily cost is exhausted, the remaining items are postponed to the next day, and the date _day_ is processed to the next day \(day=day+1\).  
The detailed simulation process of our environment is shown in Algorithm 1. \(I_{t}\) represents the current item. The properties, \(due\_day\), \(last\_day\), \(type\), \(attempts\), \(intervals\), \(recalls\), and \(probabilities\) of \(I_{t}\) represent newest day to review, last day of review, item type, number of attempts, interval history, recall result history, and recall probability history of the item, respectively. \(\mathcal{M}\) indicates the memory model used in the environment. The method \(\mathcal{M}.get()\) fetches the next item from the memory. \(\mathcal{M}.prob(I_{t})\) calculates the recall probability of the current item \(I_{t}\). The method \(\mathcal{M}.put(I_{t})\) puts the item with updated states back into the memory.  
```
Data:Action \(a_{t}\), Memory \(\mathcal{M}\) Result: Reward \(r\), Observation \(o_{t}\), Termination
1 Initialize the environment parameters;
2 whilenot terminateddo
3 Get the current item due today: \(I_{t}=\mathcal{M}.get()\) with \(I_{t}\).due_day \(==\) day;
4 if\(I_{t}\).type \(==\) REVIEWthen
5 Calculate recall probability \(p_{t}=\mathcal{M}.prob(I_{t})\); /* The item type is review item */
6\(I_{t}\).attempts \(\leftarrow\)\(I_{t}\).attempts \(+\) 1;
7\(I_{t}\).intervals.append(\(a_{t}\)); /* Record interval history */
8 ifrandom() \(<\)\(p_{t}\)then
9\(r_{t}\)\(\leftarrow\) 1; /* Successful recall */
10\(cost\)\(\leftarrow\)\(cost\)\(+\)\(cost_{r=1}\);
11
12 else
13\(r_{t}\)\(\leftarrow\) 0 Unsuccessful recall \(cost\)\(\leftarrow\)\(cost\)\(+\)\(cost_{r=0}\);
14 Update interval states of Memory \(\mathcal{M}\);
15\(I_{t}\).recalls.append(\(r_{t}\)); /* Record recall history */
16\(I_{t}\).probabilities.append(\(p_{t}\)); /* Record probability history */
17
18 else
19\(cost\)\(\leftarrow\)\(cost\)\(+\)\(cost_{\text{new}}\); /* The item type is new item */
20\(I_{t}\).type \(\leftarrow\) REVIEW;
21\(I_{t}\).due_day \(\leftarrow\)\(I_{t}\).due_day \(+\) * Set the next day to review */
22\(I_{t}\).last_day \(\leftarrow\) day;
23 Put the item back to review \(\mathcal{M}.put(I_{t})\);
24\(r\)\(\leftarrow\)\(p_{t}\) ; /* Reward is equal to recall probability */
25 if\(cost\)\(\geq\)\(cost_{limit}\)then
26\(cost\)\(\leftarrow\) 0;
27\(next\).day(); /* The learning quota of today is running out */
28 ifno next item or session endsthen
29 Terminate the session; Set reward to 0;
30 Get observation and other information;
31 Close the environment;
```  
**Algorithm 1**The simulation process of the environment