## 4 Policy Optimization
### Crane Dispatching Policy
\(C_{5}\) estimates the gain obtained if the candidate job \(j\) under evaluation is done. Since a higher gain is preferable but the policy looks for a minimum score, the weight value for \(C_{5}\),when optimized, would be negative. If \(j\) is a remarshaling job, the gain is the amount of time to be saved at the time of loading (or carry-out) by having \(j\) done in advance. Let \(c\) be the container pertaining to the main job \(m(j)\), for which the remarshaling job \(j\) is prepared. Then, the remarshaling job \(j\) may involve either a relocation of \(c\) from its original location to a new location closer to its exit HP or a rehandling of a container stacked above \(c\). In either scenario, \(j\) transforms its relevant main job from \(m(j)\) to \(m^{\prime}(j)\). If the estimated amount of time needed to process a job \(k\) is denoted by \(t_{p}(k)\), the gain obtained by performing the remarshaling job \(j\) can be estimated by \(t_{p}(m(j))-t_{p}(m^{\prime}(j))\). The gain by a repositioning job can also be estimated in a similar way. However, since repositioning is usually more important (or urgent) than remarshaling and the seaside repositioning jobs are more important than the landslide repositioning jobs, these job types are given different weights, as indicated by the different subscripts shown in Table 1. The gain obtained by doing a main job or rehandling job is computed from a quite different perspective because these jobs are not optional but mandatory. Choosing not to do such a job (\(j\)), at the current moment, only results in a postponement of \(j\). Notice that postponing \(j\) is likely to lead to a delay. The larger the expected delay, the more important it is not to postpone \(j\). Therefore, the gain by doing these jobs at the current moment can be considered to be the amount of expected delay caused by not doing them right away. To estimate the delay by a postponement, we optimistically assume that the job selected, instead of job \(j\) from the candidate job set \(j\), is always the one with a minimum processing time. Figure 5 shows the pseudocode for calculating the criterion \(C_{5}\). \(C_{6}\) is designed to promote load balancing between the two ASCs. It evaluates the degree of the relative advantage of having the job done by the other ASC, instead of the current ASC. The lower the evaluation value, the better the job is done by the current ASC than the other. If the job (\(j\)) under evaluation is a main job (i.e., seaside or landslide job), the evaluation is zero because a main job cannot be processed by the other ASC. If \(j\) is a preparatory job (i.e., a rehandling or repositioning job) or a remarshaling job, the evaluation is made in the following way:  
\[t_{p}(j)\sum_{i\in M}t_{p}(i)-t^{\prime}_{p}(j)\sum_{k\in M^{\prime}}t^{ \prime}_{p}(k) \tag{4}\]  
where \(t_{p}(j)\) and \(t^{\prime}_{p}(j)\) are the times taken to process job \(j\) by the current ASC and the other ASC, respectively, and \(M\) and \(M^{\prime}\) are the main jobs remaining in / to be done by the current ASC and the other ASC, respectively. This equation can be interpreted as the difference of the weighted processing times of the two ACSs, where each weight is the remaining workload of the respective ASC. When the remaining workloads of both ASCs are the same, job \(j\) is better done by the ASC with a shorter processing time. If the processing times of both ASCs are the same, the job is better done by the ASC with a lower remaining workload. \(C_{7}\) estimates the waiting time due to interference with the other crane. The waiting time is 0 if no interference occurs when tested by simulation. If an interference occurs, the waiting time is the time taken until the other crane finishes its currently assigned job. This criterion discourages the selection of a job prone to ASC interference. \(C_{8}\) gives precedence to the main jobs by returning the evaluation value of zero for them.