## 4 Policy Optimization
### Crane Dispatching Policy  
Now, we describe the crane dispatching policy that uses a scoring function to evaluate each candidate job and recommends the one with the best score to the ASC. The set \(J\) of candidate jobs is obtained by taking a union of the set \(M\) of the main jobs, the set \(P\) of the preparatory jobs, and the set \(R\) of the remarshaling jobs, as suggested in [5]:  
\[J=M\cup P\cup R \tag{1}\]  
\(M\) consists of the discharging, loading, carry-in, and carry-out jobs that are requested within the current horizon. The discharging and loading requests are generated from the discharging and loading plans for the vessels at the berths. The discharging and loading jobs in \(M\) cannot be for just a single QC, but for multiple QCs. The carry-in and carry-out jobs of the current horizon are those for the ETs that have already arrived at the HPs of the block but not yet serviced until the beginning of the horizon. The jobs for the ETs that will soon arrive within the horizon are not included in \(M\), as the exact arrival times of ETs are highly unpredictable. \(P\) consists of the rehandling and repositioning jobs necessary for processing the main jobs in \(M\). Finally, \(R\) consists of some of the remarshaling jobs that are preferred to be processed in the current horizon, considering the workloads of the ASCs. The remarshaling jobs consist of the loading jobs not for the current, but the upcoming vessels, the carry-out jobs by the ETs that are expected to arrive during the service time of those upcoming vessels, and the rehandling jobs needed to process these future loading and carry-out jobs. Among all these remarshaling jobs, only a few with the highest priorities are entered into \(R\), following the idea of selective remarshaling originally proposed by [22] and later adopted by [5]. The priority assignment to a remarshaling job is made based on the estimated time taken to retrieve the corresponding container out of the block. The longer the estimated time, the higher the priority. The reason for this is that the remarshaling of such jobs will contribute more to the retrieval efficiency of the future loading operation. The retrieval time of a container is the sum of the estimated time taken to move the container from its original location to an exit HP and the estimated time taken for rehandling if there are other containers above it. Refer to [5] for further details on this priority assignment. Given a set \(J\) of candidate jobs, the crane dispatching policy \(\pi_{d}\) computes the score of each job in \(J\) and selects the one with the minimum score:  
\[\pi_{d}(J)=\operatorname*{argmin}_{j\in J}s_{d}(j) \tag{2}\]  
The score \(s_{d}(j)\) for job \(j\) is calculated by taking a weighted sum of the evaluations of \(j\), based on various criteria:  
\[s_{d}(j)=\sum_{i}w_{i,t(j)}C_{i,t(j)}(j) \tag{3}\]  
where \(w_{i,t(j)}\) and \(C_{i,t(j)}\) are the \(i\)th weight and criterion used, respectively, when the job type of \(j\) is \(t(j)\). For many criteria, the weight and the criterion remain the same, regardless of the job type. In that case, \(w_{i,t(j)}\) and \(C_{i,t(j)}\) are simply written as \(w_{i}\) and \(C_{i}\), respectively. Table 1 shows the nine criteria used by our crane dispatching policy for evaluating the candidate jobs. The two criteria \(C_{2}\) and \(C_{5}\) are distinguished from the others in that a different weight is assigned depending on the type of job under evaluation. To normalize each of these criteria, all the candidate jobs in \(J\) are evaluated by the criterion and the results are linearly scaled to values in [0, 1], in such a way that the minimum and the maximum are mapped to 0 and 1, respectively. In the following description of each criterion, however, only the values returned before normalization are explained for clarity and simplicity. \(C_{1}\) measures the empty travel distance of ASC to the pickup location of the container pertaining to the candidate job. This criterion encourages the ASC to choose the nearest job.