## 4 Policy Optimization
### Noisy Optimization of Crane Dispatching Policy  
The fitness evaluation of a candidate policy \(\pi\) through simulations of operation scenarios is subject to noise, due to the lack of the depth and breadth of the scenarios. Formally, a noisy fitness function \(g(\pi)\) can be expressed as follows:  
\[g(\pi)=f(\pi)+z,\;z\sim N(0,\sigma^{2}) \tag{7}\]  
where \(\pi\) is a candidate policy, \(f(\pi)\) is a noise-free fitness function, and \(z\) is the additive noise that is often assumed to be normally distributed with zero mean and variance \(\sigma\)2. While the goal of optimization is to minimize \(f(\pi)\), only \(g(\pi)\) is observable during optimization. One practical way to approximate the real fitness is averaging the values of a sufficient number of fitness samples [28]:  
\[f(\pi)\approx\frac{1}{n}\sum_{i=1}^{n}g_{i}(\pi)=\frac{1}{n}\sum_{i=1}^{n}(f( \pi)+z_{i}) \tag{8}\]  
where \(n\) is the number of samples and \(g_{i}(\pi)\) is the \(i\)th estimate of the fitness. In a noisy environment, the fitness comparison for the survival selection should be done probabilistically, considering the influence of noise. Assuming two independent random fitness value lists, \(x\) and \(y\), the probability that \(x\) is better than \(y\), \(p(y\prec x)\), is calculated by using a Student's \(t\)-distribution [11]:  
\[p(y\prec x)=F\Big{(}\overline{d}/s_{d},v\Big{)}=\int_{-\infty}^{\overline{d}/ s_{d}}f(t,v)dt \tag{9}\]  
where the mean \(\overline{d}=\overline{x}-\overline{y}\), \(\sigma_{d}\) is the standard deviation of \(d\), and \(F\) is the cumulative distribution of the Student's \(t\)-distribution \(f(t,v)\) with \(v\) degrees of freedom. \(\sigma_{d}\) and \(v\) are calculated as follows:  
\[\sigma_{d}=\sqrt{\frac{(x_{n}-1)\sigma_{x}^{2}+(y_{n}-1)\sigma_{y}^{2}}{x_{n} +y_{n}-2}} \tag{10}\]  
\[v=n_{x}+n_{y}-2 \tag{11}\]  
where \(x_{n}\) and \(y_{n}\) are the number of samples for \(x\) and \(y\), respectively. Notice that the number of samples of the fitness of every newly born individual should be at least two for this probabilistic comparison, because otherwise, the variance becomes zero. Figure 6 shows the pseudocode of N-RTS. The algorithm begins by initializing the population \(P\) with randomly generated policies (or weight vectors). Each policy is then evaluated by having \(n_{i}\) (\(\geq\)2) samples allotted to it, where each sample is the fitness value measured by simulating a scenario randomly picked from a large pool of scenarios. The evolutionary process consists of reproduction and survival selection steps that repeat until a termination condition holds. As it is one of the overlapping-generation models, parents and offspring compete with each other for survival. In the reproduction step, \(m\) individuals are randomly (with no regard to their fitness values) selected from \(P\) to form a mating pool \(M\), from which the same number of offspring is generated by applying crossover and mutation operations. In the survival selection step that follows, each offspring \(y\) in \(M\), after being evaluated with \(n_{i}\) samples, competes with the individual \(x\) in \(P\) that is very similar to it. To find such an \(x\), \(t\) individuals are randomly selected from \(P\), and then the one most similar to \(y\) is identified. The similarity measure used is the Euclidean distance. In the original RTS, where there is no notion of noise, the one with higher fitness wins the survival selection and the loser is discarded. By discouraging similar individuals to reside together in the population, RTS avoids premature convergence by maintaining population diversity. When the environment is noisy, however, the survival selection should be done not deterministically, but probabilistically. To determine the winner between the two individuals \(x\) and \(y\), their sampled finesses are compared probabilistically by using Equation (9). We consider that \(x\) is probabilistically better than or equal to \(y\) if \(p(y\prec x)\geq 0.5\), and vice versa if \((1\,-\,p(y\prec x))\geq 0.5\).