# The Porcupine Measure for Comparing the Performance of Multi-Objective Optimization Algorithms
## 3 Background and Related Work
In order to generate the intersection lines, the minimum and maximum values for each objective over the non-dominated solutions were found. The objective values were then normalized according to the minimum and maximum values into the range \([0,1]\). Intersection lines were then generated as equally spread lines from the origin rotated from \((0,1)\) to \((1,0)\), effectively rotating \(90^{\circ}\), covering the complete approximated OP. For \(M\)-dimensional problems, where the number of objectives is \(M>2\), [PERSON] and [PERSON] suggested using a grid-based approach where points are spread equally on the \(M\), \((M-1)\)-dimensional hyperplanes. Each hyperplane corresponds to an objective value fixed at the value 1.0. The intersection lines are drawn from the origin to these equally distributed points. In the case of 3-dimensional problems, a \(6\times 6\) grid would result in 108 (\(3\times 6\times 6\)) points and, thus, 108 intersection lines. Similarly, using a \(16\times 16\) grid on a 3-dimensional problem would result in 768 intersection lines, and so forth. For statistical significance testing, [PERSON] and [PERSON] used the Mann-Whitney U test [9] with a significance level of \(\alpha=0.05\). Finally, [PERSON] and [PERSON] found that a convenient way to report the comparison results was to use simple value pairs \([a,b]\), hereafter referred to as the Knowles-Corne measure (KC), where \(a\) gives the percentage of space for which algorithm \(A\) was found to be statistically superior to algorithm \(B\), and \(b\) gives the percentage where algorithm \(B\) was found to be statistically superior to algorithm \(A\). It can be noted that \(100-(a+b)\) gives the percentage where neither algorithm was found to be statistically superior to the other. [PERSON] and [PERSON] [8] generalized the definition of the comparison to compare more than two algorithms. For \(K\) algorithms, the above comparison is carried out for all \(\binom{K}{2}\) algorithm pairs. For each algorithm, \(k\), two percentages are reported: \(a_{k}\), which is the region where algorithm \(k\) was not worse than any other algorithm, and \(b_{k}\), which is the region where algorithm \(k\) performed better than all the other \((K-1)\) algorithms. Note that \(a_{k}\geq b_{k}\) because the region described by \(b_{k}\) is contained in the region described by \(a_{k}\). [PERSON] and [PERSON] [10] found that visualization of attainment surfaces in three dimensions is difficult due to the intersection lines not being evenly spread. As an alternative, [PERSON] presented an algorithm inspired by the work conducted by [PERSON] et al. [11] to visually draw summary attainment surfaces using axis-aligned lines. The algorithm was found to be particularly well suited for drawing 3-dimensional attainment surfaces. [PERSON] et al. [12] continued work on attainment surfaces by introducing the empirical attainment function (EAF). The EAF is a mean-like, first-order moment measure of the solutions found by a multi-objective optimiser. The EAF allows for intuitive visual comparisons between bi-objective optimization algorithms by plotting the solution probabilities as a heat map [13]. [PERSON] et al. [14] studied the use of the second-order EAF, which allows for the pairwise relationship between random Pareto-set approximations to be studied. It should be noted that calculation of the EAF for three or more dimensions is not trivial [15]. Efficient algorithms to calculate the EAF for two and three dimensions have been proposed in [15]. [PERSON] and [PERSON] [16] developed approaches to visualize the EAFs in two and three dimensions.