## 3 Results
The criteria for choosing the instances are summarized in Section 2.6 The second test measured the number of distinct optima found as a function of run number. We formally define \"distinct\" optimal solutions as those that differ on variables in the objective function; two solutions with identical objective function variable values, but different non-objective function variable values, are considered identical for this test. The motivation for focusing on the objective function variables is that they are the variables of interest in applications such as MadHitter. If each optimum has an equal probability of being found (see Question 3 above), then the expected number of runs \(s\) needed to find all optima grows as \(slog(s)\) ([45] Section 2.4, pp. 32-33). For the second test, we executed \(r\) runs with MORSE and \(r\) runs with uniform weights and collected 1 optimum for each run. The value of \(r\) ranged from 100 to 2000, depending on the number of distinct optima for a given instance. Figure 1: Average normalized pairwise Hamming distance (\(y\)-axis) vs. run number (\(x\)-axis) for MIPLIB instance noswot. Figure 2: Average normalized pairwise Hamming distance (\(y\)-axis) vs. run number (\(x\)-axis) for MIPLIB instance p0033. Figure 4: Tabular results. Average normalized pairwise Hamming distance (\(y\)-axis) vs. run number (\(x\)-axis) for tested MIPLIB instances with the 10 highest maximum average normalized pairwise Hamming distances. The plots are arranged in descending order of maximum average normalized pairwise Hamming distance. Please note that the \(y\)-axis starts above 0 for some panels. Figure 3: Average normalized pairwise Hamming distance (\(y\)-axis) vs. run number (\(x\)-axis) for MIPLIB instance lp41. We then plotted the number of distinct optima (\(y\)-axis) found vs. run number (\(x\)-axis). This test was best suited for instances with \(<\)100 distinct optima, all of which we were able to find in a reasonable number of runs, thus allowing us to compare the performance of the two weights methods in finding distinct optima. Each test result fit one of the two following scenarios:  
1. While we cannot find all optima in the allotted \(r\) runs, we observe that we find more optima with MORSE than with the uniform weights method. (see Figures 6 and 7)
2. We find the same number of optima in the allotted \(r\) runs with both methods, but fewer runs are needed with MORSE than with the uniform weights method (see Figure 8)  
Both numbered scenarios favor MORSE, so we conclude empirically that our method performs better than the homogeneous weights method on the second test. For full MIPLIB results, see Figure 9.  
We tested the method of [4] as implemented in the CPLEX populate function on the 15 instances in Table 1 for which the exact number of distinct optima is known. We initially set the PopulateLim parameter, which controls the maximum number of solutions returned for the python interface, to the known number of optima for each instance. With that setting,  
Figure 5: Tabular results. Average normalized pairwise Hamming distance (\(y\)-axis) vs. run number (\(x\)-axis) for tested MIPLIB instances with the 10 lowest maximum average normalized pairwise Hamming distances. The plots are arranged in descending order of maximum average normalized pairwise Hamming distance. Please note that the \(y\)-axis starts above 0 for some panels. CPLEX found all optima for 12/15 instances, but only found 1 out of 2 known solutions for the instances set1 al and set1c1, as well as 1 out of 51 solutions for the instance khb05250. With PopulateLim set to higher values, CPLEX found the second solutions for set1 al and set1c but did not find more than 1 solution for khb05250 (Table 2). We conclude that this method works well on most instances if one knows the number of solutions beforehand (as is rarely the case in most practical settings), but does not work perfectly.