## 5 Experiments  
### Experimental Setup  
BAG-DSM was evaluated with 43 decision models: the real-life model depicted in Figure 5, and the 42 benchmark models summarized in Table 1. The real-life model demonstrates the utility of BAG-DSM in a real-life scenario. With this model, 23 different randomly sampled alternatives were used as starting alternatives and for each alternative, and BAG-DSM was run to find a set of optimal solutions. On the other hand, the 42 benchmark models allowed for a detailed analysis of BAG-DSM's performance under a variety of circumstances. For each benchmark model, nine different randomly sampled starting alternatives (current alternatives \(\overline{c}\)) were sampled. Three of those alternatives had a final attribute value of _\"low\"_, three had a final attribute value of _\"medium\"_, and three had a final attribute value of _\"high\"_. For each current alternative, BAG-DSM generates new alternatives that require the smallest change in the current alternative to obtain a desirable outcome. The desirable outcome was also varied:  
1. starting with output attribute value \"low\", generate alternatives that would improve the final attribute value to \"medium\" (positive change),
2. starting with output attribute value \"low\", generate alternatives that would improve the final attribute value to \"high\" (positive change),
3. starting with output attribute value'medium', generate alternatives that would improve the final attribute value to 'high (positive change),
4. starting with output attribute value \"medium\", generate alternatives that would change the final attribute value to \"low\" (negative change),
5. starting with output attribute value \"high\", generate alternatives that would change the final attribute value to \"medium\" (negative change),
6. starting with output attribute value _\"high\"_, generate alternatives that would change the final attribute value to \"low\"_ (negative change). Each experiment ran for a minimum of 100 epochs, a maximum of 150 epochs, and was terminated after 50 epochs without improvement (i.e., a better alternative was not discovered compared to the current best). The experiments were performed on a server with Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50 GHz and 150 GB RAM. Each experiment was repeated 10 times to minimize the influence of the randomized steps on the experimental results. BAG-DSM and the experiments were implemented in Python and are available online ([[https://repo.jis.si/martingjoreski/bag-dsm](https://repo.jis.si/martingjoreski/bag-dsm)]([https://repo.jis.si/martingjoreski/bag-dsm](https://repo.jis.si/martingjoreski/bag-dsm)) accessed on 5 May 2022). All experiments were performed both for the proposed BAG-DSM and for the baseline method (also depicted in Figure 7), i.e., a method that iteratively evaluates random alternatives and, at each iteration, saves only the alternatives that are equally good or better than the previous best-ranked alternative. To provide a fair comparison, the random alternatives were generated by the same random generator used by BAG-DSM. Additionally, for each experimental run, the baseline method was given an equal amount of execution time as the BAG-DSM. For example, if the BAG-DSM finished an experiment in 5 min with a certain setup, the baseline method was given 5 min to perform a random search for the same experimental setup. The comparisons of performances undergo statistical analysis of differences in achieved performances by both methods. For that purpose, the Shapiro-Wilk test is used for testing the normality of performance distributions, the Kruskal test is used for testing the homoscedasticity, and the Mann-Whitney U test for testing the significance of differences. The evaluation metrics used for comparison with the baseline are:  
1. distance--as defined in Equation (3).