SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Anh Viet Do
Optimisation and Logistics
The University of Adelaide, Australia
Aneta Neumann
Frank Neumann
Andrew M. Sutton
University of Minnesota Duluth, USA
We study the multi-objective minimum weight base problem, an abstraction of classical NP-hard combinatorial problems such as the multi-objective minimum spanning tree problem. We prove some important properties of the convex hull of the non-dominated front, such as its approximation quality and an upper bound on the number of extreme points. Using these properties, we give the first run-time analysis of the MOEA/D algorithm for this problem, an evolutionary algorithm that effectively optimizes by decomposing the objectives into single-objective components. We show that the MOEA/D, given an appropriate decomposition setting, finds all extreme points within expected fixed-parameter polynomial time, in the oracle model. Experiments are conducted on random bi-objective minimum spanning tree instances, and the results agree with our theoretical findings. Furthermore, compared with a previously studied evolutionary algorithm for the problem GSEMO, MOEA/D finds all extreme points much faster across all instances.
Evolutionary algorithms have been widely used to tackle multi-objective optimization problems in many areas such as robotics, pattern recognition, data mining, bioinformatics, scheduling and planning, and neural network training [35]. Their population-based search operators make them a natural choice for simultaneously handling several possibly conflicting objectives. Many generic evolutionary multi-objective frameworks have been developed to supply basic implementations for any problem, and to provide templates that can be fine-tuned for specific applications (we refer to [30] for an overview of common approaches). Such features, along with their strong empirical performances in challenging applications, have led them to becoming one of the most attractive topics to researchers and practitioners alike.
Among evolutionary multi-objective algorithms (EMOs), arguably the most exemplary are dominance-based approaches such as GSEMO and NSGA variants, with the former often being considered a baseline. Another popular technique for multi-objective optimization is to decompose the multiple objectives into a single-objective subproblem. The MOEA/D algorithm is a state-of-the-art application of this technique in evolutionary computation [29, 31].
Despite the prevalence of EMOs on practical applications, rigorous analyses of their runtime behavior on meaningful problems are scarce. Nevertheless, these kinds of analyses are critical for (1) providing performance guarantees and guidelines to practitioners who use and develop these techniques in the field, and (2) promoting the explainability of heuristic search and optimization techniques by
clarifying their working principles through a careful mathematical analysis. Run-time analyses on the performance of evolutionary algorithms have been provided for simple algorithms such as GSEMO in both artificial benchmark problems [2, 7] and others such as bi-objective minimum spanning tree [18, 28] and constrained submodular optimization [21, 22, 5, 23]. In recent years, theoretical analyses of state of the art approaches such as NSGA-II and MOEA/D have been conducted [12, 14, 13, 34, 6]. However, these run-time results have only been given for artificial benchmark problems.
In this paper, we present for the first time rigorous results on MOEA/D for a classical multi-objective optimization problem, namely the multi-objective minimum weight base problem. This problem, falling under the matroid optimization category, significantly generalizes the previously studied bi-objective minimum spanning tree problem. In this work, we focus on approximating the nondominated front, as its size can be exponential in the problem size. In particular, we show that MOEA/D obtains a factor 2-approximation for two objectives in expected polynomial time. Previous analyses for the special case of graphic matroid (i.e. spanning forests) were only able to show a pseudo-polynomial run-time for GSEMO and NSGA-II to obtain this approximation [18, 4]. We further extend the analyses by deriving a fixed-parameter polynomial expected run-time in instances with  $k \geq 2$  objectives to reach a  $k$ -approximation. That is, unlike the previous runtime bounds, ours is both polynomial and weight-free under fixed  $k$  in light of the findings by Rechel et. al. [27].
Instrumental to our analyses is a deeper understanding of the problem, and as such, we formally examine certain properties of the multi-objective minimum weight base problem. We first prove a tight approximation guarantee from computing the convex hull of the non-dominated front, extending the known guarantee for two objectives [18]. With this in mind, we explore insight regarding this convex hull, including its vertex complexity and the structural relation among solutions whose weights constitute said convex hull. In addition, we briefly formulate an efficient deterministic approach to enumerate extreme points, which achieves a smaller approximation factor in lower (and weight-free) runtime than a recently proposed framework for general multi-objective minimization instances [1]. These findings may be of interest in areas beyond runtime analysis.
2 Preliminaries & Problem
First, we give an overview of relevant matroid theory concepts, with terminologies adopted from the well-known text book [20] on the subject.
Definition 1. A tuple  $M = (E, \mathcal{I} \subseteq 2^{E})$  is a matroid if  $a) \emptyset \in \mathcal{I}, b) \forall x \subseteq y \subseteq E, y \in \mathcal{I} \Rightarrow x \in \mathcal{I}, c) \forall x, y \in \mathcal{I}, |x| < |y| \Rightarrow \exists e \in y \setminus x, x \cup \{e\} \in \mathcal{I}$ . The set  $E$  is the ground set, and  $\mathcal{I}$  is the independence collection. A base of  $M$  is a maximal set in  $\mathcal{I}$ .
Definition 2. Given a matroid  $M = (E, \mathcal{I})$ , its rank function,  $r: 2^E \to \mathbb{N}$ , is defined as  $r(x) = \max \{|y| : y \in 2^x \cap \mathcal{I}\}$ , and the rank of  $M$  is  $r(E)$ . A matroid is completely characterized by its rank function.
To give examples, a  $K$ -rank uniform matroid over  $E$  admits the independence collection  $\mathcal{I} = \{x \subseteq E : |x| \leq K\}$ , characterizing a cardinality constraint. In linear algebra, a representable matroid describes linear independence among a vector set. In graph theory, given an undirected graph  $G = (V, E)$ , a graphic matroid  $M = (E, \mathcal{I})$  defined by  $G$  is such that  $\mathcal{I}$  contains all edge sets  $x$  forming a forest subgraph in  $G$ . A base of a graphic matroid is a spanning forest, which itself is an object of much interest. Dual to the graphic matroid, the bond matroid  $M^{*} = (E, \mathcal{I}^{*})$  is such that  $\mathcal{I}^{*}$  contains all edge sets  $x$  whose removal from  $E$  preserves every pairwise connectivity in  $G$ . The matroid properties emerge in many combinatorial structures of various optimization problems [20].
A classical application of matroids in optimization is in the minimum weight base (MWB) problem. Given a weighted matroid  $(E, r, w)$ , this problem asks to find a base in this matroid minimizing  $w$ . The most arguably well-known special case of MWB problem is the minimum spanning tree (MST) problem. It is known that the classical Greedy algorithm minimizes (and maximizes) arbitrary weight over a base collection of any matroid [24, 9, 8]. From the exchange property between independent sets (specifically the symmetric-exchange property proven in [3]), we see that Greedy can also enumerate all minimum weight bases, thus characterizes the optimality of any MWB instance. A proof of this quality is also included in [8].
The multi-objective minimum weight base (MOMWB) is a natural multi-objective extension to MWB. Given a  $k$ -weighted<sup>1</sup> matroid  $(E, r, w \in (\mathbb{N}^{*})^{k \times |E|})$  where  $E$  is the ground set,  $r$  is the rank function of the matroid, and the weight vector of a solution  $x \in \{0, 1\}^{|E|}$  is  $wx$  (also called the image of  $x$  under  $w$ ), the multi-objective problem asks to find a non-dominated set of bases in  $(E, r)$  minimizing  $w$ . Here,  $k$  is the number of objectives.
We denote  $m \coloneqq |E|$ ,  $n \coloneqq r(E)$ , and observe that  $x$  is a base in  $(E, r)$  implies  $|x| = n$ . Given an objective vector function  $f$  and solutions  $x$  and  $y$ ,  $x$  dominates  $y$ , denoted with  $x \preceq_f y$ , iff  $f(y) - f(x) \in \mathbb{R}_{\geq 0}^k$ . We see that  $x \preceq_f y$  iff  $\min_{\lambda \in [0,1]^k} \lambda^\top (f(y) - f(x)) \geq 0$ , i.e.  $y$  has greater scalarized objective value than  $x$  across all linear trade-offs. We denote the set of images of non-dominated solutions with  $F$ , and vertices of its convex hull  $\operatorname{Conv}(F)$  are called extreme points. For convenience, let  $\operatorname{Conv}(F)$  contain only points in  $F$  and that its faces be conventionally defined, i.e. as continuous Euclidean subspaces.
Since  $F$  can be exponentially large, we consider instead approximating it by finding solutions mapped to  $\operatorname{Conv}(F)$ . Such a set is known to guarantee a 2-approximation of the non-dominated set for  $k = 2$  [18] under the following definition.
Definition 3 (Minimization). Given  $k$  non-negative objective functions  $f \coloneqq (f_i)_{i=1}^k$ , a solution  $x$ $c$ -approximates a solution  $y$  for some  $c \geq 0$  if  $cf(y) - f(x) \in \mathbb{R}_{\geq 0}^k$ . A solution set  $X$ $c$ -approximates (or is a  $c$ -approximation of) a solution set  $Y$  if every  $y \in Y$  is  $c$ -approximated by at least  $x \in X$ .
We formally describe categories of solutions of interest. Here, we only consider feasible solutions, e.g. bases in a MWB or MOMWB instance. Furthermore, a subset of  $E$  is characterized by a bit-string in  $\{0,1\}^{|E|}$ , so both set and bit operations on solutions are well-defined, and we use both representations throughout the paper.
Definition 4. A solution  $x$  is a supported solution to an instance with objective functions  $f = (f_i)_{i=1}^k$  and a solution set  $S$  if there is a linear trade-off  $\lambda \in [0,1]^k \setminus \{0\}$  where  $x \in \operatorname{argmin}_{y \in S} \lambda^\top f(y)$ . A trade-off set  $\Lambda$  is complete if  $\bigcup_{\lambda \in \Lambda} \operatorname{argmin}_{y \in S} \lambda^\top f(y)$  contains all supported solutions. A supported solution  $z$  is extreme if there is  $\lambda' \in [0,1]^k$  where for all  $x \in \operatorname{argmin}_{y \in S} \lambda'^\top f(y)$ ,  $f(z) = f(x)$ . A set containing a trade-off for each extreme solution is called sufficient.
Note we assume that every instance admits a solution minimizing  $\lambda^{\top}f$  for every  $\lambda \in [0,1]^k$ . Of course, this holds for MOMWB due to the solution set being finite. We see that supported solutions are precisely the solutions whose images lie on  $\mathrm{Conv}(F)$ . Intuitively, a complete trade-off set decomposes the multi-objective instance in such a way to allow enumerating all supported solutions via exactly solving scalarized instances. Since supported solutions that are not extreme are mapped to points on the faces of  $\mathrm{Conv}(F)$ , we have the following observation.
Observation 1. For each supported solution minimizing  $\lambda^{\top}f$ , there is an extreme solution minimizing  $\lambda^{\top}f$ . For every  $\lambda \in [0,1]^k$ , there is an extreme solution minimizing  $\lambda^{\top}f$ .
However, for linear functions, the number of supported solutions can be very large, so we also consider finding a representative subset which, as we will see, is sufficient to give an approximation guarantee.
Definition 5. A solution set  $X$  is sufficient to an instance with objective functions  $f$  if for every extreme solution  $y$ , there is  $x \in X$  where  $f(x) = f(y)$ . The analogy for supported solutions is called a complete solution set.
With this definition, the set of solutions that are mapped to the extreme points is sufficient. In fact, the size of a minimal sufficient set is exactly the number of extreme points. Note that while the set of all supported solutions is unique, there can be multiple distinct minimal sufficient sets due to duplicate images. We briefly prove an approximation factor by any sufficient set, which is not restricted to MOMWB.
Theorem 1 (Minimization). Given  $k \geq 1$  and a non-negative  $k$ -objective instance where for each objective  $i$  there is  $\delta_i > 0$  so that  $f_i(x) \notin (0, \delta_i]$  for all solutions  $x$ , every sufficient solution set  $P$ $k$ -approximates all solutions. This factor is tight for all  $k$ , even if  $P$  is a complete solution set.
<sup>1</sup>The integrality does not affect the algorithms' behaviors, and is used only to ease the analysis. The positivity assumption ensures that approximation factors are meaningful.
We denote the weight scalarization with trade-off  $\lambda$  with  $w^{(\lambda)}\coloneqq \lambda^{\top}w$ , so  $(E,r,w^{(\lambda)})$  is a scalarized instance at  $\lambda$ . All proofs, including the one for the above result, are included in the Appendix.
3 Properties of  $\operatorname{Conv}(F)$  in Multi-Objective Minimum Weight Base Problem
Here, we derive various properties of  $\mathrm{Conv}(F)$  with implications on the complexity of approximating  $F$ . Since these solutions are optima of linearly scalarized instances, we use the properties of the Greedy algorithm, known to guarantee and characterize optimality in linear optimization over a matroid.
The Greedy algorithm starts from an empty set and adds elements to it in increasing weight order while maintaining its independence, until a base is reached. In essence, Greedy operates on a permutation over  $E$  and produces a unique solution so we can characterize its outputs via permutations. We say a permutation  $\tau$  over  $E$ ,  $\tau : E \to \{1, \dots, m\}$ , sorts the weight  $w$  if, for all  $i = 1, \dots, m - 1$ ,  $w_{\tau^{-1}(i)} \leq w_{\tau^{-1}(i + 1)}$ . As mentioned, Greedy run on a permutation that sorts the weight to be minimized returns a minimum weight base. More importantly, all minimum weight bases can be obtained by running Greedy on all sorting permutations. This allows us to derive properties of any solution mapped onto  $\operatorname{Conv}(F)$  using Greedy's behaviors. In particular, we can circumvent the difficulty of examining weights by examining permutations instead, essentially looking at the weight-induced rankings rather than the weights themselves. As such, all results in this section are weight-free and hold for arbitrary real weights. We refer to [1] for a weight-dependent algorithmic treatment of  $\operatorname{Conv}(F)$  under general settings.
Observation 2. A MOMWB instance defined over a ground set  $E$  satisfies the following: (1) its objective functions are linear, and (2) there is a surjective mapping from the set of permutations over  $E$  to the set of supported solutions.
To simplify analysis, we restrict the trade-off space to non-negative 1-norm unit vectors  $U = \{a \in [0,1]^k : \sum_{i=1}^k a_i = 1\}$ , let  $\pi_\lambda$  be a permutation sorting  $w^{(\lambda)}$  for  $\lambda \in U$ . For each  $i \in E$ , let  $\mathbb{W}_i = (w_{j,i})_{j=1}^k$ , and for each pair  $i,j \in E$ , let  $\delta_{i,j} = \mathbb{W}_i - \mathbb{W}_j$  and  $\Delta_{i,j} = \{a \in U : \delta_{i,j}^\top a = 0\}$  be the  $(k-2)$ -dimensional set characterized by the fact that for all  $\lambda \in U$ ,  $w_i^{(\lambda)} = w_j^{(\lambda)}$  iff  $\lambda \in \Delta_{i,j}$ . Finally, let  $A$  be the multiset of non-empty  $\Delta_{i,j}$  where  $\delta_{i,j} \neq 0$ ,  $H_A$  be the multiset of convex  $(k-1)$ -polytopes in  $U$  defined by intersections of half-spaces bounded by hyperplanes in  $A$  and the boundary of  $U$ , and  $A'$  be the set of points in  $U$  where each point lies in the interior of a polytope in  $H_A$ , we show that  $A$  and  $A'$  encompass complete solution set and sufficient solution set, respectively. Note that if  $\delta_{i,j} = \mathbb{W}_i - \mathbb{W}_j = 0$ , the inclusion of either  $i$  or  $j$  in a solution does not change its image under  $w$ .<sup>2</sup>
Lemma 1. For any  $Q \in H_A$ , the set of all bases minimizing  $\mathsf{w}^{(\lambda)}$  remains constant for all  $\lambda \in \operatorname{Int}(Q)^3$ , and these bases share an image under  $\mathsf{w}$ . Furthermore, they also minimize  $\mathsf{w}^{(\lambda)}$  for all  $\lambda \in Q$ .
This immediately gives the upper bound on the number of extreme points, which is the maximum number of space partitions by hyperplanes; the formula for this is given in [32]. Note that Lemma 1 only requires properties in Observation 2, which hold for the broader class of set systems that is matroid embeddings since all minimum weight bases in such a system are exactly Greedy bases [11].
Corollary 1. The size of a minimal sufficient solution set to a  $k$ -objective instance satisfying properties in Observation 2 is at most  $\sum_{i=1}^{k} \binom{m(m-1)/2}{i-1}$ , and  $A'$  is a sufficient trade-off set.
We remark that we deliberately choose each trade-off in  $A'$  from the interior of each polytope. This is because if a zero trade-off coefficient is assigned to an objective, then bases minimizing the weight scalarized by such a trade-off may not be non-dominated. Furthermore, such scalarized weights admit optima whose images under  $w$  are identical, which is necessary to ensure that the first optimum an optimization algorithm finds using these trade-offs is an extreme solution. Moreover, this trade-off
2We include polytopes with empty interiors in  $H_{A}$  to account for overlapping hyperplanes in  $A$ . Furthermore, we assume these hyperplanes are ordered arbitrarily along their normal direction for the purpose of defining interior-free polytopes:  $h$  such hyperplanes form  $h - 1$  polytopes.
3Given a set  $A$  in a metric space,  $\operatorname {Int}(A)$  is the set of its interior points.
selection scheme also guarantees that said algorithm does not discard vertices of  $\mathrm{Conv}(F)$  over time, unless it stores all found optima.
Given a solution set  $S$ , the  $l$ -Hamming neighborhood graph of  $S$  is an undirected graph  $G_{l} = (S, \{\{a, b\} : |a \otimes b| \leq l\})$ , and  $S$  is  $l$ -Hamming connected if  $G_{l}$  is connected. Neumann [18] proved for spanning trees that given the non-dominated front being strongly convex, the set of supported solutions is 2-Hamming connected. We show that this even holds for matroid bases without the convexity assumption. For simplicity, we assume, for the rest of the analysis, fixed orderings among each class of elements  $i \in E$  sharing  $\mathbb{W}_i$ . We will see that the existence of such elements does not affect the 2-Hamming connectivity among supported solutions.
We first show that as the trade-off moves continuously within  $U$ , the permutation sorting the scalarized weight is transformed incrementally by exchanging two adjacent positions, which we call an adjacent swap.
Lemma 2. For any  $a, a' \in U$ , let  $A^* = \{a_i\}_{i=1}^h$  be the multiset of intersections between the line segment connecting  $a$  and  $a'$  and hyperplanes in  $A$ , indexed in the order from  $a$  to  $a'$ , there is a shortest sequence of adjacent swaps from  $\pi_a$  to  $\pi_{a'}$ ,  $(\pi_a, \tau_1, \ldots, \tau_h, \pi_{a'})$ , where for all  $i = 1, \ldots, h$ ,  $\tau_i$  sorts  $w^{(a_i)}$ . If  $w^{(a)}$  or  $w^{(a')}$  can be sorted by multiple permutations, the claim holds assuming that  $\pi_a$  and  $\pi_{a'}$  have maximum Kendall distance<sup>4</sup>.
Next, we show that an adjacent swap on the sorting permutation incurs an at most 2-bit change in the minimum weight base.
Lemma 3. Let  $\tau$  and  $\tau'$  be permutations over  $E$  that are one adjacent swap apart, and  $x$  and  $x'$  are Greedy solutions on them, respectively, then  $|x \otimes x'| \leq 2$ . Furthermore, let  $u, v \in E$  where  $\tau(v) = \tau(u) + 1$  and  $\tau'(v) = \tau'(u) - 1$ ,  $|x \otimes x'| = 2$  iff  $x' = x \setminus \{u\} \cup \{v\}$ .
Lemma 2 and 3 indicate that there is a sequence of 2-bit flips between any pair of supported solutions such that every step also gives a supported solution. Therefore, starting from a supported solution, we can compute the rest of  $\operatorname{Conv}(F)$  with 2-bit variations.
Regarding weight-sharing elements, for a supported solution  $x$  minimizing  $w^{(\lambda)}$ , if there is a class of equal-weight elements  $Z$  partially intersecting  $x$ , then all supported solutions minimizing  $w^{(\lambda)}$  containing different elements in  $Z$  can be reached from  $x$  by a sequence of 2-bit flips, each step in which produces a supported solution also minimizing  $w^{(\lambda)}$ . This is because  $Z$  is located consecutively in  $\pi_{\lambda}$  and can be arranged arbitrarily (leading to the Greedy solution minimizing  $w^{(\lambda)}$ ), and there is a sequence of adjacent swaps between any two such permutations touching only elements in  $Z$ . Furthermore, if there are multiple such classes whose elements share a scalarized weight at some trade-off  $\lambda$ , the relative inter-class orderings in any valid  $\pi_{\lambda}$  can be shuffled arbitrarily with an adjacent swap sequence that neither, at any step, changes any pairwise intra-class ordering, nor breaks the sorting property. For these two reasons, the set of permutations sorting all scalarized weights is 1-Kendall connected (Kendall-distance equivalence to Hamming connectivity), thus the relative intra-class orderings, and consequently the presence of multiple elements within each such class, does not affect 2-Hamming connectivity.
Corollary 2. Given solutions  $x$  and  $y$  where  $\{wx, wy\} \subseteq \mathrm{Conv}(F)$ , there is a non-empty set of solutions  $\{z_i\}_{i=1}^h$  where  $x = z_1$ ,  $y = z_h$ ,  $|z_i \otimes z_{i+1}| = 2$  for all  $i = 1, \ldots, h-1$  and  $\{wz_i\}_{i=1}^h \subseteq \mathrm{Conv}(F)$ .
Lemma 3 also lets us derive a stronger bound on the number of distinct Greedy solutions as the trade-off moves in a straight line, giving an upper bound on the number of extreme points in case  $k = 2$ .
Theorem 2. Given  $n \in (0, m)$ ,  $a, b \in U$  and  $X$  is a minimal set of extreme solutions such that for each  $\theta \in [0, 1]$ ,  $X$  contains a solution minimizing  $w^{((1 - \theta)a + \theta b)}$ ,  $|X| \leq hm - h(h + 1)/2 + 1$  where  $h := \left\lceil \sqrt{2\min\{n, m - n\} - 1} \right\rceil$ .
Corollary 3. A bi-objective MwB instance (i.e.  $k = 2$ ) admits at most  $O(m\sqrt{\min\{n,m - n\}})$  extreme points.
We remark that aside from the trivial cases  $n \in \{1, m - 1\}$ , we did not find an instance where this bound is tight. As far as we are aware, it is an open question whether this bound is optimal.
${}^{4}$  Kendall distance between two permutations equals the minimum number of adjacent swaps needed to transform one into the other [17].
Algorithm 1: Finding extreme points and a complete trade-off set (adapted from [10])
Input: Multi-weighted matroid  $(E,r,w)$  Output:  $S,\Lambda$    
1  $S,\Lambda^{\prime}\gets \emptyset$  .   
2  $\Lambda \leftarrow \{e_i\}_{i = 1}^k$    
3  $P\gets$  all permutations over  $\{1,\dots ,k\}$    
4 while  $\Lambda \backslash \Lambda^{\prime}\neq \emptyset$  do   
5 for  $\lambda \in \Lambda \backslash \Lambda '$  do   
6  $\forall p\in P,a_p\gets$  base minimizing  $w^{(\lambda)}$  priorizing weights ranked by  $p$  .   
7  $S\gets S\cup \{a_p\}_{p\in P};$    
8  $\Lambda^{\prime}\gets \Lambda^{\prime}\cup \Lambda$  .   
9  $\Lambda \gets$  non-negative normal vectors to facets of Conv(\{wx:x∈S});