Title: Accelerating ERM for data-driven algorithm design using output-sensitive techniques
Venue: NeurIPS 2024
Accepted: yes
Keywords: Learning Theory

Data-driven Algorithm Design
Abstract: Data-driven algorithm design is a promising, learning-based approach for beyond worst-case analysis of algorithms with tunable parameters. An important open problem is the design of computationally efficient data-driven algorithms for combinatorial algorithm families with multiple parameters. As one fixes the problem instance and varies the parameters, the “dual” loss function typically has a piecewise-decomposable structure, i.e. is well-behaved except at certain sharp transition boundaries. Motivated by prior empirical work, we initiate the study of techniques to develop efficient ERM learning algorithms for data-driven algorithm design by enumerating the pieces of the sum dual loss functions for a collection of problem instances. The running time of our approach scales with the actual number of pieces that appear as opposed to worst case upper bounds on the number of pieces. Our approach involves two novel ingredients – an output-sensitive algorithm for enumerating polytopes induced by a set of hyperplanes using tools from computational geometry, and an execution graph which compactly represents all the states the algorithm could attain for all possible parameter values. We illustrate our techniques by giving algorithms for pricing problems, linkage-based clustering and dynamic-programming based sequence alignment.

Accelerating ERM for data-driven algorithm design using output-sensitive techniques

Abstract

Data-driven algorithm design is a promising, learning-based approach for beyond worst-case analysis of algorithms with tunable parameters. An important open problem is the design of computationally efficient data-driven algorithms for combinatorial algorithm families with multiple parameters. As one fixes the problem instance and varies the parameters, the 'dual' loss function typically has a piecewise-decomposable structure, i.e. is well-behaved except at certain sharp transition boundaries. Motivated by prior empirical work, we initiate the study of techniques to develop efficient ERM learning algorithms for data-driven algorithm design by enumerating the pieces of the sum dual loss functions for a collection of problem instances. The running time of our approach scales with the actual number of pieces that appear as opposed to worst case upper bounds on the number of pieces. Our approach involves two novel ingredients - an output-sensitive algorithm for enumerating polytopes induced by a set of hyperplanes using tools from computational geometry, and an execution graph which compactly represents all the states the algorithm could attain for all possible parameter values. We illustrate our techniques by giving algorithms for pricing problems, linkage-based clustering and dynamic-programming based sequence alignment.

1 Introduction

The data-driven algorithm design paradigm captures a widely occuring scenario of solving multiple related problem instances and allows the design and analysis of algorithms that use machine learning to learn how to solve the instances which come from the same domain [GR16, Bal20]. Typically there are large (often infinite) parameterized algorithm families to choose from, and data-driven algorithm design approaches provide techniques to select algorithm parameters that provably perform well for instances from the same domain. Data-driven algorithms have been proposed and analyzed for a variety of combinatorial problems, including clustering, computational biology and mechanism design [BDL20, BDD + 21, BPS20]. But most of the prior work has focused on sample efficiency of learning good algorithms i.e. the number of problem instances needed to learn algorithm parameters that perform well on a typical problem from the domain. A major open question for this line of research is to design computationally efficient learning algorithms [GR20, BDS21].

The parameterized family may occur naturally in well-known algorithms used in practice, or one could potentially design new families interpolating known heuristics. For the problem of aligning pairs of genomic sequences, one typically obtains the best alignment using a dynamic program with some costs or weights assigned to edits of different kinds, such as insertions, substitutions, or reduplications [Wat89]. These costs are the natural parameters for the alignment algorithm. The quality of the alignment can strongly depend on the parameters, and the best parameters vary depending on the application (e.g. aligning DNAs or RNAs), the pair of species being compared, or

the purpose of the alignment. Similarly, item prices are natural parameters in automated mechanism design [BSV18, BPS20]. On the other hand, for linkage-based clustering, one usually chooses from a set of different available heuristics, such as single or complete linkage. Using an interpolation of these heuristics to design a parameterized family and tune the parameter, one can often obtain significant improvements in the quality of clustering [BNVW17, BDL20].

A common property satisfied by a large number of interesting parameterized algorithm families is that the loss 4 as a function of the real-valued parameters for any fixed problem instance-called the 'dual class function' [BS21]-is a piecewise structured function, i.e. the parameter space can be partitioned into 'pieces' via sharp transition boundaries such that the loss function is well-behaved (e.g. constant or linear) within each piece [Bal20]. Prior work on data-driven algorithm design has largely focused on the sample complexity of the empirical risk minimization (ERM) algorithm which finds the loss-minimizing value of the parameter over a collection of problem instances drawn from some fixed unknown distribution. The ERM on a collection of problem instances can be implemented by enumerating the pieces of the sum dual class loss function. We will design algorithms for computationally efficient enumeration of these pieces, when the transition boundaries are linear, and our techniques can be used to learn good domain-specific values of these parameters given access to multiple problem instances from the problem domain. More precisely, we use techniques from computational geometry to obtain 'output-sensitive' algorithms (formalized in Appendix C), that scale with the output-size R Σ (i.e. number of pieces in the partition) of the piece enumeration problem for the sum dual class function on a collection of problem instances. Often, on commonly occurring problem instances, the number of pieces is much smaller than worst case bounds on it [BDL20, GS96] and our results imply significant gains in running time whenever R Σ is small.

Our contributions. We design novel approaches that use tools from computational geometry and lead to output-sensitive algorithms for learning good parameters by implementing the ERM (Empirical Risk Minimization) for several distinct data-driven design problems. The resulting learning algorithms scale polynomially with the number of sum dual class function pieces R Σ in the worst case (See Table 1) and are efficient for small constant d .

We present a novel output-sensitive algorithm for enumerating the cells induced by a collection of hyperplanes. Our approach applies to any problem where the loss is piecewise linear, with convex polytopic piece boundaries. We achieve output-polynomial time by removing redundant constraints in any polytope using Clarkson's algorithm (from computational geometry) [Cla94] and performing an implicit search over the graph of neighboring polytopes (formally Definition 2). Our results are useful in obtaining output-sensitive running times for several distinct data-driven algorithm design problems. Theorem 2.3 bounds the running time of the implementation of the

ERM for piecewise-structured duals with linear boundaries (Definition 1) that scales with R Σ provided we can enumerate the pieces of the dual class function for a single problem instance. It is therefore sufficient to enumerate the pieces of dual function for a single instance, for which Theorem 2.2 is useful.

We show how to learn multidimensional parameterized families of hierarchical clustering algorithms 5 . Our approach for enumerating the pieces of the dual class function on a single instance extends the execution tree based approach introduced for the much simpler single-parameter family in [BDL20] to multiple dimensions. The key idea is to track the convex polytopic subdivisions of the parameter space corresponding to a single merge step in the linkage procedure via nodes of the execution tree and apply our output-sensitive cell-enumeration algorithm at each node.

For dynamic programming (DP) based sequence alignment, our approach for enumerating pieces of the dual class function extends the execution tree to an execution directed acyclic graph (DAG). For a small fixed d , our algorithm is efficient for small constant L , the maximum number of subproblems needed to compute any single DP update. Prior work [GBN94] only provides an algorithm for computing the full partition for d = 2 , and the naive approach of comparing all pairs of alignments takes exponential time.

Our main contribution is to show that when the number of pieces ( R Σ in Table 1) is small we can improve over computational efficiency of prior work. Our techniques involves a novel application of Clarkson's algorithm (used to remove redundant inequalities from system of linear equations) to derive output-sensitive enumeration, and execution graphs which capture problem-specific structure.

Motivation from prior empirical work. For the clustering problem, we give theoretical bounds on the output-size R depending on the specific parametric family considered (e.g. Lemmas I.1, I.2). These bounds imply a strict improvement even for worst-case R , but can be much faster for typical R . Indeed, prior empirical work indicates R is much smaller than the worst case bounds in practice [BDL20], where even for d = 1 there is a dramatic speed up of over 10 15 times by being output-sensitive. For the sequence alignment problem, again the upper bounds on R depend on the nature of cost functions involved. For example, Theorem 2.2 of [GBN94] gives an upper bound on R for the 2-parameter problem with substitutions and deletions. For the TPT pricing problem we prove new theoretical bounds on R (Theorem G.3) that show worst-case improvements (cubic to quadratic) in the running time over prior work. In practice, R is usually much smaller than the worst case bounds, making our results even stronger. In prior experimental work on computational biology data (sequence alignment with d = 2 , sequence length /tildelow 100), [GS96] observe that of a possible more than 2 200 alignments, only R = 120 appear as possible optimal sequence alignments (over R 2 ) for a pair of immunoglobin sequences (a speed-up of over 58 orders of magnitude).

Preliminaries. We follow the notation of [Bal20]. For a given algorithmic problem (say clustering, or sequence alignment), let Π denote the set of problem instances of interest. We also fix a (potentially infinite) family of algorithms A , parameterized by a set P ⊆ R d . Here d is the number of real parameters, and will also be the 'fixed' parameter in the FPT (Fixed Parameter Tractability) sense (Appendix C). Let A ρ denote the algorithm in the family A parameterized by ρ ∈ P . The performance of any algorithm on any problem instance is given by a utility (or loss) function u : Π ×P → [0 , H ] , i.e. u ( x, ρ ) measures the performance on problem instance x ∈ Π of algorithm A ρ ∈ A . The utility of a fixed algorithm A ρ from the family is given by u ρ : Π → [0 , H ] , with u ρ ( x ) = u ( x, ρ ) . We are interested in the structure of the dual class of functions u x : P → [0 , H ] , with u x ( ρ ) = u ρ ( x ) , which measure the performance of all algorithms of the family for a fixed problem instance x ∈ Π . We will use I {·} to denote the 0-1 valued indicator function. For many parameterized algorithms, the dual class functions are piecewise structured in the following sense [BDD + 21].

Definition 1 (Piecewise structured with linear boundaries) . A function class H ⊆ R P that maps a domain P ⊆ R d to R is ( F , t ) -piecewise decomposable for a class F ⊆ R P of piece functions if the following holds: for every h ∈ H , there are t linear threshold functions g 1 , . . . , g t : P → { 0 , 1 } , i.e. g i ( x ) = I { a T i x + b i } and a piece function f b ∈ F for each bit vector b ∈ { 0 , 1 } t such that for all y ∈ P , h ( y ) = f b y ( y ) where b y = ( g 1 ( y ) , . . . , g t ( y )) ∈ { 0 , 1 } t .

We will refer to connected subsets of the parameter space where the dual class function is a fixed piece function f c as the (dual) pieces or regions , when the dual class function is clear from the context. Past work [BSV18] defines a similar definition for mechanism classes called ( d, t ) -delineable classes which are a special case of Definition 1, where the piece function class F consists of linear functions, and focus on sample complexity of learning, i.e. the number of problem instances from the problem distribution that are sufficient learn near-optimal parameters with high confidence over the draw of the sample. Our techniques apply for a larger class, where F is a class of convex functions. Past work [BSV18, BDL20, BDD + 21] provides sample complexity guarantees for problems that satisfy Definition 1, on the other hand we will focus on developing fast algorithms. We develop techniques that yield efficient algorithms for computing these pieces in the multiparameter setting, i.e. constant d > 1 .The running times of our algorithms are polynomial in the output size (i.e, number of pieces). For i ∈ Z + , we will use [ i ] to denote the set of positive integers { 1 , . . . , i } .

2 Output-sensitive cell enumeration and data-driven algorithm design

Suppose H is a collection of t hyperplanes in R d . Weconsider the problem of enumerating the d -faces (henceforth cells ) of the convex polyhedral regions induced by the hyperplanes. The enumerated cells will be represented as sign-patterns 6 of facet-inducing hyperplanes. We will present an approach for enumerating these cells in OFPT time (Output-sensitive Fixed Parameter Tractable) Definition 3), which involves two key ingredients: (a) locality-sensitivity, and (b) output-sensitivity. By locality sensitivity, we mean that our algorithm exploits problem-specific local structure in the neighborhood of each cell to work with a smaller candidate set of hyperplanes which can potentially constitute the cell facets. This is abstracted out as a sub-routine COMPUTELOCALLYRELEVANTSEPARATORS which we will instantiate and analyse for each individual problem. In this section we will focus more on the output-sensitivity aspect.

To provide an output-sensitive guarantee for this enumeration problem, we compute only the non-redundant hyperplanes which provide the boundary of each cell c in the partition induced by the hyperplanes. We denote the closed polytope bounding cell c by P c . A crucial ingredient for ensuring good output-sensitive runtime of our algorithm is Clarkson's algorithm for computing non-redundant constraints in a system of linear inequalities [Cla94]. A constraint is redundant in a system if removing it does not change the set of solutions. The key idea is to maintain a set I of non-redundant constraints detected so far, and solve LPs that detect the redundancy of a remaining constraint (not in I ) when added to I . If the constraint is redundant relative to I , it must also be redundant in the full system, otherwise we can add a (potentially different) non-redundant constraint to I (see Appendix D for details). The following runtime guarantee is known for the algorithm.

Theorem 2.1 (Clarkson's algorithm) . Given a list L of k half-space constraints in d dimensions, Clarkson's algorithm outputs the set I ⊆ L of non-redundant constraints in L in time O ( k · LP( d, | I | + 1)) , where LP( v, c ) is the time for solving an LP with v variables and c constraints.

Algorithm 1 uses AUGMENTEDCLARKSON, which modifies Clarkson's algorithm with some additional bookkeeping (details in Appendix D) to facilitate a search for neighboring regions in our algorithm, while retaining the same asymptotic runtime complexity. Effectively, our algorithm can be seen as a breadth-first search over an implicit underlying graph (Definition 2), where the neighbors (and some auxiliary useful information) are computed dynamically by AUGMENTEDCLARKSON.

Definition 2 (Cell adjacency graph) . Define the cell adjacency graph for a set H of hyperplanes in R d , written G H = ( V H , E H ) , as:

There is a vertex v ∈ V H for each cell in the partition ˜ C of R d induced by the hyperplanes;

̸

For v, v ′ ∈ V H , add the edge { v, v ′ } to E H if the corresponding polytopes intersect, i.e. P v ∩ P v ′ = ∅ .

This generalizes to a subdivision (Definition 6) of a polytope in R d .

This allows us to state the following guarantee about the runtime of Algorithm 1. In the notation of Table 1, we have R = | V H | and | E H | = O ( R 2 ) .

Theorem 2.2. Let H be a set of t hyperplanes in R d . Suppose that | E H | = E and | V H | = V in the cell adjacency graph G H = ( V H , E H ) of H ; then if the domain P is bounded by | P | ≤ t hyperplanes,

Algorithm 1: OUTPUTSENSITIVEPARTITIONSEARCH

Algorithm 1 computes the set V H in time ˜ O ( dE + V T CLRS + t LRS · ∑ c ∈ V H LP( d, | I c | +1)) , where LP(r , s) denotes the time to solve an LP in r variables and s constraints, I c denotes the number of facets for cell c ∈ V H , T CLRS denotes the running time of COMPUTELOCALLYRELEVANTSEPARATORS and t LRS denotes an upper bound on the number of locally relevant hyperplanes in Line 10.

We illustrate the significance of output-sensitivity and locality-sensitivity in our results with examples.

Example 1. The worst-case size of V H is O ( t d ) and standard (output-insensitive) enumeration algorithms for computing V H (e.g. [EG86, Xu20]) take O ( t d ) time even when the output size may be much smaller. For example, if H is a collection of parallel planes in R 3 , the running time of these approaches is O ( t 3 ) . Even a naive implementation of COMPUTELOCALLYRELEVANTSEPARATORS which always outputs the complete set H gives a better runtime of O ( t 2 ) . By employing a straightforward algorithm which binary searches the closest hyperplanes to x (in a pre-processed H ) as COMPUTELOCALLYRELEVANTSEPARATORS we have t LRS = O (1) and T CLRS = ˜ O (1) , and Algorithm 1 attains a running time of ˜ O ( t ) . Analogously, if H is a collection of t hyperplanes in R d with 1 ≤ k < d distinct unit normal vectors, then output-sensitivity improves the runtime from O ( t d ) to O ( t k +1 ) , and locality-sensitivity can be used to further improve it to ˜ O ( t k ) .

ERMinthe statistical learning setting. Wenowuse Algorithm 1 to compute the sample mimimum (aka ERM, Empirical Risk Minimization) for the ( F , t ) piecewise-structured dual losses with linear boundaries (Definition 1) over a problem sample S ∈ Π m , provided piece functions in F can be efficiently optimized over a polytope (typically the piece functions are constant or linear functions in our examples). Formally, we define search-ERM for a given parameterized algorithm family A with parameter space P and the dual class utility function being ( F , t ) -piecewise decomposable (Definition 1) as follows: given a set of m problem instances S ∈ Π m , compute the pieces, i.e. a partition of the parameter space into connected subsets such that the utility function is a fixed piece function in F over each subset for each of the m problem instances. The following result gives a recipe for efficiently solving the search-ERM problem provided we can efficiently compute the dual function pieces in individual problem instances, and the the number of pieces in the sum dual class function over the sample S is not too large. The key idea is to apply Algorithm 1 for each problem instance, and once again for the search-ERM problem. In the notation of Table 1, we have R Σ = | V S | ,

the number of vertices in the cell adjacency graph corresponding to the polytopic pieces in the sum utility function ∑ m i =1 u i . | E S | = O ( R Σ ) when d = 2 and O ( R 2 Σ ) for general d .

Theorem 2.3. Let ˜ C i denote the cells partitioning the polytopic parameter space P ⊂ R d corresponding to pieces of the dual class utility function u i on a single problem instance x i ∈ Π , from a collection s = { x 1 , . . . , x m } of m problem instances. Let ( V S , E S ) be the cell adjacency graph corresponding to the polytopic pieces in the sum utility function ∑ m i =1 u i . Then there is an algorithm for computing V S given the cells ˜ C i in time ˜ O (( d + m ) | E S | + mt LRS · ∑ c ∈ V S LP( d, | I c | +1)) , where I c denotes the number of facets for cell c ∈ V S , and t LRS is the number of locally relevant hyperplanes in a single instance.

An important consequence of the above result is an efficient output-sensitive algorithm for data-driven algorithm design when the dual class utility function is ( F , t ) -piecewise decomposable. In the following sections, we will instantiate the above results for various data-driven parameter selection problems where the dual class functions are piecewise-structured with linear boundaries (Definition 1). Prior work [BSV18, BDL20, BDD + 21] has shown polynomial bounds on the sample complexity of learning near-optimal parameters via the ERM algorithm for these problems in the statistical learning setting, i.e. the problem instances are drawn from a fixed unknown distribution. In other words, ERM over polynomially large sample size m is sufficient for learning good parameters. In particular, we will design and analyze running time for problem-specific algorithms for computing locally relevant hyperplanes. Given Theorem 2.3, it will be sufficient to give an algorithm for computing the pieces of the dual class function for a single problem instance.

Remark 1. Our results in this section directly imply efficient algorithms for pricing problems in mechanism design which are known to be ( F , t ) -decomposable where F is the class of linear functions (summarized in Table 1, see Appendix F for details).

3 Linkage-based clustering

Clustering data into groups of similar points is a fundamental tool in data analysis and unsupervised machine learning. A variety of clustering algorithms have been introduced and studied but it is not clear which algorithms will work best on specific tasks. Also the quality of clustering is heavily dependent on the distance metric used to compare data points. Interpolating multiple metrics and clustering heuristics can result in significantly better clustering [BDL20].

Problem setup. Let X be the data domain. A clustering instance from the domain consists of a point set S = { x 1 , . . . , x n } ⊆ X and an (unknown) target clustering C = ( C 1 , . . . , C k ) , where the sets C 1 , . . . , C k partition S into k clusters. Linkage-based clustering algorithms output a hierarchical clustering of the input data, represented by a cluster tree. We measure the agreement of a cluster tree T with the target clustering C in terms of the Hamming distance between C and the closest pruning of T that partitions it into k clusters (i.e., k disjoint subtrees that contain all the leaves of T ). More formally, the loss ℓ ( T, C ) = min P 1 ,...,P k min σ ∈ S n 1 | S | ∑ k i =1 | C i \ P σ i | , where the first minimum is over all prunings P 1 , . . . , P k of the cluster tree T into k subtrees, and the second minimum is over all permutations of the k cluster indices.

A merge function D defines the distance between a pair of clusters C i , C j ⊆ X in terms of the pairwise point distances given by a metric d . Cluster pairs with smallest values of the merge function are merged first. For example, single linkage uses the merge function D sgl ( C i , C j ; d ) = min a ∈ C i ,b ∈ C j d ( a, b ) and complete linkage uses D cmpl ( C i , C j ; d ) = max a ∈ C i ,b ∈ C j d ( a, b ) . Instead of using extreme points to measure the distance between pairs of clusters, one may also use more central points, e.g. we define median linkage as D med ( C i , C j ; d ) = median ( { d ( a, b ) | a ∈ C i , b ∈ C j } ) , where median ( · ) is the usual statistical median of an ordered set S ⊂ R 7 . Appendix H provides synthetic clustering instances where one of single, complete or median linkage leads to significantly better clustering than the other two, illustrating the need to learn an interpolated procedure. Single,

median and complete linkage are 2-point-based (Definition 4, Appendix I), i.e. the merge function D ( A,B ; d ) only depends on the distance d ( a, b ) for two points ( a, b ) ∈ A × B .

̸

Parameterized algorithm families. Let ∆ = { D 1 , . . . , D l } denote a finite family of merge functions (measure distances between clusters) and δ = { d 1 , . . . , d m } be a finite collection of distance metrics (measure distances between points). We define a parameterized family of linkage-based clustering algorithms that allows us to learn both the merge function and the distance metric. It is given by the interpolated merge function D ∆ α ( A,B ; δ ) = ∑ D i ∈ ∆ ,d j ∈ δ α i,j D i ( A,B ; d j ) , where α = { α i,j | i ∈ [ l ] , j ∈ [ m ] , α i,j ≥ 0 } . In order to ensure linear boundary functions for the dual class function, our interpolated merge function D ∆ α ( A,B ; δ ) takes all pairs of distance metrics and linkage procedures. Due to invariance under constant multiplicative factors, we can set ∑ i,j α i,j = 1 and obtain a set of parameters which allows α to be parameterized by d = lm -1 values 8 . Define the parameter space P = ▲ d = { ρ ∈ ( R ≥ 0 ) d | ∑ i ρ i ≤ 1 } ; for any ρ ∈ P we get α ( ρ ) ∈ R d +1 as α ( ρ ) i = ρ i for i ∈ [ d ] , α ( ρ ) d +1 = 1 -∑ i ∈ [ d ] ρ i . We focus on learning the optimal ρ ∈ P for a single instance ( S, Y ) . With slight abuse of notation we will sometimes use D ∆ ρ ( A,B ; δ ) to denote the interpolated merge function D ∆ α ( ρ ) ( A,B ; δ ) . As a special case we have the family D ∆ ρ ( A,B ; d 0 ) that interpolates merge functions (from set ∆ ) for different linkage procedures but the same distance metric d 0 . Another interesting family only interpolates the distance metrics, i.e. use a distance metric d ρ ( a, b ) = ∑ d j ∈ δ α ( ρ ) j d j ( a, b ) and use a fixed linkage procedure. We denote this by D 1 ρ ( A,B ; δ ) . We will extend the execution tree approach introduced by [BDL20] which computes the pieces (intervals) of single-parameter linkage-based clustering. A formal treatment of the execution tree, and how it is extended to the multi-parameter setting, is deferred to Appendix I.1. Informally, for a single parameter, the execution tree is defined as the partition tree where each node represents an interval where the first t merges are identical, and edges correspond to the subset relationship between intervals obtained by refinement from a single merge. The execution, i.e. the sequence of merges, is unique along any path of this tree. The same properties, i.e. refinement of the partition with each merge and correspondence to the algorithm's execution, continue to hold in the multidimensional case, but with convex polytopes instead of intervals (see Definition 5). Computing the children of any node of the execution tree corresponds to computing the subdivision of a convex polytope into polytopic cells where the next merge step is fixed. The children of any node of the execution tree can be computed using Algorithm 1. We compute the cells by following the neighbors, keeping track of the cluster pairs merged for the computed cells to avoid recomputation. For any single cell, we find the bounding polytope along with cluster pairs corresponding to neighboring cells by computing the tight constraints in a system of linear inequalities. Theorem 2.2 gives the runtime complexity of the proposed algorithm for computing the children of any node of the execution tree. It only remains to specify COMPUTELOCALLYRELEVANTSEPARATORS. For a given x = ρ we find the next merge candidate in time O ( dn 2 ) by computing the merge function D ∆ ρ ( A,B ; δ ) for all pairs of candidate (unmerged) clusters A,B . If ( A ∗ , B ∗ ) minimizes the merge function, the locally relevant hyperplanes are given by D ∆ ρ ( A ∗ , B ∗ ; δ ) ≤ D ∆ ρ ( A ′ , B ′ ; δ ) for ( A ′ , B ′ ) = ( A ∗ , B ∗ ) i.e. t LRS ≤ n 2 . Using Theorem 2.2, we give the following bound for the overall runtime of the algorithm (soft-O notation suppresses logarithmic terms and multiplicative constants in d , proof in Appendix I.2).

̸

Theorem 3.1. Let S be a clustering instance with | S | = n , and let R i = |P i | and R = R n . Let H t = ∣ ∣ { ( Q 1 , Q 2 ) ∈ P 2 t | Q 1 ∩ Q 2 = ∅} ∣ ∣ denote the total number of adjacencies between any two pieces of P i and H = H n . Then, the leaves of the execution tree on S can be computed in time ˜ O (∑ n i =1 ( H i + R i T M ) ( n -i +1) 2 ) , where T M is the time to compute the merge function.

In the case of single, median, and complete linkage, we may assume T M = O ( d ) by carefully maintaining a hashtable containing distances between every pair of clusters. Each merge requires overhead at most O ( n 2 t ) , n t = n -t being the number of unmerged clusters at the node at depth t , which is absorbed by the cost of solving the LP corresponding to the cell of the merge. We have the following corollary which states that our algorithm is output-linear for d = 2 .

Corollary 3.2. For d = 2 the leaves of the execution tree of any clustering instance S with | S | = n can be computed in time O ( RT M n 3 ) .

Above results yield bounds on T S , the enumeration time for dual function of the pieces in a single problem instance. Theorem 2.3 further implies bounds on the runtime of ERM (Table 1).

4 Dynamic Programming based sequence alignment

Sequence alignment is a fundamental combinatorial problem with applications to computational biology. For example, to compare two DNA, RNA or amino acid sequences the standard approach is to align two sequences to detect similar regions and compute the optimal alignment [NW70, Wat89, CB00]. However, the optimal alignment depends on the relative costs or weights used for specific substitutions, insertions/deletions, or gaps (consecutive deletions) in the sequences. Given a set of weights, the optimal alignment computation is typically a simple dynamic program. Our goal is to learn the weights, such that the alignment produced by the dynamic program has application-specific desirable properties.

̸

Problem setup. Given a pair of sequences s 1 , s 2 over some alphabet Σ of lengths m = | s 1 | and n = | s 2 | , and a 'space' character -/ ∈ Σ , a space-extension t of a sequence s over Σ is a sequence over Σ ∪ {-} such that removing all occurrences of -in t gives s . A global alignment (or simply alignment) of s 1 , s 2 is a pair of sequences t 1 , t 2 such that | t 1 | = | t 2 | , t 1 , t 2 are spaceextensions of s 1 , s 2 respectively, and for no 1 ≤ i ≤ | t 1 | we have t 1 [ i ] = t 2 [ i ] = -. Let s [ i ] denote the i -th character of a sequence s and s [: i ] denote the first i characters of sequence s . For 1 ≤ i ≤ | t 1 | , if t 1 [ i ] = t 2 [ i ] we call it a match . If t 1 [ i ] = t 2 [ i ] , and one of t 1 [ i ] or t 2 [ i ] is the character -we call it a space , else it is a mismatch . A sequence of -characters (in t 1 or t 2 ) is called a gap . Matches, mismatches, gaps and spaces are commonly used features of an alignment, i.e. functions that map sequences and their alignments ( s 1 , s 2 , t 1 , t 2 ) to Z ≥ 0 (for example, the number of spaces). A common measure of cost of an alignment is some linear combination of features. For example if there are d features given by l k ( · ) , k ∈ [ d ] , the cost may be given by c ( s 1 , s 2 , t 1 , t 2 , ρ ) = ∑ d k =1 ρ k l k ( s 1 , s 2 , t 1 , t 2 ) where ρ = ( ρ 1 , . . . , ρ d ) are the parameters that govern the relative weight of the features [KK06]. Let τ ( s, s ′ , ρ ) = argmin t 1 ,t 2 c ( s, s ′ , t 1 , t 2 , ρ ) and C ( s, s ′ , ρ ) = min t 1 ,t 2 c ( s, s ′ , t 1 , t 2 , ρ ) denote the optimal alignment and its cost respectively.

̸

A general DP update rule. For a fixed ρ , suppose the sequence alignment problem can be solved, i.e. we can find the alignment with the smallest cost, using a dynamic program A ρ with linear parameter dependence (described below). Our main application will be to the family of dynamic programs A ρ which compute the optimal alignment τ ( s 1 , s 2 , ρ ) given any pair of sequences ( s 1 , s 2 ) ∈ Σ m × Σ n = Π for any ρ ∈ R d , but we will proceed to provide a more general abstraction. See Section J.1 for example DPs using well-known features in computational biology, expressed using the abstraction below. For any problem ( s 1 , s 2 ) ∈ Π , the dynamic program A ρ ( ρ ∈ P ⊆ R d , the set of parameters) solves a set π ( s 1 , s 2 ) = { P i | i ∈ [ k ] , P k = ( s 1 , s 2 ) } of k subproblems (typically, π ( s 1 , s 2 ) ⊆ Π s 1 ,s 2 = { ( s 1 [: i ′ ] , s 2 [: j ′ ]) | i ′ ∈ { 0 , . . . , m } , j ′ ∈ { 0 , . . . , n }} ⊆ Π ) in some fixed order P 1 , . . . , P k = ( s 1 , s 2 ) . Crucially, the subproblems sequence P 1 , . . . , P k do not depend on ρ 9 . In particular, a problem P j can be efficiently solved given optimal alignments and their costs ( τ i ( ρ ) , C i ( ρ )) for problems P i for each i ∈ [ j -1] . Some initial problems in the sequence P 1 , . . . , P k of subproblems are base case subproblems where the optimal alignment and its cost can be directly computed without referring to a previous subproblem. To solve a (non base case) subproblem P j , we consider V alternative cases q : Π → [ V ] , i.e. P j belongs to exactly one of the V cases (e.g. if P j = ( s 1 [: i ′ ] , s 2 [: j ′ ]) , we could have two cases corresponding to s 1 [ i ′ ] = s 2 [ j ′ ] and s 1 [ i ′ ] = s 2 [ j ′ ] ). Typically, V will be a small constant. For any case v = q ( P j ) ∈ [ V ] that P j may belong to, the cost of the optimal alignment of P j is given by a minimum over L v terms of the form c v,l ( ρ, P j ) = ρ · w v,l + σ v,l ( ρ, P j ) , where l ∈ [ L v ] , w v,l ∈ R d , σ v,l ( ρ, P j ) = C t ( ρ ) ∈ { C 1 ( ρ ) , . . . , C j -1 ( ρ ) } is the cost of some previously solved subproblem P t = ( s 1 [: i ′ t ] , s 2 [: j ′ t ]) = ( s 1 [: i ′ v,l,j ] , s 2 [: j ′ v,l,j ]) (i.e. t depends on v, l, j but not on ρ ), and c v,l ( ρ, P j ) is the cost of alignment τ v,l ( ρ, P j ) = T v,l ( τ t ( ρ )) which extends the optimal alignment for subproblem P t by a ρ -independent transformation T v,l ( · ) . That is, the DP update for computing the cost of the optimal alignment takes the form

and the optimal alignment is given by DP ′ ( ρ, P j ) = τ q ( P j ) ,l ∗ ( ρ, P j ) , where l ∗ = argmin l { ρ · w q ( P j ) ,l + σ q ( P j ) ,l ( ρ, P j ) } . The DP specification is completed by including base cases { C ( s, s ′ , ρ ) = ρ · w s,s ′ | ( s, s ′ ) ∈ B ( s 1 , s 2 ) } (or { τ ( s, s ′ , ρ ) = τ s,s ′ | ( s, s ′ ) ∈ B ( s 1 , s 2 ) } for the

optimal alignment DP) corresponding to a set of base case subproblems B ( s 1 , s 2 ) ⊆ Π s 1 ,s 2 . Let L = max v ∈ [ V ] L v denote the maximum number of subproblems needed to compute a single DP update in any of the cases. L is often small, typically 2 or 3 (see examples in Section J.1). Our main result is to provide an algorithm for computing the polytopic pieces of the dual class functions efficiently for small constants d and L .

As indicated above, we consider the family of dynamic programs A ρ which compute the optimal alignment τ ( s 1 , s 2 , ρ ) given any pair of sequences ( s 1 , s 2 ) ∈ Σ m × Σ n = Π for any ρ ∈ R d . For any alignment ( t 1 , t 2 ) , the algorithm has a fixed real-valued utility (different from the cost function above) which captures the quality of the alignment, i.e. the utility function u (( s 1 , s 2 ) , ρ ) only depends on the alignment τ ( s 1 , s 2 , ρ ) . The dual class function is piecewise constant with convex polytopic pieces (Lemma J.5 in Appendix J.3). For any fixed problem ( s 1 , s 2 ) , the space of parameters ρ can be partitioned into R convex polytopic regions where the optimal alignment is fixed. The optimal parameter can then be found by simply comparing the costs of the alignments in each of these pieces. For the rest of this section we consider the algorithmic problem of computing these R pieces efficiently.

For the clustering algorithm family, as we have seen in Section 3, we get a refinement of the parameter space with each new step (merge) performed by the algorithm. This does not hold for the sequence alignment problem. Instead we obtain the following DAG, from which the desired pieces can be obtained by looking at nodes with no out-edges (call these terminal nodes). Intuitively, the DAG is built by iteratively adding nodes corresponding to subproblems P 1 , . . . , P k and adding edges directed towards P j from all subproblems that appear in the DP update for it. That is, for base case subproblems, we have singleton nodes with no incoming edges. Using the recurrence relation (1), we note that the optimal alignment for the pair of sequences ( s 1 [: i ] , s 2 [: j ]) can be obtained from the optimal alignments for subproblems { ( s 1 [: i v ′ ,l ] , s 2 [: j v ′ ,l ]) } l ∈ [ L v ′ ] where v ′ = q ( s 1 [: i ] , s 2 [: j ]) . The DAG for ( s 1 [: i ] , s 2 [: j ]) is therefore simply obtained by using the DAGs G v ′ ,l for the subproblems and adding directed edges from the terminal nodes of G v ′ ,l to new nodes v p,i,j corresponding to each piece p of the partition P [ i ][ j ] of P given by the set of pieces of u ( s 1 [: i ] ,s 2 [: j ]) ( ρ ) . A more compact representation of the execution graph would have only a single node v i,j for each subproblem ( s 1 [: i ] , s 2 [: j ]) (the node stores the corresponding partition P [ i ][ j ] ) and edges directed towards v i,j from nodes of subproblems used to solve ( s 1 [: i ] , s 2 [: j ]) . Note that the graph depends on the problem instance ( s 1 , s 2 ) as the relevant DP cases v ′ = q ( s 1 [: i ] , s 2 [: j ]) depend on the sequences s 1 , s 2 . A naive way to encode the execution graph would be an exponentially large tree corresponding to the recursion tree of the recurrence relation (1).

Execution DAG. Formally we define a compact execution graph G e = ( V e , E e ) as follows. For the base cases, we have nodes labeled by ( s, s ′ ) ∈ B ( s 1 , s 2 ) storing the base case solutions ( w s,s ′ , τ s,s ′ ) over the unpartitioned parameter space P = R d . For i, j > 0 , we have a node v i,j labeled by ( s 1 [: i ] , s 2 [: j ]) and the corresponding partition P [ i ][ j ] of the parameter space, with incoming edges from nodes of the relevant subproblems { ( s 1 [: i v ′ ,l ] , s 2 [: j v ′ ,l ]) } l ∈ [ L v ′ ] where v ′ = q ( s 1 [: i ] , s 2 [: j ]) . This graph is a DAG since every directed edge is from some node v i,j to a node v i ′ ,j ′ with i ′ + j ′ > i + j in typical sequence alignment dynamic programs (Appendix J.1). Algorithm 5 (Appendix J.2)) gives a procedure to compute the partition of the parameter space for any given problem instance ( s 1 , s 2 ) using the compact execution DAG. We give intuitive overviews of the three main routines in Algorithm 5.

COMPUTEOVERLAYDP computes an overlay P i of the input polytopic subdivisions {P s | s ∈ S i } and uses Clarkson's algorithm for intersecting polytopes with output-sensitive efficiency. We show that the overlay can be computed by solving at most ˜ R L linear programs (Algorithm 6, Lemma J.1).

COMPUTESUBDIVISIONDP applies Algorithm 1, in each piece of the overlay we need to find the polytopic subdivision induced by O ( L 2 ) hyperplanes (the set of hyperplanes depends on the piece). This works because all relevant subproblems have the same solution within any piece of the overlay.

Finally RESOLVEDEGENERACIESDP merges pieces where the optimal alignment is identical using a simple search over the resulting subdivision.

For our implementation of the subroutines, we have the following guarantee for Algorithm 5.

Theorem 4.1. Let R i,j denote the number of pieces in P [ i ][ j ] , and ˜ R = max i ≤ m,j ≤ n R i,j . If the time complexity for computing the optimal alignment is O ( T DP ) , then Algorithm 5 can be used to compute the pieces for the dual class function for any problem instance ( s 1 , s 2 ) , in time O ( d ! L 4 d ˜ R 2 L +1 T DP ) .

For the special case of d = 2 , we show that (Theorem J.6, Appendix J.3) the pieces may be computed in O ( RT DP ) time using the ray search technique of [Meg78].

5 Acknowledgments

We thank Dan DeBlasio for useful discussions on the computational biology literature. We also thank Avrim Blum and Mikhail Khodak for helpful feedback. This material is based on work supported by the National Science Foundation under grants CCF-1910321, IIS-1901403, and SES-1919453; the Defense Advanced Research Projects Agency under cooperative agreement HR00112020003; a Simons Investigator Award; an AWS Machine Learning Research Award; an Amazon Research Award; a Bloomberg Research Grant; a Microsoft Research Faculty Fellowship.

References

[AF96] David Avis and Komei Fukuda. Reverse search for enumeration. Discrete applied mathematics , 65(1-3):21-46, 1996.

[Bal20] Maria-Florina Balcan. Data-Driven Algorithm Design. In Tim Roughgarden, editor, Beyond Worst Case Analysis of Algorithms . Cambridge University Press, 2020.

[BB24] Maria-Florina Balcan and Hedyeh Beyhaghi. Learning revenue maximizing menus of lotteries and two-part tariffs. Transactions on Machine Learning Research (TMLR) , 2024.

[BBSZ23] Maria-Florina Balcan, Avrim Blum, Dravyansh Sharma, and Hongyang Zhang. An analysis of robustness of non-lipschitz networks. Journal of Machine Learning Research (JMLR) , 24(98):1-43, 2023.

[BDD + 21] Maria-Florina Balcan, Dan DeBlasio, Travis Dick, Carl Kingsford, Tuomas Sandholm, and Ellen Vitercik. How much data is sufficient to learn high-performing algorithms? Generalization guarantees for data-driven algorithm design. In Symposium on Theory of Computing (STOC) , pages 919-932, 2021.

[BDL20] Maria-Florina Balcan, Travis Dick, and Manuel Lang. Learning to link. In International Conference on Learning Representations (ICLR) , 2020.

[BDS20] Maria-Florina Balcan, Travis Dick, and Dravyansh Sharma. Learning piecewise Lipschitz functions in changing environments. In International Conference on Artificial Intelligence and Statistics , pages 3567-3577. PMLR, 2020.

[BDS21] Avrim Blum, Chen Dan, and Saeed Seddighin. Learning complexity of simulated annealing. In International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 1540-1548. PMLR, 2021.

[BDSV18] Maria-Florina Balcan, Travis Dick, Tuomas Sandholm, and Ellen Vitercik. Learning to branch. In International Conference on Machine Learning (ICML) , pages 344-353. PMLR, 2018.

[BDV18] Maria-Florina Balcan, Travis Dick, and Ellen Vitercik. Dispersion for data-driven algorithm design, online learning, and private optimization. In 2018 IEEE 59th Annual Symposium on Foundations of Computer Science (FOCS) , pages 603-614. IEEE, 2018.

[BFM97] David Bremner, Komei Fukuda, and Ambros Marzetta. Primal-dual methods for vertex and facet enumeration (preliminary version). In Symposium on Computational Geometry (SoCG) , pages 49-56, 1997.

[BIW22] Peter Bartlett, Piotr Indyk, and Tal Wagner. Generalization bounds for data-driven numerical linear algebra. In Conference on Learning Theory (COLT) , pages 2013-2040. PMLR, 2022.

[BKST22] Maria-Florina Balcan, Mikhail Khodak, Dravyansh Sharma, and Ameet Talwalkar. Provably tuning the ElasticNet across instances. Advances in Neural Information Processing Systems , 35:27769-27782, 2022.

[BNVW17] Maria-Florina Balcan, Vaishnavh Nagarajan, Ellen Vitercik, and Colin White. Learningtheoretic foundations of algorithm configuration for combinatorial partitioning problems. In Conference on Learning Theory (COLT) , pages 213-274. PMLR, 2017.

[BPS20] Maria-Florina Balcan, Siddharth Prasad, and Tuomas Sandholm. Efficient algorithms for learning revenue-maximizing two-part tariffs. In International Joint Conferences on Artificial Intelligence (IJCAI) , pages 332-338, 2020.

[BS21] Maria-Florina Balcan and Dravyansh Sharma. Data driven semi-supervised learning. Advances in Neural Information Processing Systems , 34, 2021.

[BSV16] Maria-Florina Balcan, Tuomas Sandholm, and Ellen Vitercik. Sample complexity of automated mechanism design. Advances in Neural Information Processing Systems , 29, 2016.

[BSV18] Maria-Florina Balcan, Tuomas Sandholm, and Ellen Vitercik. A general theory of sample complexity for multi-item profit maximization. In Economics and Computation (EC) , pages 173-174, 2018.

[Buc43] Robert Creighton Buck. Partition of space. The American Mathematical Monthly , 50(9):541-544, 1943.

[CAK17] Vincent Cohen-Addad and Varun Kanade. Online optimization of smoothed piecewise constant functions. In Artificial Intelligence and Statistics , pages 412-420. PMLR, 2017.

[CB00] Peter Clote and Rolf Backofen. Computational Molecular Biology: An Introduction . John Wiley Chichester; New York, 2000.

[CE92] Bernard Chazelle and Herbert Edelsbrunner. An optimal algorithm for intersecting line segments in the plane. Journal of the ACM (JACM) , 39(1):1-54, 1992.

[Cha96] Timothy M Chan. Optimal output-sensitive convex hull algorithms in two and three dimensions. Discrete & Computational Geometry , 16(4):361-368, 1996.

[Cha18] Timothy M Chan. Improved deterministic algorithms for linear programming in low dimensions. ACM Transactions on Algorithms (TALG) , 14(3):1-10, 2018.

[Cla94] Kenneth L Clarkson. More output-sensitive geometric algorithms. In Symposium on Foundations of Computer Science (FOCS) , pages 695-702. IEEE, 1994.

[DF12] Rodney G Downey and Michael Ralph Fellows. Parameterized complexity . Springer Science & Business Media, 2012.

[EG86] Herbert Edelsbrunner and Leonidas J Guibas. Topologically sweeping an arrangement. In Symposium on Theory of Computing (STOC) , pages 389-403, 1986.

[Fer02] Henning Fernau. On parameterized enumeration. In International Computing and Combinatorics Conference , pages 564-573. Springer, 2002.

[FGS + 19] Henning Fernau, Petr Golovach, Marie-France Sagot, et al. Algorithmic enumeration: Output-sensitive, input-sensitive, parameterized, approximative (Dagstuhl Seminar 18421). In Dagstuhl Reports , volume 8. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik, 2019.

[GBN94] Dan Gusfield, Krishnan Balasubramanian, and Dalit Naor. Parametric optimization of sequence alignment. Algorithmica , 12(4):312-326, 1994.

[GR16] Rishi Gupta and Tim Roughgarden. A PAC approach to application-specific algorithm selection. In Innovations in Theoretical Computer Science Conference (ITCS) , 2016.

[GR20] Rishi Gupta and Tim Roughgarden. Data-driven algorithm design. Communications of the ACM , 63(6):87-94, 2020.

[GS96] Dan Gusfield and Paul Stelling. Parametric and inverse-parametric sequence alignment with XPARAL. Methods in Enzymology , 266:481-494, 1996.

[KK06] John Kececioglu and Eagu Kim. Simple and fast inverse alignment. In Annual International Conference on Research in Computational Molecular Biology , pages 441-455. Springer, 2006.

[KKW10] John Kececioglu, Eagu Kim, and Travis Wheeler. Aligning protein sequences with predicted secondary structure. Journal of Computational Biology , 17(3):561-580, 2010.

[KPT + 21] Krishnan Kumaran, Dimitri J Papageorgiou, Martin Takac, Laurens Lueg, and Nicolas V Sahinidis. Active metric learning for supervised classification. Computers & Chemical Engineering , 144:107132, 2021.

[Lew41] WArthur Lewis. The two-part tariff. Economica , 8(31):249-270, 1941.

[Meg78] Nimrod Megiddo. Combinatorial optimization with rational objective functions. In Symposium on Theory of Computing (STOC) , pages 1-12, 1978.

[MR15] Jamie H Morgenstern and Tim Roughgarden. On the pseudo-dimension of nearly optimal auctions. Advances in Neural Information Processing Systems , 28, 2015.

[Nak04] Shin-ichi Nakano. Efficient generation of triconnected plane triangulations. Computational Geometry , 27(2):109-122, 2004.

[NW70] Saul B Needleman and Christian D Wunsch. A general method applicable to the search for similarities in the amino acid sequence of two proteins. Journal of Molecular Biology , 48(3):443-453, 1970.

[Oi71] Walter Y Oi. A Disneyland dilemma: Two-part tariffs for a Mickey Mouse monopoly. The Quarterly Journal of Economics , 85(1):77-96, 1971.

[RC18] Miroslav Rada and Michal Cerny. A new algorithm for enumeration of cells of hyperplane arrangements and a comparison with Avis and Fukuda's reverse search. SIAM Journal on Discrete Mathematics , 32(1):455-473, 2018.

[SC11] Shiliang Sun and Qiaona Chen. Hierarchical distance metric learning for large margin nearest neighbor classification. International Journal of Pattern Recognition and Artificial Intelligence (IJPRAI) , 25:1073-1087, 11 2011.

[Sei91] Raimund Seidel. Small-dimensional linear programming and convex hulls made easy. Discrete & Computational Geometry , 6:423-434, 1991.

[SJ23] Dravyansh Sharma and Maxwell Jones. Efficiently learning the graph for semisupervised learning. The Conference on Uncertainty in Artificial Intelligence (UAI) , 2023.

[Sle99] Nora H Sleumer. Output-sensitive cell enumeration in hyperplane arrangements. Nordic Journal of Computing , 6(2):137-147, 1999.

[Syr17] Vasilis Syrgkanis. A sample complexity measure with applications to learning optimal auctions. Advances in Neural Information Processing Systems , 30, 2017.

[Wat89] Michael S Waterman. Mathematical methods for DNA sequences . Boca Raton, FL (USA); CRC Press Inc., 1989.

[WS09] Kilian Q Weinberger and Lawrence K Saul. Distance metric learning for large margin nearest neighbor classification. Journal of Machine Learning Research (JMLR) , 10(2), 2009.

[Xu20] Haifeng Xu. On the tractability of public persuasion with no externalities. In Symposium on Discrete Algorithms (SODA) , pages 2708-2727. SIAM, 2020.

A Additional insights and challenges

We first present an algorithm for enumerating the cells induced by a finite collection of hyperplanes in R d in output-sensitive time. Our approach is based on an output-sensitive algorithm for determining the non-redundant constraints in a linear system due to Clarkson [Cla94]. At any point, we compute the locally relevant hyperplanes that constitute the candidate hyperplanes which could bound the cell (or piece) containing that point, using a problem-specific sub-routine, and apply Clarkson's algorithm over these to determine the facet-inducing hyperplanes for the cell as well as a collection of points in neighboring cells. This allows us to compute all the induced cells by traversing an implicit cell adjacency graph (Definition 2) in a breadth-first order. Our approach gives a recipe for computing the pieces of piecewise structured dual class functions with linear boundaries, which can be used to find the best parameters over a single problem instance. We further show how to compute the pieces of the sum dual class function for a collection of problem instances in output-sensitive time, by applying our approach to the collection of facet-inducing hyperplanes from each problem instance. Our approach is useful for several mechanism design problems which are known to be piecewise structured [BSV18], and we instantiate our results for two-part tariff pricing and item pricing with anonymous prices. For the single menu two-part tariff case ( d = 2 ), we use additional structure of the dual class function to give a further improvement in the running time.

For linkage-based clustering, we extend the execution tree based approach of [BDL20] for singleparameter families to d > 1 . The key idea is that linkage-based clustering algorithms involve a sequence of steps (called merges ), and the algorithmic decision at any step can only refine the partition of the parameter space corresponding to a fixed sequence of steps so far. Thus, the pieces can be viewed as leaves of a tree whose nodes at depth t correspond to a partition of the parameter space such that the first t steps of the algorithm are identical in each piece of the partition. While in the single parameter family the pieces are intervals, we approach the significantly harder challenge of efficiently computing convex polytopic subdivisions (formally Definition 6). We essentially compute the execution tree top-down for any given problem instance, and use our hyperplane cell enumeration to compute the children nodes for any node of the tree.

We further extend the execution tree approach to a problem where all the algorithmic states corresponding to different parameter values cannot be concisely represented via a tree. For dynamicprogramming based sequence alignment, we define a compact representation of the execution graph and use primitives from computational geometry, in particular for overlaying two convex subdivisions in output-polynomial time. A key challenge in this problem is that the execution graph is no longer a tree but a DAG, and we need to design an efficient representation for this DAG. We cannot directly apply Clarkson's algorithm since the number of possible alignments of two strings (and therefore the number of candidate hyperplanes) is exponential in the size of the strings. Instead, we carefully combine the polytopes of subproblems in accordance with the dynamic program update for solving the sequence alignment (for a fixed value of parameters), and the number of candidate hyperplanes is output-sensitive inside regions where all the relevant subproblems have fixed optimal alignments.

B Additional related work

Data-driven algorithm design. The framework for data-driven design was introduced by [GR16] and is surveyed in [Bal20]. It allows the design of algorithms with powerful formal guarantees when used repeatedly on inputs consisting of related problem instances, as opposed to designing and analyzing for worst-case instances. Data-driven algorithms have been proposed and analyzed for a wide variety of combinatorial problems, including clustering [BDL20], computational biology [BDD + 21], mechanism design [BSV18], mixed integer linear programming [BDSV18], semi-supervised learning [BS21, SJ23], linear regression [BKST22], low-rank approximation [BIW22] and adversarially robust learning [BBSZ23]. Typically these are NP-hard problems with a variety of known heuristics. Data-driven design provides techniques to select the heuristic (from a collection or parameterized family) which is best suited for data coming from some domain. Tools for data-driven design have also been studied in the online learning setting where problems arrive in an online sequence [CAK17, BDV18, BDS20]. In this work, we focus on the statistical learning setting where the 'related' problems are drawn from a fixed but unknown distribution.

Linkage-based clustering. [BNVW17] introduce several single parameter families to interpolate well-known linkage procedures for linkage-based clustering, and study the sample complexity of

learning the parameter. [BDL20] consider families with multiple parameters, and also consider the interpolation of different distance metrics. However the proposed execution-tree based algorithm for computing the constant-performance regions in the parameter space only applies to single-parameter families, and therefore the results can be used to interpolate linkage procedures or distance metrics, but not both. Our work removes this limitation and provides algorithms for general multi-parameter families.

Other approaches to learning a distance metric to improve the quality of clustering algorithms have been explored. For instance, [WS09] and [KPT + 21] consider versions of global distance metric learning, a technique to learn a linear transformation of the input space before applying a known distance metric. [SC11] also consider a hierarchical distance metric learning, which allows the learned distance function to depend on already-merged clusters. For such distance learning paradigms, the underlying objective function is often continuous, admitting gradient descent techniques. However, in our setting, the objective function is neither convex nor continuous in ρ ; instead, it is piecewise constant for convex pieces. As a result, instead of relying on numerical techniques for finding the optimum ρ , our multidimensional approach enumerates all the pieces for which the objective function is constant. Furthermore, this technique determines the exact optimum ρ rather than an approximation; one consequence of doing so is that our analysis can be extended to apply generalization guarantees from [BDL20] when learning the optimal ρ over a family of instances.

Sequence alignment. For the problem of aligning pairs of genomic sequences, one typically obtains the best alignment using a dynamic program with some costs or weights assigned to edits of different kinds, such as insertions, substitutions, or reduplications [Wat89]. Prior work has considered learning these weights by examining the cost of alignment for the possible weight settings. [BDD + 21] considers the problem for how many training examples (say pairs of DNA sequences with known ancestral or evolutionary relationship) are needed to learn the best weights for good generalization on unseen examples. More closely related to our work is the line of work which proposes algorithms which compute the partition of the space of weights which result in different optimal alignments [GBN94, GS96]. In this work, we propose a new algorithm which computes this partition more efficiently.

Pricing problems. Data-driven algorithms have been proposed and studied for automated mechanism design [MR15, BSV18, BDV18]. Prior work on designing multi-dimensional mechanisms has focused mainly on the generalization guarantees, and include studying classes of structured mechanisms for revenue maximization [MR15, BSV16, Syr17, BSV18]. Computationally efficient algorithms have been proposed for the two-part tariff pricing problem [BPS20]. Our algorithms have output-sensitive complexity and are more efficient than [BPS20] even for worst-case outputs. Also, our techniques are more general and apply to mechanism design besides two-part tariff pricing. [BB24] consider discretization-based techniques which are not output-sensitive and known to not work in problems beyond mechanism design, for example, data-driven clustering [BNVW17].

On cell enumeration of hyperplane arrangements. A finite set of t hyperplanes in R d induces a collection of faces in dimensions ≤ d , commonly referred to as an arrangement . A well-known inductive argument implies that the number of d -faces, or cells , in an arrangement of t hyperplanes in R d is at most O ( t d ) . Several algorithms, ranging from topological sweep based [EG86] to incremental construction based [Xu20], can enumerate all the cells (typically represented by collections of facetinducing hyperplanes) in optimal worst-case time O ( t d ) . The first output-sensitive algorithm proposed for this problem was the reverse search based approach of [AF96], which runs in O ( tdR · LP( d, t )) time, where R denotes the output-size (number of cells in the output) and LP( d, t ) is the time for solving a linear program in d variables and t constraints. This was further improved by a factor of t via a more refined reverse-search [Sle99], and by additive terms via a different incremental construction based approach by [RC18]. We propose a novel approach for cell enumeration based on Clarkson's algorithm [Cla94] for removing redundant constraints from linear systems of inequalities, which asymptotically matches the worst case output-sensitive running times of these algorithms while improving over their runtime in the presence of additional structure possessed by the typical piecewise structured (Definition 1) loss functions encountered in data-driven algorithm design.

C Output-sensitive Parameterized Complexity

Output-sensitive algorithms have a running time that depends on the size of the output for any input problem instance. Output-sensitive analysis is frequently employed in computational geometry, for example Chan's algorithm [Cha96] computes the convex hull of a set of 2-dimensional points in time O ( n log R ) , where R is the size of the (output) convex hull. Output-sensitive algorithms are useful if the output size is variable, and 'typical' output instances are much smaller than worst case instances. Parameterized complexity extends classical complexity theory by taking into account not only the total input length n , but also other aspects of the problem encoded in a parameter k 10 . The motivation is to confine the super-polynomial runtime needed for solving many natural problems strictly to the parameter. Formally, a parameterized decision problem (or language) is a subset L ⊆ Σ ∗ × N , where Σ is a fixed alphabet, i.e. an input ( x, k ) to a parameterized problem consists of two parts, where the second part k is the parameter. A parameterized problem L is fixed-parameter tractable if there exists an algorithm which on a given input ( x, k ) ∈ Σ ∗ × N , decides whether ( x, k ) ∈ L in f ( k ) · poly ( | x | ) time, where f is an arbitrary computable function in k [DF12]. FPT is the class of all parameterized problems which are fixed-parameter tractable. In contrast, the class XP (aka slicewise-polynomial ) consists of problems for which there is an algorithm with running time | x | f ( k ) . It is known that FPT ⊊ XP. We consider an extension of the above to search problems and incorporate output-sensitivity in the following definition.

Definition 3 (Output-polynomial Fixed Parameter Tractable) . A parameterized search problem P : Σ ∗ × N → ˜ Σ ∗ is said to be output-polynomial fixed-parameter tractable if there exists an algorithm which on a given input ( x, k ) ∈ Σ ∗ × N , computes the output P ( x, k ) ∈ ˜ Σ ∗ in time f ( k ) · poly ( | x | , R ) , where R = | P ( x, k ) | is the output size and f is an arbitrary computable function in k .

As discussed above, output-sensitvity and fixed-parameter tractability both offer more fine-grained complexity analysis than traditional (input) polynomial time complexity. Both techniques have been employed in efficient algorithmic enumeration [Fer02, Nak04] and have gathered recent interest [FGS + 19]. In this work, we consider the optimization problem of selecting tunable parameters over a continuous domain C ⊂ R d with the fixed-parameter k = d , the number of tunable parameters. We will design OFPT enumeration algorithms which, roughly speaking, output a finite 'search space' which can be used to easily find the best parameter for the problem instance.

D Augmented Clarkson's algorithm

We describe here the details of the Augmented Clarkson's algorithm, which modifies the algorithm of Clarkson [Cla94] with additional bookkeeping needed for tracking the partition cells in Algorithm 1. The underlying problem solved by Clarkson's algorithm may be stated as follows.

Problem Setup. Given a linear system of inequalities Ax ≤ b , an inequality A i x ≤ b i is said to be redundant in the system if the set of solutions is unchanged when the inequality is removed from the system. Given a system ( A ∈ R m × d , b ∈ R m ) , find an equivalent system with no redundant inequalities.

Note that to test if a single inequality A i x ≤ b i is redundant, it is sufficient to solve the following LP in d variables and m constraints.

Using this directly to solve the redundancy removal problem gives an algorithm with running time m · LP(d , m) , where LP(d , m) denotes the time to solve an LP in d variables and m constraints. This can be improved using Clarkson's algorithm if the number of non-redundant constraints s is much less than the total number of constraints m (Theorem 2.1).

We assume that an interior point z ∈ R d satisfying Ax < b is given. At a high level, one maintains the set of non-redundant constraints I discovered so far i.e. A i x ≤ b i is not redundant for each

̸

i ∈ I . When testing a new index k , the algorithm solves an LP of the form 2 and either detects that A k x ≤ b k is redundant, or finds index j ∈ [ m ] \ I such that A j x ≤ b j is non-redundant. The latter involves the use of a procedure RayShoot ( A,b, z, x ) which finds the non-redundant hyperplane hit by a ray originating from z in the direction x -z ( x ∈ R d ) in the system A,b . The size of the LP needed for this test is LP(d , | I | +1) from which the complexity follows.

To implement the RayShoot procedure, we can simply find the intersections of the ray x ∗ -z with the hyperplanes A j x ≤ b j and output the one closest to z (defining the cell facet in that direction). We also output an interior point from the adjacent cell during this computation, which saves us time relative to [Sle99] where the interior point is computed for each cell (our Clarkson based approach also circumvents their need for the Raindrop procedure [BFM97]). Finally we state the running time guarantee of Algorithm 2, which follows from the original result of Clarkson [Cla94].

Algorithm 3: RayShoot

̸

Theorem D.1 (Augmented Clarkson's algorithm) . Given a list L of k half-space constraints in d dimensions, Algorithm 2 outputs the set I ⊆ [ L ] of non-redundant constraints in L , as well as auxiliary neighbor information, in time O ( k · LP( d, | I | +1)) , where LP( v, c ) is the time for solving an LP with v variables and c constraints.

E Additional details and proofs from Section 2

Proof of Theorem 2.2. Algorithm 1 maintains a set of visited or explored cells in C , and their bounding hyperplanes (corresponding to cell facets) in ˜ C . It also maintains a queue of cells, such that each cell in the queue has been discovered in Line 12 as a neighbor of some visited cell in C , but is yet to be explored itself. The algorithm detects if a cell has been visited before by using its sign pattern on the hyperplanes in H . This can be done in O ( d log t ) time, since | C | = O ( t d ) using a well-known combinatorial fact (e.g. [Buc43]). For a new cell c , we run AUGMENTEDCLARKSON to compute its bounding hyperplanes I c as well as sign patterns for neighbors in time O ( t LRS · LP( d, | I c | +1)) by Theorem 2.1. The computed neighbors are added to the queue in Line 14. A cell c gets added to the queue at this step up to | I c | times, but is not explored if already visited. Thus we run up to V iterations

of AUGMENTEDCLARKSON and up to 1 + ∑ c ∈ V H | I c | = 2 E +1 queue insertions/removals. Using efficient union-find data-structures, the set union and membership queries (for C and ˜ C ) can be done in ˜ O (1) time per iteration of the loop. So total time over the V cell explorations and no more than 2 E +1 iterations of the while loop is ˜ O ( dE ) + O ( t LRS · ∑ c ∈ V H LP( d, | I c | +1))+ O ( V T CLRS ) .

Proof of Theorem 2.3. We will apply Algorithm 1 and compute the locally relevant hyperplanes by simply taking a union of the facet-inducing hyperplanes at any point x , across the problem instances.

Let G ( i ) = ( V ( i ) , E ( i ) ) denote the cell adjacency graph for the cells in ˜ C i . We apply Algorithm 1 implicitly over H = ∪ i H ( i ) where H ( i ) is the collection of facet-inducing hyperplanes in ˜ C i . To implement COMPUTELOCALLYRELEVANTSEPARATORS ( x , H ) we simply search the computed partition cell ˜ C i for the cell containing x in each problem instance x i in time O ( ∑ i | E ( i ) | ) = O ( m | E S | ) , obtain the set of corresponding facet-inducing hyperplanes H ( i ) x , and output ˜ H x = ∪ i H ( i ) x in time O ( mt LRS ) . The former step only needs to be computed once, as the partition cells for subsequent points can be tracked in O ( m ) time. Theorem 2.2 now gives a running time of ˜ O ( d | E S | + m | E S | + mt LRS | V S | + mt LRS · ∑ c ∈ V S LP( d, | I c | +1)) .

F Profit maximization in pricing problems

Prior work [MR15, BSV18] on data-driven mechanism design has shown that the profit as a function of the prices (parameters) is ( F , t ) -decomposable with F the set of linear functions on R d for a large number of mechanism classes (named ( d, t ) -delineable mechanisms in [BSV18]). We will instantiate our approach for multi-item pricing problems which are ( F , t ) -decomposable and analyse the running times. In contrast, recent work [BB24] employs data-independent discretization for computationally efficient data-driven algorithm design for mechanism design problems even in the worst-case. This discretization based approach is not output-sensitive and is known to not work well for other applications like data-driven clustering.

F.1 Two-part tariff pricing

In the two-part tariff problem [Lew41, Oi71], the seller with multiple identical items charges a fixed price, as well as a price per item purchased. For example, cab meters often charge a base cost for any trip and an additional cost per mile traveled. Subscription or membership programs often require an upfront joining fee plus a membership fee per renewal period, or per service usage. Often there is a menu or tier of prices, i.e. a company may design multiple subscription levels (say basic, silver, gold, platinum), each more expensive than the previous but providing a cheaper per-unit price. Given access to market data (i.e. profits for different pricing schemes for typical buyers) we would like to learn how to set the base and per-item prices to maximize the profit. We define these settings formally as follows.

Two-part tariffs. The seller has K identical units of an item. Suppose the buyers have valuation functions v i : { 1 , . . . , K } → R ≥ 0 where i ∈ { 1 , . . . , m } denotes the buyer, and the value is assumed to be zero if no items are bought. Buyer i will purchase q quantities of the item that maximizes their utility u i ( q ) = v i ( q ) -( p 1 + p 2 q ) , buying zero units if the utility is negative for each q > 0 . The revenue, which we want to maximize as the seller, is zero if no item is bought, and p 1 + p 2 q if q > 0 items are bought. The algorithmic parameter we want to select is the price ρ = ⟨ p 1 , p 2 ⟩ , and the problem instances are specified by the valuations v i . We also consider a generalization of the above scheme: instead of just a single two-part tariff (TPT), suppose the seller provides a menu of TPTs ( p 1 1 , p 1 2 ) , . . . , ( p ℓ 1 , p ℓ 2 ) of length ℓ . Buyer i selects a tariff ( p j 1 , p j 2 ) from the menu as well as the item quantity q to maximize their utility u j i ( q ) = v i ( q ) -( p j 1 + p j 2 q ) . This problem has 2 ℓ parameters, ρ = ( p 1 1 , p 1 2 , . . . , p ℓ 1 , p ℓ 2 ) , and the single two-part tariff setting corresponds to ℓ = 1 .

The dual class functions in this case are known to be piecewise linear with linear boundaries ([BSV18], restated as Lemma G.2 in Appendix G). We will now implement Algorithm 1 for this problem by specifying how to compute the locally relevant hyperplanes. For any price vector x = ρ , say the buyers buy quantities ( q 1 , . . . , q m ) ∈ { 0 , . . . , K } m according to tariffs ( j 1 , . . . , j m ) ∈ [ ℓ ] m . For a fixed price vector this can be done in time O ( mKℓ ) by computing argmax q,j u j i ( q ) for each buyer

̸

at that price for each two-part tariff in the menu. Then for each buyer we have K ( ℓ -1) potential alternative quantities and tariffs given by hyperplanes u j i i ( q i ) ≥ u j ′ i ( q ′ ) , q ′ = q i , j ′ = j i , for a total of t LRS = mK ( ℓ -1) locally relevant hyperplanes. Thus T CLRS = O ( mKℓ ) for the above approach, and Theorem 2.2 implies the following runtime bound.

̸

Theorem F.1. There exists an implementation of COMPUTELOCALLYRELEVANTSEPARATORS in Algorithm 1, which given valuation function v ( · ) for a single problem instance, computes all the R pieces of the dual class function u v ( · ) in time ˜ O ( R 2 (2 ℓ ) ℓ +2 K ) , where the menu length is ℓ , and there are K units of the good.

Theorem F.1 together with Theorem 2.3 implies an implementation of the search-ERM problem over m buyers (with valuation functions v i ( · ) for i ∈ [ m ] ) in time O ( R 2 Σ (2 ℓ ) ℓ +2 mK ) , where R Σ denotes the number of pieces in the total dual class function U ⟨ v 1 ,...,v m ⟩ ( · ) = ∑ i u v i ( · ) (formally, Corollary G.1 in Appendix G). In contrast, prior work for this problem has only obtained an XP runtime of ( mK ) O ( ℓ ) [BPS20]. For the special case ℓ = 1 , we also provide an algorithm (Algorithm 4 in Appendix G.2) that uses additional structure of the polytopes and employs a computational geometry algorithm due to [CE92] to compute the pieces in optimal O ( mK log( mK ) + R Σ ) time, improving over the previously best known runtime of O ( m 3 K 3 ) due to [BPS20] even for worst-case R Σ . The worst-case improvement follows from a bound of R Σ = O ( m 2 K ) on the number of pieces, which we establish in Theorem G.3. We further show that our running time for ℓ = 1 is asymptotically optimal under the algebraic decision-tree model of computation, by a linear time reduction from the element uniqueness problem (Theorem G.6).

F.2 Item-Pricing with anonymous prices

We will consider a market with a single seller, interested in designing a mechanism to sell m distinct items to n buyers. We represent a bundle of items by a quantity vector q ∈ Z m ≥ 0 , such that the number of units of the i th item in the bundle denoted by q is given by its i th component q [ i ] . In particular, the bundle consisting of a single copy of item i is denoted by the standard basis vector e i , where e i [ j ] = I { i = j } , where I {·} is the 0-1 valued indicator function. Each buyer j ∈ [ n ] has a valuation function v j : Z m ≥ 0 → R ≥ 0 over bundles of items. We denote an allocation as Q = ( q 1 , . . . , q n ) where q j is the bundle of items that buyer j receives under allocation Q . Under anonymous prices, the seller sets a price p i per item i . There is some fixed but arbitrary ordering on the buyers such that the first buyer in the ordering arrives first and buys the bundle of items that maximizes their utility, then the next buyer in the ordering arrives, and so on. For a given buyer j and bundle pair q 1 , q 2 ∈ { 0 , 1 } m , buyer j will prefer bundle q 1 over bundle q 2 so long as u j ( q 1 ) > u j ( q 2 ) where u j ( q ) = v j ( q ) -∑ i : q [ i ]=1 p i . Therefore, for a fixed set of buyer values and for each buyer j , their preference ordering over the bundles is completely determined by the ( 2 m 2 ) hyperplanes. The dual class functions are known to be piecewise linear with linear boundaries [BSV18].

̸

To implement Algorithm 1 for this problem we specify how to compute the locally relevant hyperplanes. For any price vector x = ( p 1 , . . . , p m ) , say the buyers buy bundles ( q 1 , . . . , q n ) . For a fixed price vector this can be done in time O ( n 2 m ) by computing argmax q ⊆ I j u j ( q ) for each buyer at that price vector, where I j denotes the remaining items after allocations to buyers 1 , . . . , j -1 at that price. Then for each buyer we have at most 2 m -1 potential alternative bundles given by hyperplanes u j ( q ) ≥ u j ( q ′ ) , q ′ = q i , for a total of t LRS ≤ n (2 m -1) locally relevant hyperplanes. Thus T CLRS = O ( n 2 m ) for the above approach, and Theorem 2.2 implies the following runtime bound.

Theorem F.2. There exists an implementation of COMPUTELOCALLYRELEVANTSEPARATORS in Algorithm 1, which given valuation functions v i ( · ) for i ∈ [ n ] , computes all the R pieces of the dual class function in time ˜ O ( R 2 (2 m ) m n ) , where there are m items and n buyers.

Proof. In the terminology of Theorem 2.2, we have d = m , E ≤ R 2 , V = R , T CLRS = O ( n 2 m ) , t LRS = n (2 m -1) . By [Cha18], we have ∑ c ∈ V H LP( d, | I c | +1) ≤ O ( Ed d ) ≤ O ( R 2 m m ) . Thus, Theorem 2.2 implies a runtime bound on Algorithm 1 of ˜ O ( dE + V T CLRS + t LRS · ∑ c ∈ V H LP( d, | I c | + 1)) = ˜ O ( R 2 (2 m ) m n ) .

Our approach yields an efficient algorithm when the number of items m and the number of dual class function pieces R are small. Prior work on item-pricing with anonymous prices has only focussed on sample complexity of the data-driven design problem [BSV18].

G Additional details and proofs from Section F

Proof of Theorem F.1. In the terminology of Theorem 2.2, we have d = 2 ℓ , E ≤ R 2 , V = R , T CLRS = O ( Kℓ ) , t LRS ≤ Kℓ . By [Cha18], we have ∑ c ∈ V H LP( d, | I c | + 1) ≤ O ( Ed d/ 2+1 ) ≤ O ( R 2 (2 ℓ ) ℓ +1 ) . Thus, Theorem 2.2 implies a runtime bound on Algorithm 1 of ˜ O ( dE + V T CLRS + t LRS · ∑ c ∈ V H LP( d, | I c | +1)) = ˜ O ( R 2 (2 ℓ ) ℓ +2 K ) .

Corollary G.1. There exists an implementation of COMPUTELOCALLYRELEVANTSEPARATORS in Algorithm 1, which given valuation functions v i ( · ) for i ∈ [ m ] , computes all the R Σ pieces of the total dual class function U ⟨ v 1 ,...,v m ⟩ ( · ) = ∑ i u v i ( · ) in time ˜ O ( R 2 Σ (2 ℓ ) ℓ +2 mK ) , where the menu length is ℓ , there are K units of the good and m is the number of buyers.

Proof. We first compute the pieces for each of the problem instances (single buyers) and then the pieces in the sum dual class function using Theorem 2.3. By Theorem F.1, the former takes time ˜ O ( mR 2 (2 ℓ ) ℓ +2 K ) and the latter can be implemented in time O (( m + 2 ℓ ) R 2 Σ + mKℓ ∑ c ∈ V S LP( d, | I c | +1)) = ˜ O ( R 2 Σ (2 ℓ ) ℓ +2 mK ) , which dominates the overall running time.

G.1 Piecewise structure of the dual class function

The following lemma restates the result from [BSV18] in terms of Definition 1. Note that u ρ in the following denotes the revenue function (or seller's utility) and should not be confused with the buyer utility function u i .

Lemma G.2. Let U be the set of functions { u ρ : v ( · ) ↦→ p j ∗ 1 + p j ∗ 2 q ∗ | q ∗ , j ∗ = argmax q,j v ( q ) -ρ j · ⟨ 1 , q ⟩ , ρ j = ⟨ p j 1 , p j 2 ⟩} that map valuations v ( · ) to R . The dual class U ∗ is ( F , ( Kℓ ) 2 ) -piecewise decomposable, where F = { f c : U → R | c ∈ R 2 ℓ } consists of linear functions f c : u ρ ↦→ ρ · c .

We also bound the number of pieces R in the worst-case for ℓ = 1 . The following bound implies that our algorithm is better than prior best algorithm which achieves an O ( m 3 K 3 ) runtime bound, even for worst case outputs.

Theorem G.3. Let menu length ℓ = 1 . The number of pieces R Σ in the total dual class function U ⟨ v 1 ,...,v m ⟩ ( · ) = ∑ i u v i ( · ) is at most O ( m 2 K ) .

Proof. By Lemma G.2, the dual class is ( F , K 2 ) -piecewise decomposable, where F is the class of linear functions. That is, the two-dimensional parameter space ( p 1 , p 2 ) can be partitioned into polygons by at most K 2 straight lines such that any dual class function u v i is a linear function inside any polygon.

We first tighten the above result to show that the dual class is in fact ( F , 2 K +2) -piecewise decomposable, that is the number of bounding lines for the pieces is O ( K ) . This seems counterintuitive since for any buyer i , we have Θ( K 2 ) lines u i ( q ) ≥ u i ( q ′ ) for q < q ′ ∈ { 0 , . . . , K } . If q > 0 , u i ( q ) = u i ( q ′ ) are axis-parallel lines with intercepts v i ( q ′ ) -v i ( q ) q ′ -q . Since for any pair q, q ′ the buyer has a fixed (but opposite) preference on either side of the axis-parallel line, we have at most K distinct horizontal 'slabs' corresponding to buyer's preference of quantities, i.e. regions between lines p 2 = a and p 2 = b for some a, b > 0 . Thus we have at most K non-redundant lines. Together with another K lines u i (0) = u i ( q ′ ) and the axes, we have 2 K +2 bounding lines in all as claimed.

We will next use this tighter result to bound the number of points of intersection of non-collinear non-axial bounding lines, let's call them crossing points , across all instances ⟨ v 1 , . . . , v m ⟩ . Consider a pair of instances given by buyer valuation functions v i , v j . We will establish that the number of crossing points are at most 4 K for the pair of instances. Let l i and l j be bounding lines for the pieces of u v i and u v j respectively. If they are both axis-parallel, then they cannot result in a crossing point. For pairs of bounding lines l i and l j such that l i is axis-parallel and l j is not, we can have at most K crossing points in all. This is because any fixed l i can intersect at most one such l j since buyer j 's

preferred quantity q j is fixed along any horizontal line, unless q j changes across l i in which case the crossing points for the consecutive l j 's coincide. Thus, there is at most one such crossing point for each of at most K axis-parallel l i 's. By symmetry, there are at most K crossing points between l i , l j where l j is axis parallel and l i is not. Finally, if neither l i , l j is axis parallel, we claim there can be no more than 2 K crossing points. Indeed, if we arrange these points in the order of increasing p 2 , then the preferred quantity of at least one of the buyers i or j strictly decreases between consecutive crossing points. Thus, across all instances, there are at most 2 m 2 K crossing points.

Finally, observe that the cell adjacency graph G U for the pieces of the total dual class function U ⟨ v 1 ,...,v m ⟩ ( · ) is planar in this case. The vertices of this graph correspond to crossing points, or intersections of bounding lines with the axes. The latter is clearly O ( mK ) since there are O ( K ) bounding lines in any problem instance. Using the above bound on crossing points, the number of vertices in G U is O ( m 2 K ) . Since G U is a simple, connected, planar graph, the number of faces is no more than twice the number of vertices and therefore the number of pieces R Σ is also O ( m 2 K ) .

G.2 Optimal algorithm for Single TPT pricing

Consider the setting with menu-length ℓ = 1 . The key insight is to characterize the polytopic structure of the pieces of the dual class function for a single buyer. We do this in Lemma G.4.

Lemma G.4. Consider a single buyer with valuation function v ( · ) . The buyer buys zero units of the item except for a set ϱ v ⊂ R 2 , where ϱ v is a convex polygon with at most K +2 sides. Moreover, ϱ v can be subdivided into K ′ ≤ K polygons ϱ ( i ) v , each a triangle or a trapezoid with bases parallel to the ρ 1 -axis, such that for each i ∈ [ K ′ ] the buyer buys the same quantity q ( i ) of the item for all prices in ϱ ( i ) v .

Proof. We proceed by an induction on K , the number of items. For K = 1 , it is straightforward to verify that ϱ v is the triangle p 1 ≥ 0 , p 2 ≥ 0 , p 1 + p 2 ≤ v (1) .

Let K > 1 . If we consider the restriction of the valuation function v ( · ) to K -1 items, we have a convex polygon ϱ ′ v satisfying the induction hypothesis. To account for the K -th item we only need to consider the region p 1 ≥ 0 , p 2 ≥ 0 , p 2 ≤ v ( K ) -v ( q ) K -q for 0 < q < K , and p 1 + p 2 K ≤ v ( K ) . If this region is empty, ϱ v = ϱ ′ v , and we are done. Otherwise, denoted by ϱ ( K ′ ) v with q ( K ′ ) = K , the region where the buyer would buy K units of the item is a trapezoid with bases parallel to the ρ 1 -axis. We claim that ϱ v = ( ϱ ′ v ∩ p 2 ≤ v ( K ) -v ( q ) K -q ) ∪ ϱ ( K ′ ) v and it satisfies the properties in the lemma.

We have q ( K ′ -1) = argmin q v ( K ) -v ( q ) K -q such that the buyer's preference changes from q ′ = q ( K ′ -1) to q ( K ′ ) = K units across the line p 2 = v ( K ) -v ( q ′ ) K -q ′ . To prove ϱ v is convex, we use the inductive hypothesis on ϱ ′ v and observe that ρ 1 = v ( K ) -p 2 K coincides with ρ ′ 1 = v ( q ′ ) -p 2 q ′ for p 2 = v ( K ) -v ( q ′ ) K -q ′ . Also the only side of ϱ v that is not present in ϱ ′ v lies along the line p 1 + p 2 K = v ( K ) , thus ϱ v has at most K +2 sides. The subdivison property is also readily verified given the construction of ϱ v from ϱ ′ v and ϱ ( K ′ ) v .

Based on this structure, we propose Algorithm 4 which runs in in O ( mK log( mK ) + R Σ ) time.

Theorem G.5. There is an algorithm (Algorithm 4) that, given valuation functions v i ( · ) for i ∈ [ m ] , computes all the R pieces of the total dual class function U ⟨ v 1 ,...,v m ⟩ ( · ) for K units of the good from the m samples in O ( mK log( mK ) + R Σ ) time.

Proof. Note that if 0 < q < q ′ and v i ( q ) > v i ( q ′ ) , then for any ρ 1 ≥ 0 , ρ 2 ≥ 0 we have that u i ( q ) = v i ( q ) -( p 1 + p 2 q ) > v i ( q ′ ) -( p 1 + p 2 q ) , or the buyer will never prefer quantity q ′ of the item over the entire tariff domain. Thus, we will assume the valuations v i ( q ) are monotonic in q (we can simply ignore valuations at the larger value for any violation). Algorithm 4 exploits the structure in Lemma G.4 and computes the O ( K ) line segments bounding the dual class pieces for a single buyer i in O ( K ) time. Across m buyers, we have O ( mK ) line segments (computed in O ( mK ) time). The topological plane-sweep based algorithm of [CE92] now computes all the intersection points in O ( mK (log mK ) + R Σ ) time. Here we have used that the number of polytopic vertices is O ( R Σ ) using standard result for planar graphs.

Algorithm 4: ComputeFixedAllocationRegions

We further show that this bound is essentially optimal. A runtime lower bound of Ω( mK + R Σ ) follows simply due to the amount of time needed for reading the complete input and producing the complete output. We prove a stronger lower bound which matches the above upper bound by reduction to the element uniqueness problem (given a list of n numbers, are there any duplicates?) for which an Ω( n log n ) lower bound is known in the algebraic decision-tree model of computation.

Theorem G.6. Given a list of n numbers L = ⟨ x 1 , . . . , x n ⟩ ∈ N n , there is a linear time reduction to a m -buyer, K -item TPT pricing given by v i ( · ) , i ∈ [ m ] , with mK = Θ( n ) , such that the pieces of the total dual class function U ⟨ v 1 ,...,v m ⟩ ( · ) can be used to solve the element uniqueness problem for L in O ( n ) time.

Proof. Let mK = n be any factorization of n into two factors. We construct a m -buyer, K +1 item single TPT pricing scheme as follows. Define y j = x j + max k x k + 1 for each x j in the list L . For every buyer i ∈ [ m ] , we set v i (1) = max k x k +1 and v i ( q +1) = ∑ q j =1 y j +( i -1) K for each q ∈ [ K ] . Buyer i 's pieces include the segments p 2 = ( v i ( q +1) -v i ( q )) / ( q +1 -1) = x q +( i -1) K for q ∈ [ K ] (Lemma G.4). Thus, across all buyers i ∈ [ m ] , we have mK = n segments along the lines p 2 = x j for j ∈ [ n ] . We say a duplicate is present if there are fewer than mK segments parallel to the p 1 -axis in the pieces of the total dual class function, otherwise we say 'No' (i.e. all elements are distinct). This completes the linear-time reduction.

H Comparing the quality of single, complete and median linkage procedures on different data distributions

We will construct clutering instances where each of two-point based linkage procedures, i.e. single, complete and median linkage, dominates the other two procedures. Let T S sgl , T S cmpl and T S med denote the cluster tree on clustering instance S using D sgl , D cmpl and D med as the merge function (defined in Section 3) respectively for some distance metric d which will be evident from context. We have the following theorem.

Theorem H.1. For any n ≥ 10 , for i ∈ { 1 , 2 , 3 } , there exist clustering instances S i with | S i | = n and target clusterings C i such that the hamming loss of the optimal pruning of the cluster trees constructed using single, complete and median linkage procedures (using the same distance metric d ) satisfy

Proof. In the following constructions we will have S i ⊂ R 2 and the distance metric d will be the Euclidean metric. Also we will have number of target clusters k = 2 .

Construction of S 1 , C 1 . For S 1 , we will specify the points using their polar coordinates. We place a single point x at the origin (0 , ϕ ) and n -1 8 points each along the unit circle at locations y 1 = (1 , 0) , y 2 = (1 , π 4 -ϵ ) , y 3 = (1 , π 2 ) , y 4 = (1 , 3 π 4 -ϵ ) , y 5 = (1 , π ) , y 6 = (1 , 5 π 4 -ϵ ) , y 7 = (1 , 3 π 2 ) and y 8 = (1 , 7 π 4 -ϵ ) , where ϵ = 0 . 001 . Also set C 1 = {{ x } , S 1 \ { x }} .

̸

In each linkage procedure, the first n -9 merges will join coincident points at locations y j , j ∈ [8] , let ˜ y j denote the corresponding sets of merged points. The next four merges will be z j := { ˜ y j , ˜ y j +1 } for j ∈ { 1 , 3 , 5 , 7 } for each procedure since d ( y j , y j +1 ) = √ 2 -2 cos( π 4 -ϵ ) < min { √ 2 -2 cos( π 4 + ϵ ) , 1 } , again common across all procedures. Now single linkage will continue to merge clusters on the unit circle since √ 2 -2 cos( π 4 + ϵ ) < 1 , however both complete and median linkage will join each of z j , j ∈ { 1 , 3 , 5 , 7 } to the singleton cluster { x } since the median (and therefore also the maximum distance between points in z j , z j ′ , j = j ′ is at least √ 2 > 1 . Therefore a 2-pruning 11 of T S 1 sgl yields C 1 i.e ℓ ( T S 1 sgl , C 1 ) = 0 , while a 2-pruning of T S 1 cmpl or T S 1 med would yield { z j , S 1 \ z j } for some j ∈ { 1 , 3 , 5 , 7 } , corresponding to a hamming loss of 1 4 +Ω( 1 n ) = Ω(1) .

Construction of S 2 , C 2 . For S 2 , we will specify the points using their Cartesian coordinates. We place single points x 1 , x 2 at (0 , 0) and (3 . 2 , 0 . 5) and n -2 2 points each at y 1 = (1 . 1 , 1 . 8) and y 2 = (1 . 8 , 0 . 5) . We set C 2 = {{ ( x, y ) ∈ S 2 | y > 1 } , { ( x, y ) ∈ S 2 | y ≤ 1 }} . The distances between pairs of points may be ordered as

All linkage procedures will merge the coincident points at y 1 and y 2 (respectively) for the first n -4 merges. Denote the corresponding clusters by ˜ y 1 and ˜ y 2 respectively. The next merge will be z 2 := { x 2 , ˜ y 2 } in all cases. Now single linkage will join z 2 with ˜ y 1 . Further, since n ≥ 10 , n -2 2 ≥ 4 and therefore the median distance between z 2 and ˜ y 1 is also d ( y 1 , y 2 ) . However, since d ( x 1 , y 1 ) < d ( x 2 , y 1 ) , the complete linkage procedure will merge { x 1 , z 2 } . Finally, the two remaining clusters are merged in each of the two procedures. Clearly, 2-pruning of T S 2 cmpl yields C 2 or ℓ ( T S 2 cmpl , C 2 ) = 0 . However, ℓ ( T S 2 sgl , C 2 ) = ℓ ( T S 2 med , C 2 ) = 1 2 -O ( 1 n ) = Ω(1) .

Construction of S 3 , C 3 . We specify the points in S 3 using their Cartesian coordinates. We place n -1 6 points each at x 1 = (0 , 0) , x ′ 2 = (0 , 1 + 2 ϵ ) , n -1 12 points each at x ′ 1 = (0 , ϵ ) , x 2 = (0 , 1 + ϵ ) , n -1 4 points each at y 1 = (1+0 . 9 ϵ, ϵ ) , y 2 = (1+ ϵ, 1+1 . 9 ϵ ) , and one point z 1 = (0 , 2) with ϵ = 0 . 3 . With some abuse of notation we will use the coordinate variables defined above to also denote the collection of points at their respective locations. We set C 3 = {{ ( x, y ) ∈ S 2 | x ≤ 0 } , { ( x, y ) ∈ S 2 | x > 0 }} .

After merging the coincident points, all procedures will merge clusters ˜ x 1 := { x 1 , x ′ 1 } and ˜ x 2 := { x 2 , x ′ 2 , z 1 } . Let's now consider the single linkage merge function. We have D sgl (˜ x 1 , ˜ x 2 ; d ) = 1 and all other cluster pairs are further apart. The next merge is therefore ˜ x := { ˜ x 1 , ˜ x 2 } . Also, D sgl (˜ x, y 1 ; d ) = 1 + 0 . 9 ϵ < min { D sgl (˜ x, y 2 ; d ) , D sgl ( y 1 , y 2 ; d ) } leading to the merge { ˜ x, y 1 } , and finally y 2 is merged in. A 2-pruning therefore has loss ℓ ( T S 3 sgl , C 3 ) = Ω(1) . On the other hand, D med (˜ x 1 , ˜ x 2 ; d ) = 1 + ϵ > D med ( y 1 , y 2 ; d ) = √ (1 + 0 . 9 ϵ ) 2 +0 . 01 ϵ 2 and D med (˜ x 1 , y 1 ; d ) = √ (1 + 0 . 9 ϵ ) 2 + ϵ 2 > { D med ( y 1 , y 2 ; d ) , D med (˜ x 1 , ˜ x 2 ; d ) } . As a result, median linkage would first

y3

y4

y2

y1

C2

y2

T

4

y5

y1

y6

y8

y2

C2

y1

C1

y7

C1

S1, C1

S2,C2

S3, C3

merge { y 1 , y 2 } , followed by { ˜ x 1 , ˜ x 2 } , and 2-pruning yields C 3 . Complete linkage also merges { y 1 , y 2 } first. But D cmpl (˜ x 1 , ˜ x 2 ; d ) = 2 > D cmpl (˜ x 1 , { y 1 , y 2 } ; d ) . Thus, ℓ ( T S 3 cmpl , C 3 ) = Ω(1) .

I Additional details and proofs from Section 3

Definition 4 (2-point-based merge function [BDL20]) . A merge function D is 2-point-based if for any pair of clusters A,B ⊆ X and any metric d , there exists a set of points ( a, b ) ∈ A × B such that D ( A,B ; d ) = d ( a, b ) . Furthermore, the selection of a and b only depend on the relative ordering of the distances between points in A and B . More formally, for any metrics d and d ′ such that d ( a, b ) ≤ d ( a ′ , b ′ ) if and only if d ′ ( a, b ) ≤ d ′ ( a ′ , b ′ ) , then D ( A,B ; d ) = d ( a, b ) implies D ( A,B ; d ′ ) = d ′ ( a, b ) .

For instance, single, median and complete linkage are 2-point-based, since the merge function D ( A,B ; d ) only depends on the distance d ( a, b ) for some a ∈ A,b ∈ B . We have the following observation about our parameterized algorithm families D ∆ ρ ( A,B ; δ ) when ∆ consists of 2-pointbased merge functions which essentially establishes piecewise structure with linear boundaries (in the sense of Definition 1).

Lemma I.1. Suppose S ∈ Π is a clustering instance, ∆ is a set of 2-point-based merge functions with | ∆ | = l , and δ is a set of distance metrics with | δ | = m . Consider the family of clustering algorithms with the parameterized merge function D ∆ ρ ( A,B ; δ ) . The corresponding dual class function u S ( · ) is piecewise constant with O ( | S | 4 lm ) linear boundaries.

Proof. Let ( a ij , b ij , a ′ ij , b ′ ij ) 1 ≤ i ≤ l, 1 ≤ j ≤ m ⊆ S be sequences of lm points each; for each such a , let g a : P → R denote the function

and let G = { g a | ( a ij , b ij , a ′ ij , b ′ ij ) 1 ≤ i ≤ l, 1 ≤ j ≤ m ⊆ S } be the collection of all such linear functions; notice that |G| = O ( | S | 4 lm ) . Fix ρ, ρ ′ ∈ P with g ( ρ ) and g ( ρ ′ ) having the same sign patterns for all such g . For each A,B,A ′ , B ′ ⊆ S , D i ∈ ∆ , and d j ∈ δ , we have D i ( A,B ; d j ) = d j ( a, b ) and D i ( A ′ , B ′ ; d j ) = d j ( a ′ , b ′ ) for some a, b, a ′ , b ′ ∈ S (since D i is 2-point-based). Thus we can write D ρ ( A,B ; δ ) = ∑ i ∈ [ m ] ,d j ∈ δ α i,j ( ρ ) d j ( a ij , b ij ) for some a ij , b ij ∈ S ; similarly, D ρ ( A ′ , B ′ ; δ ) = ∑ i ∈ [ m ] ,d j ∈ δ α i,j ( ρ ) d j ( a ′ ij , b ′ ij ) for some a ′ ij , b ′ ij ∈ S . As a result, D ρ ( A,B ; δ ) ≤ D ρ ( A ′ , B ′ ; δ ) if and only if

which is exactly when g a ( ρ ) ≤ 0 for some sequence a . Since g a ( ρ ) and g a ( ρ ′ ) have the same sign pattern, we have D ρ ( A,B ; δ ) ≤ D ρ ( A ′ , B ′ ; δ ) if and only if D ρ ′ ( A,B ; δ ) ≤ D ρ ′ ( A ′ , B ′ ; δ ) . So ρ

180

and ρ ′ induce the same sequence of merges, meaning the algorithm's output is constant on each piece induced by g , as desired.

From Lemma I.1, we obtain a bound on the number of hyperplanes needed to divide P into outputconstant pieces. Let H be a set of hyperplanes which splits P into output-constant pieces; then, a naive approach to finding a dual-minimizing ρ ∈ P is to enumerate all pieces generated by H , requiring O ( | H | d ) runtime. However, by constructing regions merge-by-merge and successively refining the parameter space, we can obtain a better runtime bound which is output-sensitive in the total number of pieces.

Proof of Lemma ?? . This is a simple corollary of Lemma I.1 for m = 1 . In this case, we have l = d +1 .

Lemma I.2. Consider the family of clustering algorithms with the parameterized merge function D 1 ρ ( A,B ; δ ) . Let T S ρ denote the cluster tree computed using the parameterized merge function D ∆ ρ ( A,B ; d 0 ) on sample S . Let U be the set of functions { u ρ : S ↦→ ℓ ( T S ρ , C ) | ρ ∈ R d } that map a clustering instance S to R . The dual class U ∗ is ( F , | S | 4 ) -piecewise decomposable, where F = { f c : U → R | c ∈ R } consists of constant functions f c : u ρ ↦→ c .

The key observation for the proof comes from [BDL20] where it is observed that two parameterized distance metrics d ρ 1 , d ρ 2 behave identically (yield the same cluster tree) on a given dataset S if the relative distance for all pairs of two points ( a, b ) , ( a ′ , b ′ ) ∈ S 2 × S 2 , d ρ ( a, b ) -d ρ ( a ′ , b ′ ) , has the same sign for ρ 1 , ρ 2 . This corresponds to a partition of the parameter space with | S | 4 hyperplanes, with all distance metrics behaving identically in each piece of the partition. More formally, we have

Proof of Lemma I.2. Let S be any clustering instance. Fix points a, b, a ′ , b ′ ∈ S . Define the linear function g a,b,a ′ ,b ′ ( ρ ) = ∑ i ρ i ( d i ( a, b ) -d i ( a ′ , b ′ )) . If d ρ ( · , · ) denotes the interpolated distance metric, we have that d ρ ( a, b ) ≤ d ρ ( a ′ , b ′ ) if and only if g a,b,a ′ ,b ′ ( ρ ) ≤ 0 . Therefore we have a set H = { g a,b,a ′ ,b ′ ( ρ ) ≤ 0 | a, b, a ′ , b ′ ∈ S } of | S | 4 hyperplanes such that in any piece of the sign-pattern partition of the parameter space by the hyperplanes, the interpolated distance metric behaves identically, i.e. for any ρ, ρ ′ in the same piece d ρ ( a, b ) ≤ d ρ ( a ′ , b ′ ) iff d ρ ′ ( a, b ) ≤ d ρ ′ ( a ′ , b ′ ) . The resulting clustering is therefore identical in these pieces. This means that for any connected

component R of R d \ H , there exists a real value c R such that u ρ ( s 1 , s 2 ) = c R for all ρ ∈ R d . By definition of the dual, u ∗ s 1 ,s 2 ( u ρ ) = u ρ ( s 1 , s 2 ) = c R . For each hyperplane h ∈ H , let g ( h ) ∈ G denote the corresponding halfspace. Order these k = | S | 4 functions arbitrarily as g 1 , . . . , g k . For a given connected component R of R d \ H , let b R ∈ { 0 , 1 } k be the corresponding sign pattern. Define the function f ( b R ) = f c R and for b not corresponding to any R , f ( b ) = f 0 . Thus, for each ρ ∈ R d ,

Corollary I.3. For any clustering instance S ∈ Π , the dual class function u S ( · ) for the family in Lemma I.2 is piecewise constant with O ( | S | 4 d ) pieces.

Lemma I.4. Let S ∈ Π be a clustering instance, ∆ be a set of merge functions, and δ be a set of distance metrics. Then, the corresponding dual class function u S ( · ) is piecewise constant with O (16 | S | ) linear boundaries of pieces.

Proof. For each subset of points A,B,A ′ , B ′ ⊆ S , let g A,B,A ′ ,B ′ : P → R denote the function

and let G be the collection of all such functions for distinct subsets A,B,A ′ , B ′ . Observe that G is a class of linear functions with |G| ≤ ( 2 | S | ) 4 = 16 | S | . Suppose that for ρ, ρ ′ ∈ P , g ( ρ ) and g ( ρ ′ ) have the same sign for all g ∈ G ; then, the ordering over all cluster pairs A,B of D ρ ( A,B ; δ ) is the same as that of D ρ ′ ( A,B ; δ ) . At each stage of the algorithm, the cluster pair A,B ⊆ S minimizing D ρ ( A,B ; δ ) is the same as that which minimizes D ρ ′ ( A,B ; δ ) , so the sequences of merges produced by ρ and ρ ′ are the same. Thus the algorithm's output is constant on the region induced by g A,B,A ′ ,B ′ , meaning u S ( · ) is piecewise constant on the regions induced by G , which have linear boundaries.

I.1 Execution Tree

Formally we define an execution tree (Figure 2) as follows.

Definition 5 (Execution tree) . Let S be a clustering instance with | S | = n , and ∅ ̸ = P ⊆ [0 , 1] d . The execution tree on S with respect to P is a depthn rooted tree T , whose nodes are defined recursively as follows: r = ([] , P ) is the root, where [] denotes the empty sequence; then, for any node v = ([ u 1 , u 2 , . . . , u t ] , Q ) ∈ T with t < n -1 , the children of v are defined as

For an execution tree T with v ∈ T and each i with 0 ≤ i ≤ n , we let P i denote the set of Q such that there exists a depthi node v ∈ T and a sequence of merges M with v = ( M , Q ) . Intuitively, the execution tree represents all possible execution paths (i.e. sequences for the merges) for the algorithm family when run on the instance S as we vary the algorithm parameter ρ ∈ P . Furthermore, each P i is a subdivision of the parameter space into pieces where each piece has the first i merges constant. We establish the execution tree captures all possible sequences of merges by some algorithm A ρ in the parameterized family via its nodes, and each node corresponds to a convex polytope if the parameter space P is a convex polytope (Lemmata I.5 and I.6).

Our cell enumeration algorithm for computing all the pieces of the dual class function now simply computes the execution tree, using Algorithm 1 to compute the children nodes for any given node, starting with the root.

Lemma I.5. Let S be a clustering instance and T be its execution tree with respect to P . Then, if a sequence of merges M = [ u 1 , u 2 , . . . , u t ] is attained by A ρ for some ρ ∈ P , then there exists some v ∈ T at depth t with v = ( M , Q ) and with Q ⊆ P being the exact set of values of ρ for which A ρ may attain M . Conversely, for every node v = ( M , Q ) ∈ T , M is a valid sequence of merges attainable by A ρ for some ρ ∈ P .

Proof. We proceed by induction on t . For t = 0 , the only possible sequence of merges is the empty sequence, which is obtained for all ρ ∈ P . Furthermore, the only node in T at depth 0 is the root ([] , P ) , and the set P is exactly where an empty sequence of merges occurs.

Now, suppose the claim holds for some t ≥ 0 . We show both directions in the induction step.

For the forward direction, let M t +1 = [ u 1 , u 2 , . . . , u t , u t +1 ] , and suppose M t +1 is attained by A ρ for some ρ ∈ P . This means that M t = [ u 1 , u 2 , . . . , u t ] is attained by A ρ as well; by the induction hypothesis, there exists some node v t = ( M t , Q t ) ∈ T at depth t , where ρ ∈ Q t and Q t is exactly the set of values for which A may attain M t . Now, u t +1 is a possible next merge by A ρ for some ρ ∈ Q t ; by definition of the execution tree, this means v t has some child v t +1 = ( M t +1 , Q t +1 ) in T such that Q t +1 is the set of values where u t +1 is the next merge in Q t . Moreover, Q t +1 is exactly the set of values ρ ∈ P for which A ρ can attain the merge sequence M t +1 . In other words for any ρ ′ ∈ P \ Q t +1 , A ρ ′ cannot attain the merge sequence M t +1 . Otherwise, either some ρ ′ ∈ P \ Q t attains M t +1 , meaning A ρ ′ attains M t (contradicting the induction hypothesis), or A ρ ′ attains M t +1 for some ρ ′ ∈ Q t +1 \ Q t , contradicting the definition of Q t +1 .

For the backward direction, let v t +1 = ( M t +1 , Q t +1 ) ∈ T at depth t + 1 . Since v t +1 is not the root, v t +1 must be the child of some node v t , which has depth t . By the induction hypothesis, v t = ( M t , Q t ) , where M t = [ u 1 , u 2 , . . . , u t ] is attained by A ρ for some ρ ∈ P . Thus by definition of the execution tree, M t +1 has the form [ u 1 , u 2 , . . . , u t , ( A,B )] , for some merging of cluster pairs ( A,B ) which is realizable for ρ ∈ Q t . Thus M t +1 is a valid sequence of merges attainable by A ρ for some ρ ∈ P .

Lemma I.6. Let S be a clustering instance and T be its execution tree with respect to P . Suppose P is a convex polytope; then, for each v = ( M , Q ) ∈ T , Q is a convex polytope.

Proof. We proceed by induction on the tree depth t . For t = 0 , the only node is ([] , P ) , and P is a convex polytope. Now, consider a node v ∈ T at depth t +1 ; by definition of the execution tree, v = ( M v , Q v ) is the child of some node u ∈ T , where the depth of u is t . Inductively, we know that w = ( M w , Q w ) , for some convex polytope Q w . We also know M w has the form M w = [ u 1 , u 2 , . . . , u t ] , and thus Q v is defined to be the set of points ρ ∈ Q w where the merge sequence M v = [ u 1 , u 2 , . . . , u t , ( A,B )] is attainable for some fixed A,B ⊆ S . Notice that the definition of being attainable by the algorithm A ρ is that D ρ ( A,B ; δ ) is minimized over all choices of next cluster pairs A ′ , B ′ to merge. That is, Q v is the set of points

Q v = { ρ ∈ Q w | D ρ ( A,B ; δ ) ≤ D ρ ( A ′ , B ′ ; δ ) for all available cluster pairs A ′ , B ′ after M w }

Since D ρ ( A,B ; δ ) is an affine function of ρ , the constraint D ρ ( A,B ; δ ) ≤ D ρ ( A ′ , B ′ ; δ ) is a halfspace. In other words, Q v is the intersection of a convex polytope Q w with finitely many half space constraints, meaning Q v is itself a convex polytope.

It follows from Lemma I.6 that P i forms a convex subdivision of P , where each P i +1 is a refinement of P i ; Figure 2 (in the appendix) shows an example execution tree corresponding to a partition of a 2 -dimensional parameter space. From Lemma I.5, the sequence of the first i merges stays constant on each region P ∈ P i . Our algorithm computes a representation of the execution tree of an instance S with respect to P ; to do so, it suffices to provide a procedure to list the children of a node in the execution tree. Then, a simple breadth-first search from the root will enumerate all the leaves in the execution tree.

Now, our goal is to subdivide P into regions in which the ( j +1) st merge is constant. Each region corresponds to a cluster pair being merged at step j +1 . Since we know these regions are always convex polytopes (Lemma I.6), we can provide an efficient algorithm for enumerating these regions.

Our algorithm provides an output-sensitive guarantee by ignoring the cluster pairs which are never merged. Supposing there are n t unmerged clusters, we start with some point x ∈ P and determine which piece W it is in. Then, we search for more non-empty pieces contained in P by listing the 'neighbors' of W . The neighbors of W are pieces inside P that are adjacent to W ; to this end, we will more formally define a graph G P associated with P where each vertex is a piece and two vertices have an edge when the pieces are adjacent in space. Then we show that we can enumerate neighbors of a vertex efficiently and establish that G P is connected. It follows that listing the pieces is simply a matter of running a graph search algorithm from one vertex of G P , thus only incurring a cost for each non-empty piece rather than enumerating through all n 4 t pairs of pieces.

Proof of Corollary 3.2. The key observation is that on any iteration i , the number of adjacencies H i = O ( R i ) . This is because for any region P ∈ P i , P is a polygon divided into convex subpolygons, and the graph G P has vertices which are faces and edges which cross between faces. Since the subdivision of P can be embedded in the plane, so can the graph G P . Thus G P is planar, meaning H i = O ( R i ) . Plugging into Theorem 3.1, noting that ( n -i +1) 2 ≤ n 2 , H i ≤ H , and R i ≤ R , we obtain the desired runtime bound of O (∑ n i =1 ( R + RT M ) n 2 ) = O ( RT M n 3 ) .

I.2 Auxiliary lemmas and proofs of runtime bounds for our algorithm

Lemma I.7. Fix an affine function f : R → R d via f ( x ) = xa + b , for a, b ∈ R d and a = 0 d . For a subset S ⊆ R , if f ( S ) is convex and closed, then S is also convex and closed.

̸

̸

Proof. First note that f is injective, since a = 0 d . To show convexity of S , take arbitrary x, y ∈ S and λ ∈ [0 , 1] ; we show that λx +(1 -λ ) y ∈ S . Consider f ( λx +(1 -λ ) y ) :

By definition, ya + b, xa + b ∈ f ( S ) , so it follows that f ( λx + (1 -λ ) y ) ∈ f ( S ) by convexity of f ( S ) . So there exists some z ∈ S with f ( z ) = f ( λx + (1 -λ ) y ) , but since f is injective, λx +(1 -λ ) y = z ∈ S . Thus S is convex.

To show closedness of S , we show R \ S is open. Let N ( x, r ) denote the open ball of radius r around x , in either one-dimensional or d -dimensional space. Let x ∈ R \ S ; we know f ( x ) / ∈ f ( S ) since f is injective. Since R d \ f ( S ) is open, there exists some r > 0 with N ( f ( x ) , r ) ⊆ R d \ f ( S ) . Then, take e = r ∥ a ∥ 2 > 0 ; for every y ∈ N ( x, e ) , we have

and so f ( y ) ∈ N ( f ( x ) , r ) ⊆ R d \ f ( S ) , meaning y / ∈ S since f is injective. Thus N ( x, e ) ⊆ R \ S , meaning S is closed as desired.

This allows us to prove the following key lemma. We describe a proof sketch first. For arbitrary ( i, j ) , ( i ′ , j ′ ) ∈ V ∗ P , we show that there exists a path from ( i, j ) to ( i ′ , j ′ ) in G P . We pick arbitrary points w ∈ Q i,j , x ∈ Q i ′ ,j ′ ; we can do this because by definition, V ∗ P only has elements corresponding to non-empty cluster pairs. Then, we draw a straight line segment in P from w to x . When we do so, we may pass through other sets on the way; each time we pass into a new region, we traverse an edge in G P , so the sequence of regions we pass through on this line determines a G P -path from w to x .

Lemma I.8. The vertices V ∗ P of the region adjacency graph G P form a connected component; all other connected components of G P are isolated vertices.

Proof. It suffices to show that for arbitrary vertices ( i 1 , j 1 ) , ( i 2 , j 2 ) ∈ V P , there exists a path from ( i 1 , j 1 ) to ( i 2 , j 2 ) in G P . For ease of notation, define Q 1 = Q i 1 ,j 1 and Q 2 = Q i 2 ,j 2 .

̸

Fix arbitrary points u ∈ Q 1 and w ∈ Q 2 . If u = w then we're done, since the edge from ( i 1 , j 1 ) to ( i 2 , j 2 ) exists, so suppose u = w . Consider the line segment L defined as

Since Q 1 , Q 2 ⊆ P , we have u, w ∈ P . Furthermore, by convexity of P , it follows that L ⊆ P . Define the sets R i,j as

Since each Q i,j and L are convex and closed, so is each R i,j . Furthermore, since ⋃ i,j Q i,j = P , we must have ⋃ i,j R i,j = L . Finally, define the sets S i,j as

̸

Note that S i,j is convex and closed; the affine map f : S i,j → R i,j given by f ( x ) = xu +(1 -x ) w = x ( u -w ) + w has R i,j as an image. Furthermore, u -w = 0 d ; by Lemma I.7, the preimage S i,j must be convex and closed. Furthermore, ⋃ i,j S i,j = [0 , 1] .

The only convex, closed subsets of [0 , 1] are closed intervals. We sort the intervals in increasing order based on their lower endpoint, giving us intervals I 1 , I 2 , . . . , I ℓ . We also assume all intervals are non-empty (we throw out empty intervals). Let σ ( p ) denote the corresponding cluster pair associated with interval I p ; that is, if the interval I p is formed from the set S i,j , then σ ( p ) = ( i, j ) .

Define a i , b i to be the lower and upper endpoints, respectively, of I i . We want to show that for all 1 ≤ i ≤ ℓ -1 , the edge { σ ( i ) , σ ( i +1) } ∈ E P ; this would show that σ (1) is connected to σ ( ℓ ) in the V P . But σ (1) = ( i 1 , j 1 ) and σ ( ℓ ) = ( i 2 , j 2 ) , so this suffices for our claim.

Now consider intervals I i = [ a i , b i ] and I i +1 = [ a i +1 , b i +1 ] . It must be the case that b i = a i +1 ; otherwise, some smaller interval would fit in the range [ b i , a i +1 ] , and it would be placed before I i +1 in the interval ordering.

Since b i ∈ I i ∩ I i +1 , by definition, ub i +(1 -b i ) w ∈ R σ ( i ) ∩ R σ ( i +1) . In particular, ub i +(1 -b i ) w ∈ Q σ ( i ) ∩ Q σ ( i +1) ; by definition of E P , this means { σ ( i ) , σ ( i +1) } ∈ E P , as desired.

̸

Theorem 3.1. Let S be a clustering instance with | S | = n , and let R i = |P i | and R = R n . Let H t = ∣ ∣ { ( Q 1 , Q 2 ) ∈ P 2 t | Q 1 ∩ Q 2 = ∅} ∣ ∣ denote the total number of adjacencies between any two pieces of P i and H = H n . Then, the leaves of the execution tree on S can be computed in time ˜ O (∑ n i =1 ( H i + R i T M ) ( n -i +1) 2 ) , where T M is the time to compute the merge function.

Proof. Let T be the execution tree with respect to S , and let T t denote the vertices of T at depth t . From Theorem 2.2, for each node v = ( M , Q ) ∈ T with depth t , we can compute the children of v in time O ( n 2 t · LP ( d, E v ) + V v · n 2 t K ) , where V v is the number of children of v , and

̸

Now, observe

since H t +1 counts all adjacent pieces Q 1 , Q 2 in P t +1 ; each pair is counted at most once by some E v . Similarly, we have ∑ v ∈ T t +1 V v ≤ R t +1 , since R t +1 counts the total size of P t +1 . Note that n t +1 = ( n -t ) , since t merges have been executed by time t +1 , so n i = n -i +1 . Seidel's algorithm is a randomized algorithm that may be used for efficiently solving linear programs in low dimensions, the expected running time for solving an LP in d variables and m constraints is O ( d ! · E v ) (also holds with high probability, e.g. Corollary 2.1 of [Sei91]). There are also deterministic algorithms with the same (in fact slightly better) worst-case runtime bounds [Cha18]. Therefore, we can set LP ( d, E v ) = O ( d ! · E v ) . So that the total cost of computing P i is

J Further details and proofs from Section 4

We provide further details for Algorithm 5 in Appendix J.2, and other proofs from Section 4 are located in Appendix J.3.

J.1 Example dynamic programs for sequence alignment

We exhibit how two well-known sequence alignment formulations can be solved using dynamic programs which fit our model in Section 4. In Section J.1.1 we show a DP with two free parameters ( d = 2 ), and in Section J.1.2 we show another DP which has three free parameters ( d = 3 ).

J.1.1 Mismatches and spaces

Suppose we only have two features, mismatches and spaces . The alignment that minimizes the cost c may be obtained using a dynamic program in O ( mn ) time. The dynamic program is given by the following recurrence relation for the cost function which holds for any i, j > 0 , and for any ρ = ( ρ 1 , ρ 2 ) ,

̸

The base cases are C ( ϕ, ϕ, ρ ) = 0 , C ( ϕ, s 2 [: j ] , ρ ) = jρ 2 , = C ( s 1 [: i ] , ϕ, ρ ) = iρ 2 for i, j ∈ [ m ] × [ n ] . Here ϕ denotes the empty sequence. One can write down a similar recurrence for computing the optimal alignment τ ( s 1 , s 2 , ρ ) .

Wecan solve the non base-case subproblems ( s 1 [: i ] , s 2 [: j ]) in any non-decreasing order of i + j . Note that the number of cases V = 2 , and the maximum number of subproblems needed to compute a single DP update L = 3 ( L 1 = 1 , L 2 = 3 ). For a non base-case problem (i.e. i, j > 0 ) the cases are given by q ( s 1 [: i ] , s 2 [: j ]) = 1 if s 1 [ i ] = s 2 [ j ] , and q ( s 1 [: i ] , s 2 [: j ]) = 2 otherwise. The DP update in each case is a minimum of terms of the form c v,l ( ρ, ( s 1 [: i ] , s 2 [: j ])) = ρ · w v,l + σ v,l ( ρ, ( s 1 [: i ] , s 2 [: j ])) . For example if q ( s 1 [: i ] , s 2 [: j ]) = 2 , we have w 2 , 1 = ⟨ 1 , 0 ⟩ and σ 2 , 1 ( ρ, ( s 1 [: i ] , s 2 [: j ])) equals C ( s 1 [: i -1] , s 2 [: j -1] , ρ ) , i.e. the solution of previously solved subproblem ( s 1 [: i -1] , s 2 [: j -1]) , the index of this subproblem depends on l, v and index of ( s 1 [: i ] , s 2 [: j ]) but not on ρ itself.

J.1.2 Mismatches, spaces and gaps

Suppose we have three features, mismatches , spaces and gaps . Typically gaps (consecutive spaces) are penalized in addition to spaces in this model, i.e. the cost of a sequence of three consecutive gaps in an alignment ( . . . a ---b . . . , . . . a ′ p q r b ′ . . . ) would be 3 ρ 2 + ρ 3 where ρ 2 , ρ 3 are costs for spaces and gaps respectively [KKW10]. The alignment that minimizes the cost c may again be obtained using a dynamic program in O ( mn ) time. We will need a slight extension of our DP model from Section 4 to capture this. We have three subproblems corresponding to any problem in Π s 1 ,s 2 (as opposed to exactly one subproblem, which was sufficient for the example in J.1.1). We have a set of subproblems π ( s 1 , s 2 ) with | π ( s 1 , s 2 ) | ≤ 3 | Π s 1 ,s 2 | for which our model is applicable. For each ( s 1 [: i ] , s 2 [: j ]) we can compute the three costs (for any fixed ρ )

C s ( s 1 [: i ] , s 2 [: j ] , ρ ) is the cost of optimal alignment that ends with substitution of s 1 [ i ] with s 2 [ j ] .

C i ( s 1 [: i ] , s 2 [: j ] , ρ ) is the cost of optimal alignment that ends with insertion of s 2 [ j ] .

C d ( s 1 [: i ] , s 2 [: j ] , ρ ) is the cost of optimal alignment that ends with deletion of s 1 [ i ] .

The cost of the overall optimal alignment is simply C ( s 1 [: i ] , s 2 [: j ] , ρ ) = min { C s ( s 1 [: i ] , s 2 [: j ] , ρ ) , C i ( s 1 [: i ] , s 2 [: j ] , ρ ) , C d ( s 1 [: i ] , s 2 [: j ] , ρ ) } .

The dynamic program is given by the following recurrence relation for the cost function which holds for any i, j > 0 , and for any ρ = ( ρ 1 , ρ 2 , ρ 3 ) ,

Algorithm 5: COMPUTECOMPACTEXECUTIONDAG 1: Input : Execution DAG G e = ( V e , E e ) , problem instance ( s 1 , s 2 ) P 0 ←P 2: v 1 , . . . , v n ← topological ordering of vertices V e 3: for i = 1 to n do 4: Let S i be the set of nodes with incoming edges to v i 5: For v s ∈ S i , let P s denote the partition corresponding to v s 6: P i ← COMPUTEOVERLAYDP ( {P s | s ∈ S i } ) 7: for each p ∈ P i do 8: p ′ ← COMPUTESUBDIVISIONDP ( p, ( s 1 , s 2 )) 9: P i ←P i \ { p } ∪ p ′ 10: P i ← RESOLVEDEGENERACIESDP ( P i ) 11: return Partition P n

By having three subproblems for each ( s 1 [: i ] , s 2 [: j ]) and ordering the non base-case problems again in non-decreasing order of i + j , the DP updates again fit our model (1).

J.2 Details of the Execution-DAG based algorithm

We start with some well-known terminology from computational geometry.

Definition 6. A (convex) subdivision S of P ⊆ R d is a finite set of disjoint d -dimensional (convex) sets (called cells) whose union is P . The overlay S of subdivisions S 1 , . . . , S n is defined as all nonempty sets of the form ⋂ i ∈ [ n ] s i with s i ∈ S i . With slight abuse of terminology, we will refer to closures of cells also as cells.

The COMPUTEOVERLAY procedure takes a set of partitions, which are convex polytopic subdivisions of R d , and computes their overlay. We will represent a convex polytopic subdivision as a list of cells, each represented as a list of bounding hyperplanes. Now to compute the overlay of subdivisions P 1 , . . . , P L , with lists of cells C 1 , . . . , C L respectively, we define |C 1 | × · · · × |C L | sets of hyperplanes H j 1 ,...,j L = { ⋃ l ∈ [ L ] H ( c ( l ) j l ) } , where c ( l ) j l is the j l -th cell of P l and H ( c ) denotes the hyperplanes bounding cell c . We compute the cells of the overlay by applying Clarkson's algorithm [Cla94] to each H j 1 ,...,j L . We have the following guarantee about the running time of Algorithm 6.

Algorithm 6: COMPUTEOVERLAYDP

Input : Convex polytopic subdivisions P 1 , . . . , P L of R , represented as lists C j of hyperplanes

C ← non-empty lists of hyperplanes in H j 1 ,...,j L for j l ∈ C l return Partition represented by C

Lemma J.1. Let R i,j denote the number of pieces in P [ i ][ j ] , and ˜ R = max i ≤ m,j ≤ n P [ i ][ j ] . There is an implementation of the COMPUTEOVERLAYDP routine in Algorithm 5 which computes the overlay of L convex polytopic subdivisions in time O ( L ˜ R L +1 · LP( d, ˜ R L +1)) , which is O ( d ! L ˜ R 2 L +1 ) using algorithms for solving low-dimensional LPs [Cha18].

Proof. Consider Algorithm 6. We apply the Clarkson's algorithm at most ˜ R L times, once corresponding to each L -tuple of cells from the L subdivisions. Each iteration corresponding to cell c in the output overlay O (corresponding to C ) has a set of at most L ˜ R hyperplanes and yields at most R c non-redundant hyperplanes. By Theorem 2.1, each iteration takes time O ( L ˜ R · LP( d, R c +1)) , where LP( d, R c +1) is bounded by O ( d ! R c ) for the algorithm of [Cha18]. Note that ∑ c R c corresponds to

Algorithm 7: COMPUTESUBDIVISIONDP

̸

the total number of edges in the cell adjacency graph of O , which is bounded by ˜ R 2 L . Further note that R c ≤ ˜ R L for each c ∈ C and |C| ≤ ˜ R L to get a runtime bound of O ( L ˜ R L +1 · LP( d, ˜ R L +1)) .

We now consider an implementation for the COMPUTESUBDIVISIONDP subroutine. The algorithm computes the hyperplanes across which the subproblem used for computing the optimal alignment changes in the recurrence relation (1) by adapting Algorithm 1. We restate the algorithm in the context of sequence alignment as Algorithm 7.

Lemma J.2. Let R i,j denote the number of pieces in P [ i ][ j ] , and ˜ R = max i ≤ m,j ≤ n R i,j . There is an implementation of COMPUTESUBDIVISIONDP routine in Algorithm 5 with running time at most O (( L 2 d +2 + L 2 d ˜ R L ) · LP( d, L 2 + ˜ R L )) for each outer loop of Algorithm 5. If the algorithm of [Cha18] is used to solve the LP , this is at most O ( d ! L 2 d +2 ˜ R 2 L ) .

Proof. Consider Algorithm 7. For any piece p in the overlay, all the required subproblems have a fixed optimal alignment, and we can find the subdivision of the piece by adapting Algorithm 1 (using O ( L 2 + ˜ R L ) hyperplanes corresponding to subproblems and piece boundaries). The number of pieces in the subdivision is at most L 2 d since we have at most L 2 hyperplanes intersecting the piece, so we need O ( L 2 d +2 + L 2 d ˜ R L ) time to list all the pieces C p . The time needed to run Clarkson's algorithm is upper bounded by O ( ∑ c ∈C p ( L 2 + ˜ R L ) · LP( d, R c + 1)) = O ( ∑ c ∈C p ( L 2 + ˜ R L ) · LP( d, L 2 + ˜ R L )) = O (( L 2 d +2 + L 2 d ˜ R L ) · LP( d, L 2 + ˜ R L )) . Using [Cha18] to solve the LP, this is at most O ( d ! ˜ R 2 L L 2 d +4 ) .

Lemma J.3. Let R i,j denote the number of pieces in P [ i ][ j ] , and ˜ R = max i ≤ m,j ≤ n R i,j . There is an implementation of RESOLVEDEGENERACIESDP routine in Algorithm 5 with running time at most O ( ˜ R 2 L L 4 d ) for each outer loop of Algorithm 5.

Proof. The RESOLVEDEGENERACIESDP is computed by a simple BFS over the cell adjacency graph G c = ( V c , E c ) (i.e. the graph with polytopic cells as nodes and edges between polytopes sharing facets). We need to find (maximal) components of a subgraph of the cell adjacency graph where each node in the same component has the same optimal alignment. This is achieved by a simple BFS in O ( | V c | + | E c | ) time. Indeed, by labeling each polytope with the corresponding optimal alignment, we can compute the components of the subgraph of G c with edges restricted

to nodes joining the same optimal alignment. Note that the resulting polytopic subdivision after the merge is still a convex subdivision using arguments in Lemma J.5, but applied to appropriate sequence alignment subproblem. As noted in the proof of Lemma J.2, we have | V c | ≤ L 2 d ˜ R L since the number of cells within each piece p is at most L 2 d and there are at most ˜ R L pieces in the overlay. Since | E c | ≤ | V c | 2 , we have an implementation of RESOLVEDEGENERACIESDP in time O (( L 2 d ˜ R L ) 2 ) = O ( ˜ R 2 L L 4 d ) .

Finally we can put all the above together to give a proof of Theorem 4.1.

Proof of Theorem 4.1. The proof follows by combining Lemma J.1, Lemma J.2 and Lemma J.3. Note that in the execution DAG, we have | V e | ≤ | E e | = O ( T DP ) . Further, we invoke COMPUTEOVERLAYDP and RESOLVEDEGENERACIESDP | V e | times across all iterations and COMPUTESUBDIVISIONDP across the | V e | outer loops.

J.3 Additional Proofs

The following results closely follow and extend the corresponding results from [BDD + 21]. Specifically, we generalize to the case of two sequences of unequal length, and provide sharper bounds on the number of distinct alignments and boundary functions in the piecewise decomposition (even in the special case of equal lengths). We first have a bound on the total number of distinct alignments. Lemma J.4. For a fixed pair of sequences s 1 , s 2 ∈ Σ m × Σ n , with m ≤ n , there are at most m ( m + n ) m distinct alignments.

̸

Proof. For any alignment ( t 1 , t 2 ) , by definition, we have | t 1 | = | t 2 | and for all i ∈ [ | t 1 | ] , if t 1 [ i ] = -, then t 2 [ i ] = -and vice versa. This implies that t 1 has exactly n -m more gaps than t 2 . To prove the upper bound, we count the number of alignments ( t 1 , t 2 ) where t 2 has exactly i gaps for i ∈ [ m ] . There are ( n + i i ) choices for placing the gap in t 2 . Given a fixed t 2 with i gaps, there are ( n n -m + i ) choices for placing the gap in t 1 . Thus, there are at most ( n + i i )( n n -m + i ) = ( n + i )! i !( m -i )!( n -m + i )! ≤ ( m + n ) m possibilities since i ≤ m . Summing over all i , we have at most m ( m + n ) m alignments of s 1 , s 2 .

This implies that the dual class functions are piecewise-structured in the following sense.

Lemma J.5. Let U be the set of functions { u ρ : ( s 1 , s 2 ) ↦→ u ( s 1 , s 2 , ρ ) | ρ ∈ R d } that map sequence pairs s 1 , s 2 ∈ Σ m × Σ n to R by computing the optimal alignment cost C ( s 1 , s 2 , ρ ) for a set of features ( l i ( · )) i ∈ [ d ] . The dual class U ∗ is ( F , G , m 2 ( m + n ) 2 m ) -piecewise decomposable, where F = { f c : U → R | c ∈ R } consists of constant functions f c : u ρ ↦→ c and G = { g w : U → { 0 , 1 } | w ∈ R d } consists of halfspace indicator functions g w : u ρ ↦→ I { w · ρ < 0 } .

Proof. Fix a pair of sequences s 1 and s 2 . Let τ be the set of optimal alignments as we range over all parameter vectors ρ ∈ R d . By Lemma J.4, we have | τ | ≤ m ( m + n ) m . For any alignment ( t 1 , t 2 ) ∈ τ , the algorithm A ρ will return ( t 1 , t 2 ) if and only if

for all ( t ′ 1 , t ′ 2 ) ∈ τ \ { ( t 1 , t 2 ) } . Therefore, there is a set H of at most ( | τ | 2 ) ≤ m 2 ( m + n ) 2 m hyperplanes such that across all parameter vectors ρ in a single connected component of R d \ H , the output of the algorithm A ρ on ( s 1 , s 2 ) is fixed. This means that for any connected component R of R d \ H , there exists a real value c R such that u ρ ( s 1 , s 2 ) = c R for all ρ ∈ R d . By definition of the dual, u ∗ s 1 ,s 2 ( u ρ ) = u ρ ( s 1 , s 2 ) = c R . For each hyperplane h ∈ H , let g ( h ) ∈ G denote the corresponding halfspace. Order these k = ( | τ | 2 ) functions arbitrarily as g 1 , . . . , g k . For a given connected component R of R d \ H , let b R ∈ { 0 , 1 } k be the corresponding sign pattern. Define the function f ( b R ) = f c R and for b not corresponding to any R , f ( b ) = f 0 . Thus, for each ρ ∈ R d ,

For the special case of d = 2 , we have an algorithm which runs in time O ( RT DP ) , where R is the number of pieces in P [ m ][ n ] which improves on the prior result O ( R 2 + RT DP ) for two-parameter sequence alignment problems. The algorithm employs the ray search technique of [Meg78] (also employed by [GBN94] but for more general sequence alignment problems) and enjoys the following runtime guarantee.

Theorem J.6. For the global sequence alignment problem with d = 2 , for any problem instance ( s 1 , s 2 ) , there is an algorithm to compute the pieces for the dual class function in O ( RT DP ) time, where T DP is the time complexity of computing the optimal alignment for a fixed parameter ρ ∈ R 2 , and R is the number of pieces of u ( s 1 ,s 2 ) ( · ) .

Proof. We note that for any alignment ( t 1 , t 2 ) , the boundary functions for the piece where ( t 1 , t 2 ) is an optimal alignment are straight lines through the origin of the form

for some alignment ( t ′ 1 , t ′ 2 ) different from ( t 1 , t 2 ) . The intersection of these halfplanes is either the empty set or the region between two straight lines through the origin. The output subdivision therefore only consists of the axes and straight lines through the origin in the positive orthant.

We will present an algorithm using the ray search technique of [Meg78]. The algorithm computes the optimal alignment ( t 1 , t 2 ) , ( t ′ 1 , t ′ 2 ) at points ρ = (0 , 1) and ρ = (1 , 0) . If the alignments are identical, we conclude that ( t 1 , t 2 ) is the optimal alignment everywhere. Otherwise, we find the optimal alignment ( t ′′ 1 , t ′′ 2 ) for the intersection of line L joining ρ = (0 , 1) and ρ = (1 , 0) , with the line L ′ given by

If ( t ′′ 1 , t ′′ 2 ) = ( t 1 , t 2 ) or ( t ′′ 1 , t ′′ 2 ) = ( t ′ 1 , t ′ 2 ) , we have exactly 2 optimal alignments and the piece boundaries are given by L ′ and the axes. Otherwise we repeat the above process for alignment pairs ( t ′′ 1 , t ′′ 2 ) , ( t ′ 1 , t ′ 2 ) and ( t ′′ 1 , t ′′ 2 ) , ( t 1 , t 2 ) . Notice we need to compute at most R +1 dynamic programs to compute all the pieces, giving the desired time bound.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We clear state the claims and make sure they accurately reflect our contributions.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We clearly state that our results lead to faster algorithms when the output-size is small, and worst-case improved runtimes are left for future work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Complete proofs and details are in the Appendix and appropriately referenced from the main body.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We follow the Ethics Guidelines.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: We view our work as theoretical, with techniques leading to faster machine learning algorithms.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of learning optimal parameters for data-driven algorithm design. A characteristic of the problem is that the dual loss function, which measures the performance of an algorithm as a function of parameters, is discontinuous. Nevertheless, the dual loss is typically piecewise structured (constant, linear, etc.) with linear boundaries. Thus, roughly speaking, the problem of finding optimal parameters reduces to exploring polytopic cells partitioned by boundary hyperplanes. \n\nThe main contribution is a cell enumeration algorithm that runs in output-polynomial time. The algorithm can be seen as a breadth-first search on a cell adjacency graph, where the enumeration of neighbors is done in an output-sensitive manner based on Clarkson's algorithm. The resulting output-sensitive complexity can be significantly better than the worst-case, as demonstrated in Example 1.\n\nThe authors then instantiate the ERM method based on the cell enumeration for linkage-based clustering and DP-based sequence alignment. The applications involve designing execution graphs, which originate from the execution tree of [BDL20]. Combining appropriate problem-specific execution graphs with the cell enumeration leads to the improved time complexity of ERM in several data-driven algorithm design problems, as in Table 1.\n\nSTRENGTHS:\n1. The paper addresses the important problem of optimizing algorithm parameters in data-driven algorithm design. \n2. The theoretical results given in Table 1 appear strong compared with previous ones. \n3. The output-sensitive cell enumeration might be of independent interest in the context of computational geometry.\n\nWEAKNESSES:\n1. The paper would have been more appealing if implementations of the proposed methods and experimental results were provided.\n2. The paper is somewhat dense and it is not easy to follow the technical details.\n\nQUESTIONS:\n1. I would like to know more intuition about how AugmentedClarkson differs from the original Clarkson and why it is important. \n2. While the paper focuses on linear boundaries, some studies on data-driven algorithm design consider parameter spaces partitioned by polynomials:\n\nhttps://proceedings.neurips.cc/paper_files/paper/2022/hash/db2cbf43a349bc866111e791b58c7bf4-Abstract-Conference.html\n\nhttps://proceedings.mlr.press/v178/bartlett22a.html\n\nhttps://proceedings.mlr.press/v206/sakaue23a.html\n\nIs there a possibility of applying similar enumeration ideas to such situations?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn data-driven algorithm design, we are given a collection of problem instances sampled from an unknown distribution, and a family of algorithms for solving the problem, typically parameterized by a real-valued multivariate parameter. The goal is to find a setting of parameters such that the performance of the algorithm they parameterize is close to optimal among the family of algorithms, in expectation over the distribution of instances. Most prior work is firstly focused on the generalization aspect, i.e., on showing that a small number of samples suffices (in the sample complexity sense) for finding approximately optimal parameters for the distribution (these are the ERM parameters for the given sample of instance), and thus the family of algorithms is \"learnable\". It then (sometimes) proceeds to develop an efficient algorithm for finding those ERM parameters, based on the structure used to prove the generalization bound.\n\nThis paper focuses more systematically on the ERM efficiency aspect. To this end, it starts with a common theme in many prior works on data-driven algorithms, that had been abstracted out and formulated in a generalized form in Balcan et al. (STOC 2021): for any fixed problem instance, the function that maps a setting of parameters to utility (of invoking their associated algorithm on that instance) admits a simple piecewise structure. Say, there is a small number of \"simple\" boundary functions (say, linear thresholds) that induce a partition of the parameter space R^d such that the utility function restricted to each piece is \"simple\" (say, constant). This is helpful in bounding the VC dimension of the utility functions and thus proving generalization bounds, and also potentially for navigating the parameter space efficiently to find the ERM parameters.\n\nThe novelty in this paper is an attempt to give a more systematic recipe for the second part (navigating the piecewise structure for efficient ERM), with two main advantages -- (1) creating a unified framework that takes care of some parts of the ERM procedure in a general way, thus restricting the portion that needs to be figured out per each problem individually, and (2) obtaining algorithms whose running time depends on the actual number of pieces in the given instances (\"output-sensitive\") rather than worst-case number of pieces. The per-problem part to figure out is a subroutine that, given a problem instance and parameter setting p, returns a list of candidate boundary functions for the piece that contains p, and this subroutine depends on the specific problem in question. The unified part of the framework uses this subroutine to search through the pieces in an \"output-sensitive\" running time.\n\n__Post-rebuttal__: I appreciate the authors' elaborations on the technical content, and the conceptual aspect of the paper. I have raised my score to support acceptance.\n\nSTRENGTHS:\nThe strength of this paper is that the matter of efficient ERM in data-driven algorithms indeed merits its own systematic study rather than being left as an afterthought of the generalization bounds.\n\nWEAKNESSES:\nThe main weakness is that the end result isn't very strong: the framework is restricted to linear boundary functions and (more disconcertingly) to a constant number of parameters, and for the most part does not yield improved running times in the worst case, but a different notion of efficiency (output sensitivity). It tends more to systematically organizing ideas that have appeared in the literature in one form or another and less to introducing new algorithmic insights or techniques. I also feel that the presentation and writing could be too opaque for a wide readership like that of NeurIPS.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

comments: SUMMARY:
The paper explores computational aspects of implementing ERM in data-driven algorithm design. 

The paper contributes an efficient algorithm to ennumerate cells induced by a collection of hyperplanes. The paper then shows how to utilize this as a subprocedure to solve ERM problems for algorithm design, focusing on linkage-based clustering, sequence alignment, and two-part tariff pricing.

STRENGTHS:
One of the main interesting things I find about this paper is that the runtime of the ERM implementations is instance dependent, and specifically depends on the number of sum dual class loss function pieces. The paper comments that their runtime bounds imply improvements over prior work in the worst-case R but also can be faster for "typical" R.

The paper is well-written and easy to follow. The paper discusses relevant background and related work.

WEAKNESSES:
To what extent is the approach generalizable to other data-driven algorithm design problems? Is there a generic principle or a general characterization of the problems for which this approach can be utilized?

QUESTIONS:
See questions above.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12