ROSA: An Optimization Algorithm for Multi-Modal Derivative-Free Functions in High Dimensions

Authors: ilija  ilievski
wenyu  wang
christine a shoemaker
Venue: NeurIPS_cc_2024_Workshop_BDU
Published: 2024-08-28 12:00:35

Abstract: Derivative-free, multi-modal optimization problems in high dimensions are ubiquitous in science and engineering. Obtaining satisfactory solutions of high dimensional optimization problems requires many objective function evaluations. At the same time, commonly used Bayesian optimization methods are typically computationally expensive which limits the high-dimensional function approximation accuracy, leading to sub-optimal solutions. We propose, ROSA, a novel optimization algorithm based on well-known optimization techniques such as randomized optimization, simulated annealing, and surrogate optimization. ROSA is several orders of magnitude computationally more efficient than leading scalable Bayesian optimization methods, while also obtaining comparable or better solutions with as many as 4 times less objective function evaluations. We compare ROSA with a diverse set of methods on many synthetic high-dimensional benchmark functions and on real-world problems.

Main paper: 
 introduction: 
 Many important problems in science and engineering require the optimization of multi-modal derivative-free functions, often resource-limited to only several thousand function evaluations. For example, calibration of water simulation models [1, 2] , climate simulation models [3, 4] , aircraft wing design [5] [6] [7] , vehicle design optimization [8] , and machine learning hyperparameter optimization [9] [10] [11] . With the proliferation of data and computing power, the functions of interest have become increasingly higher-dimensional. Thus, we focus on optimizing functions in 50 to 500 dimensions and with an evaluation budget limited at most ten times the dimension.Applying optimization methods to high-dimensional functions is a difficult problem due to the curse of dimensionality, that is, the volume of the sampling space and the required function evaluations for an accurate function approximation grow exponentially with the number of dimensions. One line of research, led by REMBO, attempts to avoid the problem by embedding the high-dimensional space in a lower-dimensional space where the optimization is performed [12] . A closely related method is SASSBO which optimizes only some of the dimensions [13] . However, REMBO's assumption that the high-dimensional function can be accurately represented in low dimensions is often not valid in practice, and SASSBO's high computational requirements limit its application to problems with evaluation budgets of at most hundred function evaluations. Another line of research is the scalable Bayesian optimization methods, which aim to fit a sparse Gaussian process models [14] , or fit a Gaussian process models only on subset of evaluated points around a trust region [11] . But, sparsifying the solutions space or using only subset of the evaluated points often results in suboptimal solutions. Finally, the inherit computational expense of fitting and tuning Gaussian process models, commonly limits the application of these models to problems with evaluation budgets of at most thousand iterations.We propose a fundamentally different optimization algorithm that combines metaheuristic search method with a polyharmonic spline surrogate. Polyharmonic splines are fitted in closed-form and require no tuning, in addition to being accurate approximates of high-dimensional functions [15] . Our metaheuristic search method merges ideas from randomized optimization and simulated annealing to achieve optimal exploration-exploitation trade-off and efficiently avoid local optima. To summarize our contributions are: 1. We develop ROSA, a novel optimization algorithm for efficient multi-modal optimization of high-dimensional derivative-free functions with an evaluation budget of several thousand evaluations. 2 . We open-source our modern and efficient implementation of ROSA, able to run on CPUs and GPUs. We hope our code will be used by researchers and practitioners in a wide range of applications in science and engineering. 
method: 
 We develop an optimization method for a multi-modal derivative-free function over a hypercube defined by a and b in d dimensions. That isx * = argmin x † ∈[a,b] d f (x † ), notation info in Sec. A.1.The proposed algorithm, ROSA, is based on three fundamental ideas in multi-modal, derivative-free optimization: (i) Randomized Optimization, where one randomly samples points by adding a random vector to the current best point. However, instead of adding random perturbations to all dimensions, ROSA only changes a small and decreasing number of dimensions at each iteration. (ii) Simulated Annealing, where one allows accepting a worse neighbour as the current best point, but with a probability that decreases with the number of iterations. (iii) Surrogate optimization, where one fits a surrogate such as Gaussian Process and uses an acquisition function such as Expected Improvement to decide where to evaluate f next. Instead, ROSA uses computationally efficient surrogate such as polyharmonic splines to only rank the neighbours and evaluate on f the top-ranked neighbours.The algorithm takes as inputs an objective function f we wish to minimize and an evaluation budget n max . ROSA starts by evaluating the objective function at n 0 points, sampled uniformly at random from the function input space. The point that gives the lowest objective function value is set as the current best point, x ′ , from where the optimization iterations start.At each iteration, ROSA selects a set of dimensions to be perturbed at random, with a probability of being selected p φ = φ(n/n max ) . The probability of each dimension being selected is a decreasing function (φ) of the amount of the currently used evaluation budget (Sec. 2) and it is independent from the selection of the other dimensions. Formally, we define the set of selected dimensions: A = {k : υ < φ(n/n max ) | k ∈ I, υ ∼ U (0, 1)}, where I = {1, . . . , d} and U (0, 1) is Uniform distribution over (0, 1) ∈ R. In case, A = ∅, then A = {j} where j is a random sample from I, ensuring at least one dimension is always selected. The probability of a dimension being selected is also independent across iterations, resulting in different dimensions being selected at each iteration.We create a set C of q neighbouring points, with q ≫ d, i.e. C = {x i | i ∈ {1, . . . , q}}, by adding random perturbations to the selected dimensions of the current best point x ′ (Sec. 2). Selecting the dimensions is independent across the xi points, so each point may have different dimensions selected. We then evaluate each xi point in C on a polyharmonic spline surrogate and select the point with lowest surrogate value as the next evaluation point x • .The point x • is evaluated on the objective function and it is accepted as the current best point with probability p α = g(f (x • ), f (x ′ ), n), given by g, the acceptance probability function (Sec. 2). After exhausting the evaluation budget we return the current best point and its objective value. Note that ROSA is easily parallelizable by choosing the m lowest surrogate value points instead of a single point, given that one is able to run the objective function efficiently in parallel.Next, we describe the algorithm's components, and in the appendix, we define the algorithm in pseudo code (Alg. D n0 = {(x 1 , f (x 1 )), . . . , (x n0 , f (x n0 )) | x i ∼ U (a, b)}. (b) Define x ′ as (x ′ , f (x ′ )) ∈ D n0 such that f (x ′ ) ≤ f (x i ) ∀ (x i , f (x i )) ∈ D n0 .(c) Define y ′ as the current lowest found objective value, y ′ = f (x ′ ). (d) Define n as the number of currently evaluated points, n = n 0 .Step 2: Main optimization loop while n ≤ n max do (a) Fit a polyharmonic spline surrogate H on D n (Equation 2). (b) Create a set C of q neighbouring points around x ′ , C = NG(x ′ , n/n max ) (Algorithm 2). (c) Define the set S as a set of pairs of neighbour point and its surrogate value,S = {(x i , H(x i )) | xi ∈ C}. (d) Denote x • as the best point in S, i.e. H(x • ) ≤ H(x i ) ∀ (x i , H(x i )) ∈ S. (e) Perform a function evaluation y • = f (x • ) and update D n+1 = {(x • , y • )} ∪ D n . (f) With probability p α = g(y • , y ′ , n) (Section 2) accept the new solution x • and set x ′ = x • and y ′ = y • . (g) n = n + 1.end Return: The lowest objective value and the corresponding point, y ′ and x ′ . Polyharmonic Spline Surrogate A polyharmonic spline surrogate is a linear combination of radial basis functions and a polynomial tail. The polyharmonic kernels are scale invariant, so they do not require tuning of hyperparameters such as the length-scale in the case of the squared exponential and Matérn kernels, the most common Gaussian Process kernels. The polyharmonic spline surrogate is defined as: ϕ(r) = r k if k = 2n -1 for n ∈ N and as ϕ(r) = r k ln(r) if k = 2n for n ∈ N, where r = ∥x -c i ∥ 2 and c i are the centres, i.e., the evaluated points used to fit the surrogate. ROSA incorporates a cubic spline, ϕ(r) = r 3 , which is conditionally positive definite kernel of order 2. So, we include a linear polynomial tail to ensure the stability of the solutions of the system of equations used to fit the surrogate parameters. Formally, given n number of d-dimensional vectors, x 1:n , we construct an polyharmonic spline interpolation model with:EQUATIONThe model parameters λ i:n , b i:d are determined by solving the following linear system of equations:EQUATIONHere Φ ∈ R n×n is defined asΦ i,j = (||x i -x j || 2 ) 3 , i, j = 1, . . . , n, 0 ∈ R (d+1)×(d+1) , P ∈ R n×(d+1) has its i-th row defined as [1, x ⊤ i ], λ = [λ 1 , . . . , λ n ] ⊤ , F = [f (x 1 ), . . . , f (x n )] ⊤ ,and η is a regularization constant. When n < d + 1, we fit the model parameters via least squares.Acceptance Probability Function The acceptance probability function promotes exploration by accepting with some probability a point with a worse objective function value as the current best point [16] . We define the acceptance function asp α := g(f (x • ), f (x ′ ), n) = min(exp[ -(f (x • )-f (x ′ ))], 1), where f (x • ) is the new, possibly worse, objective function value, and f (x ′ ) is the current best objective value. T n is the cooling temperature with geometric schedule, i.e., T n = α n T 0 , where T 0 and α are set to common values [17] and fixed throughout our experiments. We accept x • as the current best point with probability p α .We define the function φ(n/n max ) as a simple decreasing step-function that goes from 1e-1 to 1e-6 (see code for the specific implementation). In Sec. B.5, we analyse our choice of probabilities for selecting dimensions by comparing the default ROSA, with ROSA-Const. 20D where we perturb min(20, d) dimensions, ROSA-Const. 10% where we use constant function φ(•) = 1e-1, ROSA-Increasing, where we increase the probabilities from 0.10 to 1.00, ROSA-Reversed Default, where we increase the probabilities from 1e-6 to 1e-1. In Figure B.3 we observe that the performance of ROSA is similar as long as the probabilities are decreasing.Perturbation Distribution To the selected dimensions of the current best point, we add random samples from truncated normal distribution (Eq. 5) with mean 0 and perturbation radius fixed to one sixth of the objective function domain range, i.e., σ = (b -a)/6. It is important that the samples are coming from truncated normal distribution, instead for example from uniform distribution [11] , as otherwise the distribution of the samples gets heavily skewed towards the domain bounds (Figure B.2). We recommend that σ is set to one sixth of the bounded range such that 99.7% of the hypercube space is reachable from the centre point. To empirically justify our choice, we benchmark ROSA with multiple σ values and show that ROSA is fairly robust to different values of σ, with the default version only having a small advantage on some functions (Figure B.4) . The pseudo code is in Alg 2.Inputs:1. Point x ′ around which to generate neighbouring points.2. Percentage of currently spent evaluation budget η = n/n max . Configuration:1. Bounding hypercube [a, b] d ⊂ R d of permitted values of x.2. A decreasing step-function that maps a percentage of spent budget to a probability of perturbing a dimension, p φ = φ(η) (Section 2). 3. Number of neighbour points q. Algorithm: Create a set C of q neighbour points xi where for each point:1. Define a set of selected dimensions: A = {k : υ < φ(η) | k ∈ I, υ ∼ U (0, 1)}, where I = {1, . . . , d} and U (0, 1) is Uniform distribution over (0, 1) ∈ R.z = [z (k) ] ⊤ , for k ∈ I, z (k) = τ ∼ T (0, σ, α (k) , β (k) ), if k ∈ A otherwise, z (k) = 0. Here T (0, σ, α (k) , β (k)) is truncated Normal distribution with mean 0 and standard deviation σ bounded by α = a -x ′ and β = b -x ′ . 3. Set xi = x ′ + z. Return: The set of C = {x i | i ∈ {1, . . . , q}} neighbour points. 
experiments: 
 ROSA combines ideas from simulated annealing, randomized optimization, and surrogate optimization. Accordingly we compare ROSA with baselines that employ ideas from simulated annealing, Dual Annealing [18] , randomized optimization, PSO [19] , and surrogate optimization (lq-CMA-ES [20] , TuRBO [11] , DYCORS [21] ). We also compare ROSA to an evolutionary algorithm (CMA-ES) as it have been shown to be competitive in optimization of black-box functions with large evaluation budgets [22] . ROSA is most similar to TuRBO and DYCORS, the three methods add random perturbations to the current best, so we list their similarities and differences in Table B .1.We compare the methods on three well-known synthetic functions, Ackley [23] -a function most surrogates can approximate well, Michalewicz [24] -a function with flat surface and sharp ridges where surrogates are often not helpful, and Rastrigin [25] -function riddled with many good local minima and thus algorithms often get stuck in a local minimum. 2D visualizations of the three functions are shown in Figure B.1. We vary the number of dimensions from 60 to 200 to evaluate how the algorithms' performance scale with the number of dimensions. As an additional challenging benchmark suit we use the BBOB-largescale suite from COCO [26] .As real-world problems we use the well-known benchmark problem of optimizing vehicle design introduced by General Motors at MOPTA08 [8] (in 124D), and optimizing 496 portfolio weights for maximizing volatility adjusted returns while minimizing max drawdown, a common finance application of non-convex optimization methods. We perform parallel optimization with m = 50 on problems with a budget of more than 2,000 evaluations, such as BBOB-320D, BBOB-640D, and Portfolio Optimization, on the rest we perform serial optimization (m = 1). The outperformance of ROSA is also confirmed on the ten multimodal functions in high dimensions of the BBOB benchmark suite. In 160D ROSA performance is matched by DYCORS after 75% of the budget (Figure C.8). However, DYCORS computational requirements are too great to be run on the 320D and 640D problems (Figure C.9). Which once again confirms the need of a method with not only good optimization performance but also that is scalable and computationally efficient to be able to optimize high-dimensional problems and with many function iterations.We perform sensitivity analysis of ROSA's crucial algorithm component, the neighbouring points generation. We justify our choice of probability of perturbing a dimension with the results in  
discussion: 
 With this paper, we aim to address the problem of optimization of multi-modal, derivative-free functions in high dimensions by developing an optimization method that is computationally efficient, while achieving excellent optimization performance on 15 representative functions in 9 distinct dimensions. We hope ROSA will serve as an alternative to Bayesian optimization methods when they are not suitable for the task. ROSA's modular algorithm design is based on fundamental optimization ideas developed and time-tested over decades of research. The modular design also allows for easy customization to specific problem types and further development of the algorithm. As an example, future work involves adapting the exploration vs exploitation trade-off to unseen problems by dynamically adjusting the probability of perturbing a dimension and the acceptance probability.A. Uniform distribution between a and b T (0, σ, a, b) truncated Normal with mean 0 and standard deviation σ bounded by a and bAfter n ≥ n 0 times of objective evaluation, consider the current best d-dimensional neighbour vector x • n as a random vector. Define ξ n = {x • 1 , . . . , x • n } to store all the evaluated points, i.e., the entire algorithmic running history after n iterations.Finally, let Q(x, δ) denote the open ball centred at x with radius δ, i.e., Q(x, δ) = {x † | ||x -x † || 2 < δ}, and Σ(ξ n ) denote the σ-field generated by the random vectors in ξ n .Theorem 1 Let f be a real-valued function defined on B ⊆ R d and suppose that x * is the unique global minimizer of f on B in the sense that f (x * ) = inf x∈B f (x) > -∞. Following the pseudo code in Algorithm 1, ROSA iteratively generates random vectors {x n } n≥1 , and maintains a sequence of random vectors [x ′ n ] as follows:x ′ n = x • n with probability p α = g(f (x • ), f (x ′ n ), n) (Section 2) otherwise x ′ n = x ′ n-1 . Then x ′ n converges to x * almost surely.Proof. Fix ε > 0 and n ≥ n 0 + 1. Assume that there exists δ(ε) > 0 such that |f (x) -f (x * )| < ε whenever ||x -x * || 2 < δ(ε). Hence, the event [x n ∈ B : |f (x n ) -f (x * )| < ε] ⊇ [x n ∈ B : ||x n -x * || 2 < δ(ε)],and so,EQUATIONIn ROSA, the neighbour x • n is generated by adding independent truncated normal samples to the selected dimensions of x ′ n-1 . Let E n,j denotes the event that the j-th dimension of x ′ n-1 is selected for perturbation and P (E n,j ) = p φn = φ(n/n max ) > 0 (see Section. 2). Therefore, the candidate x • n has the following conditional density function given Σ(ξ n-1 ) for each n > n 0 ,EQUATION)where T is a truncated normal density function in the following form,T (x • n,j = x j | Σ(ξ n-1 ); σ j , a j , b j ) = 1 2πσ 2 j exp -(x • n,j -x ′ n-1,j ) 2 2σ 2 j Φ( bj -x ′ n-1,j σj ) -Φ( aj -x ′ n-1,j σj ) (5)with Φ being the cumulative function of a standard normal distribution and the rest defined in Section A.1. Bounding (5) from bellow:EQUATIONThen, ( 4) is also bounded with:EQUATIONFinally, combining ( 7) with (3) yields,EQUATION)where ψ is the Lebesgue measure on R d and p φn denotes the probability used to select dimensions. Thus, from (8), we haveP [x • n ∈ B : |f (x • n ) -f (x * )| < ε | σ(ξ n-1 )] is lower bounded by L(ε). By following the same argument as in the proof of theorem in p.40 of [27] , we show that x ′ n converges to x * almost surely. ■We compare ROSA with a diverse kind of methods, namely:• Sobol, a quasi-random baseline represented by a scrambled Sobol sequence [28] ; We use PyTorch's implementation of scrambled Sobol, available at link.• PSO, a particle swarm optimization method [19] ; We follow the instructions at link to install and run pyswarms on optimizing a function with bounds. We set options = {c1 : 0.5, c2 : 0.3, w : 0.9}, n_particles = 10, and use the "GlobalBestPSO" as optimizer with the rest settings set to default.• DA, Dual Annealing, a method that combines classical simulated annealing with fast simulated annealing, coupled with a local search strategy [18] . We use the default values as implemented in SciPy [29] .• CMA-ES, a model-free evolutionary optimization method [22] ; We follow the instructions at https://github.com/CMA-ES/pycma to install CMA-ES and run CMA with the default settings. As an initial point x 0 we set the middle point of the domain, i.e. x 0 = (a + b)/2, where a and b are the upper and lower bounds respectively. Following CMA-ES authors advice, we set σ 0 to quarter of the objective function domain range.• lq-CMA-ES, a surrogate-assisted evolutionary optimization method [20] ; We follow the instructions at http://cma.gforge.inria.fr/cmaes_sourcecode_page.html# practical to install and run lq-CMA-ES with the default settings. As an initial point x 0 we tried setting the middle point of the domain and using a random point from n 0 Latin Hypercube samples, the random LHS sample version worked better so we did all experiments with x 0 set to a random LHS point. Following lq-CMA-ES authors advice on the above-mentioned URL, we set σ 0 to half of the function domain range, to make sure every space is within 2σ 0 from every possible x 0 .• TuRBO, a scalable Bayesian optimization method for high-dimensional surrogate optimization [11] ; We follow the instructions at https://github.com/uber-research/TuRBO/ to install TuRBO-1 and run TuRBO-1 with the default settings. We run only TuRBO-1 as our initial experiments with running TuRBO-m did not show any improvements over TuRBO-1 but significantly increased the required computational resources and wall-clock run time. Our initial experimental results are also supported by most experiments in [11] .• DYCORS, surrogate optimization method with dynamic coordinate search [21] . We use the implementation at https://github.com/dme65/pySOT to install pySOT and run DYCORS with the default settings [30, 31] . Due to pySOT's reliance on older software versions, we were not able to run DYCORS on MOPTA. DYCORS were also not run on the 496D portfolio optimization problem and 320D & 640D BBOB problems as it required unfeasible large amount of RAM memory.ROSA is most similar to TuRBO and DYCORS, the three methods add random perturbations to the current best, so we list their similarities and differences in Table B .1.• Ackley is implemented as in https://www.sfu.ca/~ssurjano/ackley.html using the recommended values. We evaluate Ackley in the domain x ∈ [-5, 10] D where D ∈ {60, 120, 150, 200}.• Michalewicz is implemented as in https://www.sfu.ca/~ssurjano/michal.html. We evaluate Michalewicz in the domain x ∈ [0, π] D where D ∈ {60, 120, 150, 200}. • Rastrigin is implemented as in https://www.sfu.ca/~ssurjano/rastr.html. We evaluate Rastrigin in the domain x ∈ [-5.12, 5.12] D where D ∈ {60, 120, 150, 200}.• For BBOB we use the COCO suite [26] as implementation. We use the multi-modal functions, F15 through F24, on the following instances We follow the instructions at https://www.miguelanjos.com/jones-benchmark to implement the MOPTA08 problem. As MOPTA08 is a constrained optimization problem, we transform it to unconstrained problem with f unconstr (x) = f constr (x) + 10 68 i=1 max(0, c i (x)) where c i ≤ 0 is a constrain violation. As usual, we optimize the function in x ∈ [0, 1] 124 .The Portfolio Optimization problem aims to maximize volatility adjusted returns and minimize the maximum drawdown over the three year period of 2019-2022, of a portfolio consistent of 496 stocks that were part of SP 500 during all three years. Specifically, we minimize m -r/(σ √ 252), where m is the portfolio maximum drawdown, r is the total portfolio return after the three years, and σ is the standard deviation of the daily returns. The experiments were run on a small cluster consisting of nodes with E5-2690v3 CPU and 32GB DDR4 RAM. Each experiment was limited to 24 hours of total wall-clock run time.In this section we share the progress plots of optimizing the benchmark functions. Each line represents the mean of the lowest found objective value over 30 independent runs, with 95% confidence bounds around it.  
 
 Appendix:  



Reviews: review: title: title: Good paper
main_review: main_review: **Summary:**
This manuscript presents a novel optimization algorithm called ROSA, designed for multi-modal, derivative-free optimization in high-dimensional spaces. The algorithm introduces a combination of randomized optimization, simulated annealing, and surrogate optimization, using polyharmonic spline surrogates. ROSA is shown to significantly outperform existing methods on several synthetic benchmark functions and real-world problems while being more computationally efficient. 

**Strengths:**
1. The proposed ROSA algorithm addresses a challenging problem in high-dimensional optimization, particularly in scenarios with limited function evaluations. 
2. The experimental results demonstrate superior performance across a variety of benchmark functions and outperform competing methods in both accuracy and computational efficiency.
3. The modular design of ROSA allows flexibility in adapting the algorithm to different types of optimization problems, which is a valuable contribution to the field.

**Weaknesses:**
1. The method section lacks clarity regarding how the proposed algorithm is implemented. I think moving the algorithm description from the appendix to the main text would improve readability and accessibility for readers unfamiliar with the approach.
2. The evaluation metric used in the experiments focuses on mean best values across trials. It would also be beneficial to provide visualizations that show how the proposed algorithm and baselines approximate benchmark functions, such as those shown in Figure B.1. This would give readers a clearer understanding of how the algorithm handles different problem landscapes.
3. The figures in the appendix are difficult to read due to the small font size used for labels and numbers. Figures B and C would benefit from larger and more readable captions to ensure that readers can clearly identify the performance differences across methods.
title: title: Official review
main_review: main_review: **Summary**: This work presents a novel optimization algorithm combining stochastic optimization of surrogates with simulated annealing procedures, which offers more computational efficiency than other methods for high-dimensional multi-modal derivative-free optimization problems.

**Evaluation**

- **Quality (8/10)**: The proposal is of evident quality and, in my opinion, meets the scientific standards expected for the workshop. 
- **Clarity (8/10)**: Overall, the manuscript conveys its main ideas within the tight limit of four pages. The only point that stands out as unclear to me is the perturbation procedure. As I understand it from Algorithm A.2, noise is sampled from a truncated distribution and then added to the current values along the selected dimensions. How is it ensured that this approach doesn’t push the values outside the bounds of the hypercube? Wouldn’t it be more appropriate for the perturbation to be a sample from a truncated distribution centered around the current value, as in $\hat{x}\sim T(x',\sigma,a,b)$ (as invoked in Theorem 1)?
- **Originality (7/10)**: While the proposal draws heavily on old and established ideas and methods, it innovates by integrating them in a smart modular fashion, obtaining superior performance compared to when these approaches independently feed solutions to the same problem.
- **Significance (8/10)**: Based on its experimental results, I believe the proposed method provides a strong and competitive alternative for optimization in various applied contexts, especially in engineering.

**Pros** 
- The method is well-build from the mathematical and methodological point of view.

**Cons** 
-  Multi-modal functions are the main focus and are highlighted in the title, yet it appears that both the evaluation and certain theoretical aspects, such as Theorem 1, primarily address the global optimum. It would be helpful to provide more discussion or analysis on local optima as well.
- While there are recommendations regarding the specification of the acceptance probability and perturbation distribution, and the results seem to be robust to variations in these, a more comprehensive ablation study could provide better insights and lead to more informed specifications.