DoE2Vec: Representation Learning for Exploratory Landscape Analysis
Authors: Bas Van Stein
Fu Xing Long
Moritz A. Frenzel
Peter Krause
Markus Gitterle
Thomas Bäck
Venue: Submitted to ICLR 2023 (2023)
Keywords: autoencoder
optimization
exploratory landscape analysis
representation learning

Abstract: We propose DoE2Vec, a variational autoencoder (VAE)-based methodology to learn optimization landscape characteristics for downstream meta-learning tasks, e.g., automated selection of optimization algorithms. Principally, using large training data sets generated with a random function generator, DoE2Vec self-learns an informative latent representation for any design of experiments (DoE).
Unlike the classical exploratory landscape analysis (ELA) method, our approach does not require any feature engineering and is easily applicable for high dimensional search spaces. For validation, we inspect the quality of latent reconstructions and analyze the latent representations using different experiments.
The latent representations not only show promising potentials in identifying similar (cheap-to-evaluate) surrogate functions, but also can boost performances when being used complementary to the ELA features in classification tasks.

Under review as a conference paper at ICLR 2023
DOE2VEC: REPRESENTATION LEARNING FOR EXPLORATORY LANDSCAPE ANALYSIS
Anonymous authors Paper under double-blind review
ABSTRACT
We propose DoE2Vec, a variational autoencoder (VAE)-based methodology learn optimization landscape characteristics for downstream meta-learning tasks, e.g., automated selection of optimization algorithms. Principally, using large training data sets generated with a random function generator, DeE2 self- learns an informative latent representation for any design of experiments (DoE). Unlike the classical exploratory landscape analysis (ELA) method, our approach does not require any feature engineering and is easily applicable for high dimen- sional search spaces. For validation, inspect the quality of latent reconstruc- tions and analyze the latent representations using different experiments. The latent representations only show promising potentials in identifying similar (cheap- to-evaluate) surrogate functions, but also can significantly boost performances when being used complementary the ELA features classification tasks.
INTRODUCTION
Solving real-world black-box optimization problems can be extremely complicated, particularly they are strongly nonlinear and require expensive function evaluations. As suggested by the no free lunch theorem in |Wolpert& & Macready (1997). there is no such things as a single-best optimization algorithm, that is capable of optimally solving all kind of problems. The task in identifying the most time- and resource-efficien optimization algorithms for each specific problem, also known as the algorithm selection problem (ASP) (see Rice (197)). tedious and challenging, even with domain knowledge and experience. In recent years, landscape- ing attention from the research community, where the fitness landscape characteristies are exploited to explain the effectiveness of an algorithm across different problem instances (see van Stein et al. (2033); [Simnncini et al.] (2018)). Beyond that, has been shown that landscape characteristics are sufficiently informative in reliably predicting the performance of optimization algorithms, e.g., using Machine Learning approaches (see Bischl et al. (2012); Dréo et al. 2019); Kerschke & Trautmann (2019);; Jankovic & Doerr (2020); Jankovic et al. (2021) l): Pikalov & Mironovich (2021)). In other words, the expected performance of an optimization algorithm on an unseen problem can be esti- mated, once the corresponding landscape characteristics have been identified. Interested readers
algorithm selection has gained increas-
referred to Muñoz et al. (2015b a Kerschke et al.) (2019): Kerschke & Trautmann (2019a); Malan 2202I.
Exploratory landscape analysis (ELA), for instance, considers six classes of expertly designed fea- tures, including y-distribution, level set, meta-model, local search, curvature and convexity, to nu- merically quantify the landscape complexity of an optimization problem, such as multimodality, global structure, separability, platcaus, etc. (see Mersmann et al. (2010: 201T)). Each feature class consists of set of features, which can be relatively cheaply computed. Other than typical ASP tasks, ELA has shown great potential in a wide variety of applications, such as understanding the under- lying landscape of neural architecture search problems in van Stein et al. (2020) and classifying the Black-Box Optimization Benchmarking (BBOB) problems in Renau et al. (202I). Recently, ELA has been applied not only to analyze the landscape characteristics of crash-worthiness optimization problems from automotive industry, but also to identify appropriate cheap-to- -eeallt functions as representative of the expensive real-world problems (see Long al. (2022)). While ELA is well tablished in capturing the optimization landscape characteristics, we would like to raise our concerns regarding the following aspects.

---

Under review as a conference paper at ICLR 2023
1. Many of the ELA features are highly correlated and redundant, particularly those within the same feature class (see Skvorc et al. (2020)).
2. Some of the ELA features are insufficiently expressive in distinguishing problem instances (see Renau et al. (2019)).
3. Since ELA features are manually engineered by experts, their feature computation might be biased in capturing certain landscape characteristies (see /Siler et al. (2020).
4. ELA features are less discriminative for high-dimensional problems (see Muñoz & Smith- Miles (2017))
Instead of improving the ELA method directly, e.g., searching for more discriminative landscape features, we approach the problems from a different perspective. In this paper, we introduce an automated self-supervised representation learning approach to characterize optimization landscapes by exploiting information in the latent space. Essentially, a deep variational autoencoder (VAE) model is trained to extract an informative feature vector from design of experiments (DoE), which is essentially a generic low-dimensional representation of the optimization landscape. Thus, the name of our approach: DeE2Vec. While the functionality of our approach is fully independent of ELA, experimental results reveal that its performance can be further improved when combined with ELA (and vice versa). To the best of our knowledge, a similar application approach with VAE in learning optimization landscape characteristics is still lacking. Deetion[]brieff introduces the state- of-the-art representation learning of optimization landscapes as well as the concepts of (variational) autoencoder. This is followed by the description of our methodology in Section 3 Next, we explain and discuss our experimental results Section 4 Lastly, conclusions and outlooks are included in Section 5
REPRESENTATION OF OPTIMIZATION LANDSCAPE
In the conventional ELA approach, landscape features computed primarily using a DoE of some samples points W = [w).... w, evaluated on an objective function f. i.e., f: Rd - R, with w; Rdd represents sample size, and d represents function dimension. The objective function values f(w:),, E (1....,,) n) are the inputs of VAE models in DoE2Vec. In this work, we consider ELA features similar to those in Long al. (2022). which do not require additional sampling, and compute them with the package flacco by Kerschke & Trautmann (2019b c). These features include commonly used dimensionality reduction approaches such as Principal Component Analysis (PCA) Abdi & Williams (2010), number of simple surrogate models and many others.
To overcome the drawbacks of the ELA approach, attentions have been focused on developing algorithm selection approaches without requiring landscape features. For example, Prager et al. (022)) proposed two feature-free approaches using a deep learning method, where optimization landscapes can be represented through either 1) image-based fitness maps or 2) graph-like fitness clouds. In the first approach, convolutional neural networks were employed to project data sets into two-dimensional fitness maps, using different dimensionality reduction techniques. In the second approach, data sets were embedded into point clouds using modified point cloud transformers, which can accurately capture the global landscape characteristics. Nonetheless, the fitness map approach suffered from the curse of dimensionality, while the fitness cloud approach was limited to fixed training sample size. Additional relevant works be found in Alissa al. (2019); (Selerr et al. 2020 2022); Prager et al.] (202T). Unlike these approaches, which were directly used as classi- fiers. the latent feature sets generated by our proposed approach can be easily combined with other features, such as ELA features, for classification tasks. In our work, we do not propose to replace conventional ELA features, but actually extend them with autoencoder (AE) based latent-space features. Since the implementation of both approaches mentioned above is not available, a compar- ison to our work in terms of classifying high-level properties is only feasible by directly comparing their results on a identical experimental setup. Following this, results from the downstream tasks in this work can partially be compared to the mentioned results in Seiler et al. (2022), inlcuding the standard Principal Component Analysis (PCA), reduced Multiple Channel (rMC) and transformer based approach (Transf.), taking into account that additional hyperparameter tuning was involved in their classification experiments with ELA features.
2

---

Under review as a conference paper at ICLR 2023
Our approach is capable of learning the representations of optimization landscapes in automated, generic and unsupervised manner, with the advantage that the learned features are not biased to- wards any particular landscape characteristic. Unlike previously mentioned approaches, posed method is independent of the sampling method. By using only fully connected (dense) layers that learn from one-dimensional (flattened) landscapes, AE VAE is, theory, capable of learning any number of input-dimensions without scaling difficulties. Furthermore, the fast-to-train (V)AE models can be easily shared in practice.
pro-
2.1 AUTOENCODER
A standard AE usually has symmetrical architecture, consisting of three components: an encoder, a hidden layer, also known as bottleneck, and a decoder (Figure []. short, an encoder projects the input space X' to representative feature space H. i.e., d' H. while decoder transforms the feature space back to the input space d: H. X (see Charte al. 2020 2018)). In other words, AE attempts to optimally reconstruct the original input space A, by minimizing the reconstruction error L(X, X)), e.g. mean squared error, during the (unsupervised) training process. Commonly, AE is constructed with input layer and output layer of the same dimension dim(X) and bottleneck layer of lower dimension dim (f), i.e., dim(HH) < dim(X), to improve its ability in capturing crucial representations of the input space.
Following this, AE has rapidly gained popularity in the field of dimensionality reduction as well as representation learning Bengio al. Tschannenhnnnen
al. (2018). In comparison with the simple principal component analysis (PCA) technique, AE has the advantage that nonlinear representation features can be learned with activation functions, such as sigmoid function.
2.2 VARIATIONAL AUTOENCODER
Originating from the same family as traditional AE, VAE typically has similar architecture with some modifications, as shown in Figure T Unlike AE, the latent space of VAE encoded as
Encoder
Is
Decoder
Encoder
Decoder
d
e(x)
d(e(x))
Figure 1: Architecture of a (standard) AE on the left and VAE on the right used in DoE2Vec. The latent space z is defined by = μ + a where denotes the sampling with mean j and variance
distribution by using a mean and variance layer, together with a sampling method. Following this, the latent space can be properly regularized to provide more meaningful features. During the training process, a common loss function LVar (Equation D of VAE is to be minimized, consisting regularization term (Equation 2), which be expressed the Kullback-Leibler (KI.) divergence LKL, and a reconstruction error, e.g., mean squared error LAS (Equation
&v) B Lxi. 8) + (MMSE(X,
(1)
(x, = (exp(os) - + σi) + +
(2)
1=1
(M) =
Σ (I (
(3)
#EX
where a weighting factor included to ensure good trade-off between CxL and LMSE o and J are the variance and mean latent layers the VAE and & is the reconstruction of the input space. Detailed explanations regarding VAE can be found in Kingma & Welling (2013 2019).
3

---

Under review as a conference paper at ICLR 2023
2.3 BLACK- Box OPTIMIZATION BENCHMARKING
The development of Vec is based on the well-known academic BBOB suite by Hansen et al. (209), consisting of altogther 24 noise-free real-parameter single objective optimization problems of different landscape complexity. For each BBOB problem, the global optimum (within [-5, 5) can be randomly shifted, e.g., through translation or rotation, to generate new problem instance.
DE
Generally, our proposed method uses a VAE with similar design as described in Section 2.2 Pre- cisely, our VAE model has an architecture of altogether seven fully connected layers, where rectified linear unit (ReL.U) activation functions are assigned to the hidden layers, while a sigmoid activation function is used for the final output layer of the decoder. The encoder is composed of four fully connected layers with dim depending on the DoE sample size n, starting with the input layer size two hidden layers with sizes 11/ and n/4 and the latent size Is (ls < n/44) for the mean and log variance of the latent space. The decoder is composed of three fully connected layers with sizes n/4, n/2 and for the final output layer. The focus of our approach lies on VAE, rather than AE, because it has the additional benefits of regularizing the latent space without loss of generalisation. For comparison,
consider standard AE model as well (with the same number of hidden layers, except that the latent space is now a single dense layer, instead of a mean and log variance layer with a sampling scheme). Full specifications of the different models are available in our Github repository (for Review (2022), ) while pre-trained models are also available on Huggingface
The general workflow of DoE2Vec be summarized as follows:
1. First, DoE of size 2 is created, where m is user defined parameter. By default, Sobol sequence by [Sobol" (1967) is used as sampling scheme, but any sampling scheme or even custom DoB can be utilized in practice.
2. The DoE samples, initially within the domain [0, 11 can be re-scaled to any desired func- tion boundaries, as the DoE2Vec method scale-invariant by design.
3. Next, the DoE. samples are evaluated for of functions randomly generated using the random function generator from from|Long et al. (2022), which was originally proposed by Tian al. (2200). The main advantage of using this function generator that large training set can be easily created, covering a wide range of function landscape of different complexity.
4. Following this, all objective function values are first re-scaled to [o, 1] and then used as input vectors to train (V)AE models.
5. Lastly, the latent space of the trained V(AE) models be used feature vectors for further downstream classification tasks, such as optimization algorithm selection.
In the next section, we will show that the learned feature representations have attractive characteris- tics and they are indeed useful for downstream classification and meta-learning tasks.
4 EXPERIMENTAL RESULTS AND DISCUSSIONS
In this work, we have conducted altogether four experiments for different research purposes. Basi- cally, the first two experiments are to investigate the quality of VAE models trained and to validate the latent space representation learned, while the last two experiments are to evaluate the potential of the Vec approach for downstream meta-learning tasks, follows.
1. Analyzing the reconstruction of function landscapes and the impact of the latent size and KI. loss weight (weighting factor !).
2. Investigating the differences in latent space representation between a standard AE and VAE model.
3. Identifying functions with similar landscapes based on the latent representations.
4. Three downstream multi-class classification experiment to show the potential of the pro- posed approach in practice, using the latent feature vectors inputs.
4

---

Under review as a conference paper at ICLR 2023
In experiments, we fix the sampling scheme to Sobol sequence and to eight, ending up with a DoE of 256 samples. Unless otherwise specified, all AE and VAE models are trained on a set of 250, 000 five-dimensional (5d) random functions.
4.1 RECONSTRUCTION OF FUNCTION LANDSCAPES
In our first experiment, we investigate the impact of two model parameters, namely the KL loss weight and the latent size, on the model loss functions. For this purpose, a total of 30 VAE models trained using combinations of six different latent sizes (4, 8, 16, 24, 32) and five different KL loss weights (0.0001, 0.0002, 0.001, 0.005, 0.01). Results in the left subplot of Figure latent sizes have a positive effect on the LVAR, with the improvement diminishes beyond 16. On the other hand, the bMSF can be improved with smaller KL loss weights, as shown in the right subplot. In Figure 3 the combined effects of these parameters on the LKI. and LASE are fully visualized.
show that
032
301
032
3015
0:11
3005
CX3
laion.da
Figure 2: Impact of different latent sizes and KL loss weights on the loss functions. Left: LVaB against latent sizes. Right: LMSE against KL loss weight.
Iatent size
KL weight
CRa
0088
V0000
100g
60000
18
1008
Figure 3: Parallel coordinates plot of different latent sizes and KL loss weights in relation with the validation LKL and LMSE- (Columns left to right: Latent size, KL loss weight, LMSE and LxL.) Each color or line represents a combination of latent size and KI. loss weight. Conflict between both loss terms can be observed, since parameter combinations with low LMSE have higher LxL. and vice versa.
In the remaining experiments, we use a latent size of either 24 or 32 (expressed as (V)AE-24 or (V)AE-32) and KL loss weight of 0.001 as a good compromise between the two loss terms. is to be noted that, while the (V)AE architecture can be further improved by applying neural architecture search, we leave it for future work, since the landscape reconstruction is not the ultimate purpose of this work. In fact, the reconstruction here is meant be way in evaluating the quality of learned representations. Subsequently, we verify the capability of a VAE-24 model by reconstructing large variety of functions, using the 24 BBOB functions (first problem instance) (Figure 4).
5

---

Under review as a conference paper at ICLR 2023
BRG8
R
BRGB
Recondtruted
Reconetucced fu
Reconenncttd fa
BAGA
A
moR
Recieddneted
aeranemurted frs
mosk
tecreemntted by
6808
Reconatnutedd
8808
Recerotncced fu
5908
Recerstracted fyr
8908
tocarmmnceed fo
Recorstruttdd
Recerotncttd
Recorstructed to
Reconstructions of the 24 2d BBOB funcions (labelled from fi to fe1) using a VAE-24 model. The surface plots generated based on DOE of 256 samples using triangulation. Gener- ally, the reconstructed landscapes look similar to the actual landscapes based on visual inspection.
Figure 4:
4.2 LATENT SPACE REPRESENTATION
Next, we project the latent space of an AE and a VAE from our DoE2Vec approach onto a 2d visual- ization using multidimensional scaling (MDS) method, as shown in Figure For comprehensive comparison, a similar MDS visualization for the feature vectors with all the (considered) ELA fea- tures is included as well. Interestingly, the latent space of ELA has better looking cluster-structure, while the latent space of AE and VAE have a clear structure. As expected, the latent space of AE contains few outliers and is in general more widespread than the latent space of VAE.
Figure 5: From left to right: 2d MDS visualization of the (normalized) ELA feature spaces, the latent space of an AF-32 model and the latent space of VAE-32 model. Altogether 100 problem instances for all 24 BBOB functions, resulting in a total of 2,400 dots (in each subplot), where each dot represents a feature vector of BBOB instance and the color denotes the corresponding BBOB function.
To further analyze the differences in latent space representation between the AE and VAE models, the generator part of the models iteratively generate new function landscapes. To be

---

Under review as a conference paper at ICLR 2023
precise, we select one latent representation starting point and individually vary each latent variable with small steps between -1 and +1 of its original value. In Table [ the generated landscapes are shown only for the extreme cases of an AE-4 and VAE-4 model. Based on this result, two important conclusions can be drawn.
1. The interpolation for both models is very smooth, showing that the models can learn the landscape representations well and are able learn the structure, even though the dimen- sionality of the function landscapes d is not known.
2. The VAE model utilizes a more compact feature representation, where its output variance is greater than those of the AE model for the same variation of latent features.
Table 1: Comparison of function landscapes generated by an AE-4 and a VAE-4 model for different latent variables, using the BBOB f22 encoding starting point (U1, 02 V3 u4). In each column, the latent features are separately varied by - 1 or +1. Complete variation of the latent features can be found in videos available in our Github repository. start
V1
1/2
44
-1
1
+1
1
+1
+1
AE
VAB
4.3
SIMILAR REPRESENTATIVE FUNCTIONS
The first main application of DoE2Vec is to identify space representation for given DoE, which can be very useful when dealing with real-workss ex- pensive black box optimization problems, as also described in Long et al.] (2022). In this way, large experiments can now be conducted on "similar" function group at much lower computational cost. Since a direct analytical verification of the results is very challenging, we instead use a visual inspection based on 2d functions showcase the potential of the proposed method. In Table 2 the 24 BBOB functions are paired with their respective "nearest" random function, where nearest defined as the random function that has the closest feature representation in terms of Euclidean distance. For most BBOB functions, very well fitting (almost identical) random function can be identified by the DoE2Vec model, where random functions of equal complexity are suggested for the more complex BBOB functions.
functions with similar latent
4.4 CLASSIFICATION TASKS
Secondly, the DoR2Vec approach is designed to learn characteristic representations of an optimiza- tion landscape, which can be verified through classical classification task of high level function properties. These high level properties, such as multimodality, global structure and funnel structure, are important for the ASP, as they often determine the difficulty of an optimization problem. Table
3 illustrates the BBOB functions and their associated high level properties. In this experiment, standard random forest (RF) model (using sklearn from nom/Peeregosa et al. (201T) Th) implemented for the multiclass classification tasks based on the latent representations learned by four DoE2 Vec models, consisting AE-24, AF-32, VAF-24 and VAE-32. In other words, the high level properties of a BBOB function to be predicted using the latent representations. Again, compare the DeE2 approach against the classical ELA method, which is specifically constructed to excel in exactly this kind of function property classification tasks. Beyond that, combination of classical ELA features with a VAE-32 model is included to evaluate the complimentary effect of the different feature sets.
The classification results (macro Fl scores) of the different feature sets are summarized in Table
It is not surprising that the ELA features generally perform very well and outperform the latent
7

---

Under review as a conference paper at ICLR 2023
Table 2: Four columns with pairs of BBOB function and their respective nearest random function identified based on latent feature vectors using the VAE-24 model.
(iarance
tackie
Lirssnce
Nonotrender tandke
Norotranee
Ae Istancc
Norrot henden
Bweee redeer fectie
Favel
Si Faneen
New wot ra Enariee
Hewred bection
Bawee reneer featie
Favel
Sa Fdbee
Heweed ewaiae bectan
features most of the time, especially in classifying the global structure and multimodal landscapes. Fascinatingly, the classification performances can be significantly improved when the DoE2Vec is combined with the classical ELA method, indicating that both feature sets seem to be complimentary each other.
CONCLUSIONS AND FUTURE WORK
In this work propose DoE2Vec, VAE-based approach to learn the latent representations of an optimization landscape, similar to the classical landscape features. Using a wide set of experiments, have shown that DoE2Vec is able to reconstruct large set of functions accurately and that the ap- proach can be used for downstream meta-learning tasks, such as algorithm selection. The proposed methodology be effectively used next existing techniques, such as classical ELA features, to further increase the classification accuracy of certain downstream tasks. In fact, DeE2 can learn good feature representations for optimization landscapes and has several advantages over the ELA approach, such feature engineering selection knowledge is not required, domain knowledge ELA needed and applicable optimization tasks very straightforward manner. Nonetheless, there are few known limitations to the proposed method, such as 1) Our approach is scale-invariant, but not rotation- or canslaaioo-invariann Using different loss function train the autoencoders might be able to improve this 2) If custom DoE sample is used, the model needs to be trained from scratch (no pre- pre-trained model available). This typically takes a few minutes up to an hour, depending the sample size and number of random functions train on, 3) The learned feature representations are a black-be -bx that are hard to interpret directly.
In future work, plan improve
approach by tackling some of the challenges mentioned. Apart from that, we would like to verify the usefulness of the random functions with similar latent features w.r.t. the performances of an optimization algorithm.
8

---

Under review as a conference paper at ICLR 2023
Table 3: High level properties of the 24 BBOB functions. This table first introduced in Seiler
et al. (2022)
BBOB function
Multimodal
Global Structure Funnel
1: Sphere
none none
none yes
2: Ellipsoidal separable
yes
Rastrigin separable
high
strong
yes
4: Bücch- Buiche-Rasrriinn
high
strong
yes
5: Linear Slope
none none yes
6: Attractive Sector
none
none
yes
7: Step Ellipsoidal
none
none
yes
8: Rosenbrock
low
yes
Rosenbrock rotated
low
none yes
10: Ellipsoidal high conditioned
yes
11: Discus
none
none
yes
12: Bent Cigar
none
none
yes
13: Sharp Ridge
none
none none yes
14: Different Powers
yes
15: Rastrigin multimodal
high
strong yes
16: Weierstrass
high medium none
17: Schaffer F7
high medium yes
18: Schaffer F7 moderately ill-cond.
high medium yes
19: Friewank-Rosenbreee
high strong yes
20: Schwefel
medium deceptive yes
21: Gallagher 101 Peaks
medium none none
22: Gallagher 21 Peaks
low none none
23: Katsuura
high
none
none
24: Lunacek bi- bi-Rastrigin
high weak yes
Table Classification results (averaged macro Fl scores over 10 runs with different random seeds) using standard RF model with 100 trees, trained on the feature representations (from AE, VAE, classical ELA or ELA combined with VAE-32) the first 100 instances for each BBOB function and validated on instance 101 to 120. PCA, rMC and Transformer results directly taken from the work of Seiler et al. (2022). which uses an identical experimental setup but without repetitions.
d
Task
AE-24
AE-32
VAE-24
VAE-32
ELA
PCA*
rMC*
Transf *
ELA-VAE
multimodal
0.875
0.849
0.877
0.856
0.984
0.994
0.971
0.991
0.991
2
global struct.
0.903
0.904
0.902
0.889
0.983
0.992
0.965
0.991
0.998
funnel
0.985
0.974
0.956
0.978
1.000
0.999
0.995
1.000
1.000
multimodal
0.908
0.903
0.880
0.889
0.963
0.897
0.947
0.991
0.998
5
global struct.
0.838
0.828
0.810
0.793
1.000
0.807
0.859
0.978
1.000
funnel
1.000
1.000
0.996
0.991
1.000
0.990
0.989
1.000
1.000
multimodal
0.877
0.813
0.844
0.838
1.000
0.839
0,952
0,974
L.000
10
global struct.
0.794
0.737
0.783
0.745
0.902
0.774
0.911
0.963
0.991
funnel
0.998
0.993
0.997
0.993
0.972
0.977
0.991
1.000
0.997
multimodal
0.726
0.722
0,700
0.694
0.970
0.991
20
global struct.
0.689
0.621
0.606
0.626
0.972
0.997
funnel
0.993
0.982
0.985
0.982
1.000
1.000
REPRODUCIBILITY STATEMENT
We provide open-source documented implementation of our package at with visualizations and video explanations. Pre-trained models (weights) are available Huggingface hntps://huggingfiee.o///// wttp://ccc..oxxxxxxxcxxx All models trained using a single T4 GPU and the computations are carbon neutral (C'O,-free) by using solar power. Average training time of a model on 250, 000 random functions was five minutes.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Hervé Abdi and Lynne J Williams. Principal component analysis. Wiley interdisciplinary reviews: computational statistics, 2(4))433 459, 2010.
Mohamad Alissa, Kevin Sim, and Emma Hart. Algorithm selection using deep learning without feature extraction. In Proceedings of the Genetic and Evolutionary Computation Conference, GECCO '19, pp. 198-206. Association for Computing Machinery, 2019. ISBN 9781450361118 doi: 10.11455321100..321855.
Yashua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35:1798-188 -1288, 08 2013. doi: 10. 10..1/9/..M...013...0..
Bernd Bischl, Olaf Mersmann, Heike Trautmann, and Mike PreuB. Algorithm selection based on exploratory landscape analysis and cost-sensitive learning. In Proceedings of the 14th Anmual Conference on Genetic and Evolutionary Computation, GECCO '12, pp. 313-320, New York, NY, USA, 2012. Association for Computing Machinery. ISBN 978145031 1181450311779 2330163.2330209
doi: 10.1 10.1145/
David Charte, Francisco Charte, Salvador García, María J. del Jesus, and Francisco Herrera. A practical tutorial on autoencoders for nonlinear feature fusion: Taxonomy, models, software and guidelines. Information Fusion, 44:78-998 96, nov 2018. doi: 10.1016/j.infuus infffs 12.007.
David Charte, Francisco Charte, María del Jesus, and Francisco Herrera. analysis on the use of autoencoders for representation learning: Fundamentals, learning task case studies, explainability and challenges. Neurocomputing, 404:93-107, sep 2020. doi:
Johann Dréo, Carola Doerr, and Yann Semet. Coupling the design of benchmark with algorithm in landscape-aware solver design. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, GECCO '19, pp. 1419-1420, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367486. doi: 10. 10.1145/3319119.335666
XXXXX Masked for Review. Xxxxx/doe2vec: General public release, September 2022. URL
https: http:///oooog/oooo www.11..11...ncccc. XXXXX
Nikolaus Hansen, Steffen Finck, Raymond Ros, and Anne Auger. Real-Parameter Black-Box Op-
timization Benchmarking 2009 Noiseless Functions Definitions. Research Report RR-6829,
INRIA, 2009. URL httpp:://aaat
Anja Jankovic and Carola Doerr. Landscape-aware fixed-budget performance regression and algo- rithm selection for modular cma-es variants. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference, GECCO '20, pp. 841-849, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 978145033 141441122255
doi: 10.1 00.1145/337793033390133
Anja Jankovic, Gorjan Popovski, Tome Eftimov, and Carola Doerr. The Impact of Tuning for Landscape-Aware Performance Regression and Algorithm Selection, pp. 687-696. GECCO '21. Association for Computing Machinery, New York, NY, USA, 2021. ISBN 9781450383509, doi: 10. (0.11445/449639.3454000
Pascal Kerschke and Heike Trautmann. Automated algorithm selection on continuous black-box problems by combining exploratory landscape analysis and machine learning. Evolutionary Com- putation, 27(1):99- 127, 2019a. ISSN 1063-6560.
doi: 1621ee2o a 00236.
Pascal Kerschke and Heike Trautmann. Comprehensive Feature-Batt Based Landscape Analysis of Con- timoous and Constrained Optimization Problems Using the R- Package Flacco, pp. 93-123.. Stud- ies in Classification, Data Analysis, and Knowledge Organization. Springer International Publish- 7.
ing, Cham, 2019b. ISBN 978-3-030-25147-5. doi: 10.
Pascal Kerschke and Heike Trautmann. flacco: Beature-based landscape analysis of continuous and constrained optimization problems, 2019c. URL htttss://ttttttiis com/ /etm///escccee///e
flacco Last accessed 15 January 2022.
10

---

Under review as a conference paper at ICLR 2023
Pascal Kerschke, Holger Hoos, Frank Neumann, and Heike Trautmann. Automated algorithm selec- tion: Survey and perspectives. Evolutionary Computation, 27(1):3-44 45, 2019. ISSN 1063-6560.
doi: 10. 00242.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. arXiv: 14, 2013. doi: (048550/ARXI..1112.6114
arXiv preprint
Diederik P. Kingma and Max Welling. An introduction to variational autoencoders. Foundations and Trends in Machine Learning, 12(4):307-332, 2019. doi: 10.1561/22000000566
Fu Xing Long, Bas van Stein, Moritz Frenzel, Peter Krause, Markus Gitterle, and Thomas Bäck. Learning the characteristics of engineering optimization problems with applications in automo- tive crash. In Proceedings of the Genetic and Evolutionary Computation Conference, GECCO '22, pp. 1227-1236, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450392372. doi: 10.1 011114//55122903528711
Katherine Mary Malan. A survey of advances in landscape analysis for optimisation. Algorithms,, 14(2):40, 2021. ISSN 1999-48933 doi: 10.3390/a 4020044..
Olaf Mersmann, Mike Preuss, and Heike Trautmann. Benchmarking evolutionary algorithms: To- wards exploratory landscape analysis. In Robert Schaefer, Carlos Cotta, Joanna Kotodziej, and Günter Rudolph (eds.), Parallel Problem Solving from Nature, PPSN XI, pp. 73-82, Berlin, Heidelherg, 2010. Springer Berlin Heidelherg. ISBN 11833644-15844-5. doi: 10.1007/ 1---34422-18844--
Olaf Mersmann, Bernd Bischl, Heike Trautmann, Mike Preuss, Claus Weihs, and Günter Rudolph. Exploratory landscape analysis. In Proceedings of the 13th Annual Conference on Genetic and Evolutionary Computation, GECCO 11. pp. 829-836, New York, NY, USA, 2011. Association for Computing Machinery. ISBN 978 9781450305570. doi: 10. 10.1145/2001556.200166
Mario A. Muñoz and Kate A. Milth-Miles. Performance analysis of continuous black-box opti- mization algorithms via footprints instance space. Evol. Comput., 25(4):529-554 dec 2017.
ISSN 1063-6560. doi: 10. 1162/evco_a_0011))
Mario Andrés Muñoz, Michael Kirley, and Saman K. Halgamuge. Exploratory landscape analysis of continuous space optimization problems using information content. IEEE Transactions on Evolutionary Computation, 19(1):74-87 2015a. doi: 10.1109TEVC2201422222000
Mario Andrés Muñoz, Yuan Sun, Michael Kirley, and Saman K. Halgamuge. Algorithm selection for black-box continuous optimization problems: survey on methods and challenges. Information Sciences, 317:224 245, 2015b. ISSN 0020-0255. doi:
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapaau, Matthieu Brucher, Matthieu Perrot, and Édouard Duch- esnay. Scikit-learn: Machine learning in python. Jourmal of Machine Learning Research, 12(85): 2825-2830, 2011. URL f 1ml www..orgg.pppppp..e.oeep / v1 http:/pdrrrrgsso la ht.m1
Maxim Pikalov and Vladimir Mironovich. Automated Parameter Choice with Exploratory Land- scape Analysis and Machine Learning, pp. 1982-1985. GECCO '21. Association for Computing Machinery, New York, NY, USA, 2021. ISBN 978 9781450383516. doi: 10. 00.1445/344972634463313
Raphael Patrick Prager, Moritz Vinzent Seiler, Heike Trautmann, and Pascal Kerschke. Towards feature-fres automated algorithm selection for single-objective continuous black-box optimiza- tion. In 202/ IEEE Symposium Series on Computational Intelligence (SSCI), pp. 1-8, 2021. doi: 10. N0.1109/5SC150411.000 9.661444.
Raphael Patrick Prager, Moritz Vinzent Seiler, Heike Trautmann, and Pascal Kerschke. Auto- mated algorithm selection in single-objective continuous optimization: A comparative study of deep learning and landscape analysis methods. In Günter Rudolph, Anna V. Kononova, Hernán Aguirre, Pascal Kerschke, Gabriela Ochoa, and Tea Tušar (eds.), Parallel Problem Solving from Nature - PPSN XVII. pp. 3-17, Cham, 2022. Springer International Publishing. ISBN 978-3-031- 14714-2.
11

---

Under review as a conference paper at ICLR 2023
Quentin Renau, Johann Dreo, Carola Doerr, and Benjamin Doerr. Expressiveness and robustness of landscape features. Proceedings of the Genetic and Evolutionary Computation Conference New York, NY, USA. 2019. Association for Computing Machinery. ISBN 978-1-4503-6748-6 doi: 10.1145/3319619.322
Companion, GECCO '19, pp.
Quentin Renau, Johann Dreo, Carola Doerr, and Benjamin Doerr. Towards explainable exploratory landscape analysis: Extreme feature selection for classifying bbob functions. In Pedro A. Castillo and Juan Luis Jiménez Laredo (eds.), Applications of Evolutionary Computation, pp- 17-33, Cham, 04 2021. Springer International Publishing. ISBN 978-3-030-72698-00 doi: 11100779788-3000--766999
John R. Rice. The algorithm selection problem. volume 15 of Advances in Computers, pp. 65-118. Elsevier, 1976. doi: 10.1016/S0065- 1..001660066--2488888866663633 2458(08)60020-3.
Moritz Seiler, Janina Pohl, Jakob Bossek, Pascal Kerschke, and Heike Trautmann. Deep learning as competitive feature-free approach for automated algorithm selection the traveling salesperson problem, 2020.
Moritz Vinzent Seiler, Raphael Patrick Prager, Pascal Kerschke, and Heike Trautmann. A collec- tion of deep learning-based feature-free approaches for characterizing single-objective continuous fitness landscapes, 2022.
David Simoncini, Sophie Barbe, Thomas Schiex, and Sébastien Verel. Fitness landscape analy- sis around the optimum in computational protein design. Proceedings of the Genetic and Evolutionary Computation Conference, GECCO '18, pp. 355-362, New York. NY, USA, 2018. Association for Computing Machinery. ISBN 9781450356183, doi: 10. 11155/3225555.32255666
Il ya Meerovich Sobol'. On the distribution of points in a cube and the approximate evaluation of integrals. USSR Computational Mathematics and Mathematical Physics, 7(4):86-11 -12,, 1967. ISSN 0041-5553, doi: 110016/0011-55566779900
Ye Tian, Shichen Peng, Xingyi Zhang, Tobias Rodemann, Kay Chen Tan, and Yaochu Jin. A rec- ommender system for metaheuristic algorithms for continuous optimization based on deep re- current neural networks. IEEE Transactions Artificial Intelligence, 1(1):5-18, 2020. doi:
10.1109/AA.220033222333
Michael Tschannen, Olivier Bachem, and Mario Lucic. Recent advances in autoencoder-based representation learning, 2018.
Bas van Stein, Michael Emmerich, and Zhiwei Yang. Fitness landscape analysis of nk landscapes and vehicle routing problems by expanded barrier trees. In EVOLVE-A Bridge between Probabil- ity, Set Oriented Numerics, and Evolutionary Computation IV, pp. 75-89. Springer, 2013.
Bas van Stein, Hao Wang, and Thomas Bäck. Neural network design: Learning from neural ar- chitecture search. In 2020 IEEE Symposium Series Computational Intelligence (SSCI), pp- 1341-1349. IEEF, 2020. doi: 10. STW.:/CCCI4400.222.9900000
Urban Škvore, Tome Eftimov, and Peter Korošec. Understanding the problem space in single- objective numerical optimization using exploratory landscape analysis. Applied Soft Computing, 90:106138, 2020. ISSN 1568-4946. doi: 10. 202.
David H. Wolpert and William G. Macready. No free lunch theorems for optimization. 1EEE Transactions on Evolutionary Computation, 1(1):67-82, 1997. doi: 10.1110/4235.585889.
12


Reviews:
[{"review_id": "iWnVZBnXLo", "reviewer": "Reviewer_wn3f", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In \"DoE2Vec: Representation Learning for Exploratory Landscape Analysis\" the authors propose a variational auto encoder (VAE) to learn a latent feature representation of black-box optimization problems. In the experimental section, the authors show that this VAE approach is best combined with exporatory landscape analysis (ELA) to obtain the best results. It is argued that the learnt representations can be then used in downstream (meta-learning) tasks such as algorithm selection.", "summary_of_the_review": "Overall, I think this could be an interesting direction to go for certain BBO tasks but still it is rather unclear to me how the ASP problem should be reasonably implemented with this approach. Also the claims that downstream tasks are facilitated etc. are not proved in this paper. Furthermore, the proposed method is rather an application and more about the dataset used so that the novelty of a methodological perspective is comparably low and questionable to what extent this contribution may impact the community. Therefore, I recommend to reject the paper.", "strength_and_weaknesses": "[+] Designing useful, complementary, and informative features is one of the major challenges in downstream tasks like algorithm selection and therefore reasonable feature-free or automatic feature design approaches are of relevance.\n\n[-] It is claimed that the proposed method facilitates downstream tasks but there are no results contained in the paper to support such claims. More specifically, it is only demonstrated that the method can retrieve random functions which look similar to the actual target functions based on a set of evaluations of the target function in question. While this may indicate that the retrieved random functions may be used as a surrogate it remains open whether (a) the random function is indeed faster to evaluate, (b) the random function is sufficiently close to the target function, and (c) whether the method gives any advantage over standard methods that could be applied directly.\n\n[-] One of the example downstream tasks is algorithm selection. However, it is not clear how the method would be mapped to the setting of algorithm selection where for each problem instance an algorithm needs to be picked. If a problem instance is a bbob function from the paper, one would need to observe objective function values for some algorithms in order to obtain the feature representation. However, if algorithms are already run, the current problem instance is already solved in the best case so that no features are needed anymore. Otherwise, the feature computation would already take too long.\n\n[-] The proposed method is not very innovative. It just applies VAE to some vectors sampled from specific functions. In combination with the limited experimental section regarding the downstream tasks or at least some case studies of how to employ and practically use the approach for downstream tasks, the paper is of little help, I would say.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Mq30fsxPOR", "reviewer": "Reviewer_7ZSu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a VAE-based approach, called DoE2Vec to learn the latent representation approximating the optimization landscape. The authors showed the latent feature vectors that is learned by the DoE2Vec network can reconstruct and predict high level properties of the function landscapes. They also showed that the proposed method can be used in combination with existing exploratory landscape analysis (ELA), to further improve the classification accuracy of some downstream tasks. The authors reported the effectiveness of DoE2Vec using Black-Box Optimization Benchmark (BBOB) function landscapes. \n\n", "summary_of_the_review": "The authors introduced a VAE-based approach for function landscape analysis which is an interesting and challenging problem. However, I think the authors should justify the role of autoencoders and their latent representation in the ELA task and discuss the role of each latent factor in function landscapes and clarify why this model is expected to outperform the other existing ELA techniques. \n\n", "strength_and_weaknesses": "Strength:\n\n- The proposed VAE-based methodology seems to reconstruct properly most of BBOB functions. \n\nWeaknesses:\n\n- Design of experiments in general is often suited for multivariable analyses and exploring causal relationship. One can test and optimize several variables simultaneously, thus accelerating the process of discovery and optimization. I do not see any analysis which discusses how latent representation of Doc2Vec is suitable for such tasks. It is not clear how each latent vector effectively represents such relationship.\n\n- The reported results are only demonstrating the performance of the latent representation in reconstruction and prediction of properties of a specific set of functions. It seems more like an AE-VAE analysis paper which is evaluating their performances in reconstructing a set of functions.\n\n- I think one important missing aspect is traversal analysis and quantifying the disentanglement of latent factors to show how a disentangled representation can be used to explore the function landscape.  \n\n- There is no quantification that shows how Doc2Vec performs against competing algorithms. \n\n\nQuestions / Concerns :\n\n- I do not understand how the proposed approach can be used for downstream meta-learning task?\n\n- What is novel about the results in Figure 2? Adding more dimension to the latent space improves the loss, and a lower beta provides better reconstruction error.  Is that not something already known from $\\beta$-VAE?\n\n- Can you elaborate more on the plots in Figure 5?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "MddGZcDcE7", "reviewer": "Reviewer_jSUv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a variational autoencoder-based method to learn optimization landscape characteristics for downstream meta-learning tasks. This method uses large training sets to self-learn a latent representation for any design of experiments. Four experiments were designed to evaluate the quality of learned variational autoencoders and their effectiveness on downstream meta-learning tasks.", "summary_of_the_review": "This paper has clear strengths and weaknesses. Its quality can be improved with more theoretical contribution and more experiments to demonstrate the advantages of the proposed method.", "strength_and_weaknesses": "Strength:\n\n- The paper is well-motivated. \n- The application of variational autoencoder in exploratory landscape analysis seems to be novel.\n- Four experiments were designed to show the effectiveness of the proposed solution.\n\nWeakness:\n\n- This paper directly applies variational autoencoder to exploratory landscape analysis tasks. There lack more in-depth theoretical analysis of why and how this method works well on these tasks.\n- The experiments lack comparison with other approaches.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8QLpMbpxpk", "reviewer": "Reviewer_KUFH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a new method to learn optimization landscape features using latent space information with AE/VAE for downstream meta-learning tasks.", "summary_of_the_review": "All in all, the paper has its novelty and contains a lot of simulations to verify the advantage of their method. However, I would recommend to include more numerical results, such as repeated executions and more performance measures.", "strength_and_weaknesses": "\nStrength:\n1.\tThe DoE2Vec can reconstruct a large set of functions and can be applied to downstream meta-learning tasks.\n2.\tThe combination of ELA and VAE-32 outperforms all other methods in table 4, which shows the new method can learn a good feature representation for optimization landscapes.\nQuestions/Weakness :\n1.\tIn Table 2, the VAE-24 model seems cannot represent f16, f21 and f24 well which have medium or high multimodal property. Is there a connection between its bad representation power on these functions with some of the BBOB function properties (such as multimodal level)?\n2.\tIn Table 4, is there other performance measures (except for macro F1 score) can be introduced to quantify the classification results on feature representations from different methods? And it would be more convincing to have multiple executions and report a mean value with variance.\n3.\tIn general do you think your method would increase the computing complexity & time compared with standard ELA for downstream meta-learning tasks?\n4.\tAll in all, I think there are more experiments with repeated runs can be done to make the paper more convincing to the audience.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject