Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces a variational autoencoder-based method to learn optimization landscape characteristics for downstream meta-learning tasks. This method uses large training sets to self-learn a latent representation for any design of experiments. Four experiments were designed to evaluate the quality of learned variational autoencoders and their effectiveness on downstream meta-learning tasks.
strength_and_weaknesses: Strength:

- The paper is well-motivated. 
- The application of variational autoencoder in exploratory landscape analysis seems to be novel.
- Four experiments were designed to show the effectiveness of the proposed solution.

Weakness:

- This paper directly applies variational autoencoder to exploratory landscape analysis tasks. There lack more in-depth theoretical analysis of why and how this method works well on these tasks.
- The experiments lack comparison with other approaches.
clarity,_quality,_novelty_and_reproducibility: This paper has good novelty and clarity. However, the theoretical aspects of the method shall be enhanced. The work is reproducible as the required resources to reimplement the work are available online.
summary_of_the_review: This paper has clear strengths and weaknesses. Its quality can be improved with more theoretical contribution and more experiments to demonstrate the advantages of the proposed method.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposed a VAE-based approach, called DoE2Vec to learn the latent representation approximating the optimization landscape. The authors showed the latent feature vectors that is learned by the DoE2Vec network can reconstruct and predict high level properties of the function landscapes. They also showed that the proposed method can be used in combination with existing exploratory landscape analysis (ELA), to further improve the classification accuracy of some downstream tasks. The authors reported the effectiveness of DoE2Vec using Black-Box Optimization Benchmark (BBOB) function landscapes. 


strength_and_weaknesses: Strength:

- The proposed VAE-based methodology seems to reconstruct properly most of BBOB functions. 

Weaknesses:

- Design of experiments in general is often suited for multivariable analyses and exploring causal relationship. One can test and optimize several variables simultaneously, thus accelerating the process of discovery and optimization. I do not see any analysis which discusses how latent representation of Doc2Vec is suitable for such tasks. It is not clear how each latent vector effectively represents such relationship.

- The reported results are only demonstrating the performance of the latent representation in reconstruction and prediction of properties of a specific set of functions. It seems more like an AE-VAE analysis paper which is evaluating their performances in reconstructing a set of functions.

- I think one important missing aspect is traversal analysis and quantifying the disentanglement of latent factors to show how a disentangled representation can be used to explore the function landscape.  

- There is no quantification that shows how Doc2Vec performs against competing algorithms. 


Questions / Concerns :

- I do not understand how the proposed approach can be used for downstream meta-learning task?

- What is novel about the results in Figure 2? Adding more dimension to the latent space improves the loss, and a lower beta provides better reconstruction error.  Is that not something already known from $\beta$-VAE?

- Can you elaborate more on the plots in Figure 5?

clarity,_quality,_novelty_and_reproducibility: Clarity:
The paper is well organized.

Quality and Novelty:
The paper does not have any methodological originality and lacks sufficient contribution. 

Reproducibility:
The code is not available. I cannot assess the reproducibility.

summary_of_the_review: The authors introduced a VAE-based approach for function landscape analysis which is an interesting and challenging problem. However, I think the authors should justify the role of autoencoders and their latent representation in the ELA task and discuss the role of each latent factor in function landscapes and clarify why this model is expected to outperform the other existing ELA techniques.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a new method to learn optimization landscape features using latent space information with AE/VAE for downstream meta-learning tasks.
strength_and_weaknesses: 
Strength:
1.	The DoE2Vec can reconstruct a large set of functions and can be applied to downstream meta-learning tasks.
2.	The combination of ELA and VAE-32 outperforms all other methods in table 4, which shows the new method can learn a good feature representation for optimization landscapes.
Questions/Weakness :
1.	In Table 2, the VAE-24 model seems cannot represent f16, f21 and f24 well which have medium or high multimodal property. Is there a connection between its bad representation power on these functions with some of the BBOB function properties (such as multimodal level)?
2.	In Table 4, is there other performance measures (except for macro F1 score) can be introduced to quantify the classification results on feature representations from different methods? And it would be more convincing to have multiple executions and report a mean value with variance.
3.	In general do you think your method would increase the computing complexity & time compared with standard ELA for downstream meta-learning tasks?
4.	All in all, I think there are more experiments with repeated runs can be done to make the paper more convincing to the audience.

clarity,_quality,_novelty_and_reproducibility: 
The paper is well-written and well clarified, and original as far as I see.


summary_of_the_review: All in all, the paper has its novelty and contains a lot of simulations to verify the advantage of their method. However, I would recommend to include more numerical results, such as repeated executions and more performance measures.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In "DoE2Vec: Representation Learning for Exploratory Landscape Analysis" the authors propose a variational auto encoder (VAE) to learn a latent feature representation of black-box optimization problems. In the experimental section, the authors show that this VAE approach is best combined with exporatory landscape analysis (ELA) to obtain the best results. It is argued that the learnt representations can be then used in downstream (meta-learning) tasks such as algorithm selection.
strength_and_weaknesses: [+] Designing useful, complementary, and informative features is one of the major challenges in downstream tasks like algorithm selection and therefore reasonable feature-free or automatic feature design approaches are of relevance.

[-] It is claimed that the proposed method facilitates downstream tasks but there are no results contained in the paper to support such claims. More specifically, it is only demonstrated that the method can retrieve random functions which look similar to the actual target functions based on a set of evaluations of the target function in question. While this may indicate that the retrieved random functions may be used as a surrogate it remains open whether (a) the random function is indeed faster to evaluate, (b) the random function is sufficiently close to the target function, and (c) whether the method gives any advantage over standard methods that could be applied directly.

[-] One of the example downstream tasks is algorithm selection. However, it is not clear how the method would be mapped to the setting of algorithm selection where for each problem instance an algorithm needs to be picked. If a problem instance is a bbob function from the paper, one would need to observe objective function values for some algorithms in order to obtain the feature representation. However, if algorithms are already run, the current problem instance is already solved in the best case so that no features are needed anymore. Otherwise, the feature computation would already take too long.

[-] The proposed method is not very innovative. It just applies VAE to some vectors sampled from specific functions. In combination with the limited experimental section regarding the downstream tasks or at least some case studies of how to employ and practically use the approach for downstream tasks, the paper is of little help, I would say.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and the overall structure is good.

The paper simply applies VAEs to vectors of black box functions to retrieve some surrogate function. While for exploratory landscape analysis this might be novel to some degree, the merit of the approach does not get across.

Overall the authors try to give as much details as possible. However, it is not clear in what form the random functions are represented and how they can be accessed in downstream tasks. 
summary_of_the_review: Overall, I think this could be an interesting direction to go for certain BBO tasks but still it is rather unclear to me how the ASP problem should be reasonably implemented with this approach. Also the claims that downstream tasks are facilitated etc. are not proved in this paper. Furthermore, the proposed method is rather an application and more about the dataset used so that the novelty of a methodological perspective is comparably low and questionable to what extent this contribution may impact the community. Therefore, I recommend to reject the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper explores a methodology based around using a VAE for learning representations of optimization landscapes, with an aim to improve downstream performance e.g. meta-learning of optimization algorithm selection. The paper is interesting and nicely presented, but in its initial form was not sufficiently convincing to reviewers — it was not quite obvious that the experiments in Table 4 of the current draft corresponded to state of the art benchmarks, nor is it quite clear how to interpret the magnitude of these results. 

I think this is quite a bit improved from the edits made during the review process (and clarified by the author response), but probably not enough yet to recommend acceptance as-is.

Overall this is a topical paper which will be likely be accepted on resubmission if there is a bit more focus in the presentation on the performance and applicability to downstream tasks (and less on the VAE or AE itself).
justification_for_why_not_higher_score: None of the reviewers argued for acceptance.
Decision: Reject