Title: Generative Evolutionary Strategy For Black-Box Optimizations
Authors: Changhwi Park
Seong Ryeol Kim
Young-Gu Kim
Dae Sin Kim
Venue: NeurIPS 2022
Keywords: Optimization
Blackbox
Generative model
Evolution

Abstract: Many scientific and technological problems are related to optimization. Among them, black-box optimization in high-dimensional space is particularly challenging. Recent neural network-based black-box optimization studies have shown noteworthy achievements. However, their capability in high-dimensional search space is still limited. This study proposes a black-box optimization method based on evolution strategy and generative neural network model. We designed the algorithm so that the evolutionary strategy and the generative neural network model work cooperatively with each other. This hybrid model enables reliable training of surrogate networks; it optimizes multi-objective, high-dimensional, and stochastic black-box functions. In this experiment, our method outperforms baseline optimization methods, including , including evolution strategies, and a Bayesian optimization.

TL;DR: We designed a black-box optimizer for high dimensional search space. It is a cooperative strategy of evolution and generative neural network.

Review discussion:
The reviewer did not see similar work for addressing the stochastic multi-objective high-dimensional optimization problem.
Though through related works, there are L-GSO, EGAN and other works, the authors clarify their differences inbetween.
The organization of the technique part of this manuscript should be considerably improved. For the reviewer, it is quite difficult to get how GEO exactly works and the introduction of sub-routine techques are quite vague, e.g., how generative model generates candidate solutions.
Besides, the empirical studies section is also problematic. It is unclear why Cartpole-v1 and LeNet-5 are used for verification and how to fairly assess the quality of the GEO based on them.
For example, why not compare GEO with RL methods on Cartpole-v1? What is Cartpole-v1? How representative it is? Why we have to regard it as a black-box optimization problem if some RL methods can solve it better with MDP structure. And it is also unclear what information has been delivered to the readers in section 4.4.
Though this work focuses on the  stochastic multi-objective high-dimensional optimization problem, it is not clear what real-world problem is with these features and there seems no empirical studies on the problem with all these features.
Thus, it is difficult to judge how significant the proposed GEO is.
1. Please name a few real-world applications involving stochastic multi-objective high-dimensional optimization problems. The current experiment studies seem to consider 4 benchmarks each touching one or two aspects of the above problem features. It is interesting to see an empirical case study on the stochastic multi-objective high-dimensional optimization problem.
2. How Evolution Strategies contribute to GEO? Section 3.4 states that ES can help stablize the training, but why is this case? Please clarify this part.
Besides, the ablation studies of verifying the importance of ES in GEO is missing.
3. How generative model generates candidate solutions? The review suggests first introducing the general framework of GEO at the begining of section 3, so readers can easily get how GEO works. For the current manuscript, it is hard for those whoe do not farmiliar with L-GSO and EGAN to understand the GEO, even for the basic idea of it.
4. What is network structure and hyper-parameters of the generative models and critic models?
5. For Cartpole-v1, why not compare GEO with RL methods if RL methods perform better?
Also, it would be much better to briefly introduce what is Carpole-v1 scenario.
6. It is not intuitive what does section 4.4 want to show about.
No.
No
Thank you for your sincere advice. Your questions get to the point.
I had to explain my idea in more detail.
Q1. Name of real-world applications
The real-world problem is searching Best/Worst performances of electronic-circuit designs.
Specifically, they are Verilog (device) simulators. GEO was developed to solve this problem.
The Best/Worst performance of a new device must be evaluated before mass production. If we find unexpected bad performance after mass production, it would be a disaster.
1. Verilog simulators predict performance of electronic chip on multi targets (Power, Latency, …)
2. Generally, the input is a sequential command of length 1,000~10,000
3. Target scores have random noise
It is a stochastic multi-objective high-dimension problem.
Q2. How evolution strategies contribute to GEO?
In the early development of GEO, we did not try evolution strategies.
However, we found that a simple generator-critic network model easily diverges to INF.
Also, it has no optimization performance at all.
Q2.1 Why evolution stabilize critic networks?
The divergence is a result of evil cycles of two networks.
1. The generator suggests an input variable X in a wrong direction
2. The critic network is trained with input variable X, but X has no information of a Pareto front
3. The critic network trains the generator, but it does not give meaningful information
The role of the evolution strategy is to ensure that the generator and its corresponding X are trapped near the Pareto front. Near the Pareto front, the set of X can be the best training region for the critic network.
Q2.2. Ablation study
WO evolution, it diverges from the start, and it has no optimization performance. We thought that performance comparison is impossible.
In the related studies (L-GSO : local sampling method, COMs : regularization method), they did not show ablation test, but we guess that they also had similar problems.
Still, we agree that it is much better to show ablation. It should be added in the supplement to show how the model diverges fast.
Q3. How generative model generates candidate solutions?
1. A generator is randomly sampled from the pool
2. The critic network trains the selected generator (using backpropagation, to increase prediction of critic)
3. The trained generator suggest a new variable X=G(z)
4. Check score = F(X)
5. Sort a new (G, score) set in a pool
Because we tried to explain the algorithm with a schematic figure and a pseudo-code, we wanted to avoid too much redundancy.
However, if you were hard to understand, it would be my fault not to make an easy explanation.
Q4. What is network structure and hyper-parameters?
The network structure is a stack of self-attention models. It is a modification of original “Transformer” model. The overall structure is illustrated in Figure 2 and Figure A.1 (supplement)
Hyper-parameters of the self-attention model can be found in the supplement.
In the answer of Q1, we mentioned that the real-world problem has sequential input structure. This is another reason why we used attention mechanism for networks.
Q5. For Cartpole-v1, why not compare GEO with RL.
Also, it would be much better to briefly introduce what is scenario
Cartpole has time sequential I/O structure like
Input : {Action_0, … Action_t, …}
Output : {(State_0, Reward_0),  …, (State_t, Reward_t), …}
If RL can observe all the output information, RL will be much better to solve Cartpole
However, my scenario is :
What if we cannot observe time sequential data because the cost of simulator is too high?
What if the observable information is just a single final score? (ex. Score = sum of all rewards)
Then, RL cannot be used to solve this problem.
As I mentioned in Q1, The real-world problem is optimizing Verilog simulation and it is very heavy.
Some people believe that RL can find Best/Worst case of electronic devices because simulators have sequential I/O structures.
However, this problem is not that simple.
Let’s consider simulation time is 1h, and steps are 10,000.
Then, the total time will be 10,000h just for one iteration.
We thought that a black-box optimizer is the only solution for this scenario, so we developed GEO.
This is why we tested Cartpole as a toy model.
Q6. What does section 4.4. (image generation test) want to show about?
In the related work L-GSO, authors mentioned that the performance of L-GSO is better in sub-manifolds.
Because the generator-critic work flow is similar to L-GSO, we guessed that the performance of GEO would be better in a low-dimensional manifold.
Also, we thought that image generating problem is the simplest way to show manifold structures.
Therefore, we expected that GEO will generates blurred image of numbers (because a smooth image is a low-dimensional manifold structure).
However, results seem to be noise shape, not manifolds.
There must be difference between GEO and L-GSO, but we do not clearly find what makes difference.
About real-world problems
In the Q1 and Q5, I explained what "real-world problem" is.
Unlike academia, companies are reluctant to disclose their technology. It was hard to write specific stories about real-world problems in the paper. This is why we only used the test functions, widely known.
I believe you will understand that GEO was developed to solve empirical problems.
There were many changes in the revision.
In particular, the algorithm description has been improved.
Additional explanations were added according to the comments of the reviewer, and unnecessary explanations were removed.
The background of the Cartpole-V1 experiment and the purpose of the LeNet-5 experiment were further explained.
Data has also been added.
thank you
The authors responded that "The role of the evolution strategy is to ensure that the generator and its corresponding X are trapped near the Pareto front". This is not intuitive to me. What if we replace ES with other search methods? Because the title of this paper is called "generative evolution strategy...", it is expected to see clearly why ES here is a must, not an alternative.
In this regard, the  Ablation study can be the GEO with ES replaced by other search methods like Bayesian Optimization, or PSO.
Many RL are tested on video games that are regarded as black-box simulators, i.e., no additional information of the transition model of the video games is used to train RL.
So I do not think "RL cannot be used to solve this problem".
At least, a non-customized RL method can be applied to show the baseline performance of this type of methods.
"Let’s consider simulation time is 1h, and steps are 10,000.
Then, the total time will be 10,000h just for one iteration." I do not understand this  statement. For RL, the 10000 sequential actions will be performed in one simulation, not 10000 simulations. And gradient descent may quickly converge to an optimum and thus reduce the iterations. And this statement seems like a drawback of ES that ES usually needs large number of iterations.
Thank you for your answer.
This problem happens when the cost of observation is higher than the cost of operation.
And it is strongly related to the real-world problem I mentioned earlier.
A "Verilog simulator" is an electronic device simulator, and it has a time-sequential I/O structure.
(There are many different types of simulators and device testers, but for the sake of simplicity, we will call them "Verilog".)
In this simulator, the I/O structure is defined as a time sequential structure as follows.
Input : $[Action_1, Action_2, ... ]$
Output: $[(State_1, Reward_1), (State_2, Reward_2), ...]$
And training of RL is done as follows
$Action_1$ $\rightarrow$ $Observe(State_1, Reward_1)$ $\rightarrow$ $training$ $\rightarrow$ $Action_2$ $\rightarrow$ $Observe(State_2, Reward_2)$ $\rightarrow$ $training$ $\cdots$
So some people think that this problem can be solved with RL.
However, in the Verilog, observing the time-sequential data requires much more cost than observing the sum of entire result at once.
This is because, in the Verilog simulation, the observation cost of each time sequence $(State_t, Reward_t)$ is almost equal to the observation cost of the $score = \sum (rewards)$ at the final time.
In other words, when the time sequence length is N, the cost of $[(State_1, Reward_1), (State_2, Reward_2), ...]$ observation is almost N times of the observation cost of $score = \sum (Rewards)$.
This means that we cannot practically observe time-sequential data of a Verilog simulator.
In addition, $score = \sum (Rewards)$ is much more valuable data than each $(State_t, Reward_t)$.
Our Cartpole experiment is a toy model for this situation.
We assume that the only observable information is $score = \sum (Rewards)$ rather than $[(State_1, Reward_1), (State_2, Reward_2), ...]$, as in the Verilog situation.
We are well aware that Cartpole-V1 does not have the observation cost problem, but we use Cartpole-V1 as a toy model to describe what happens in the real-world problems.
PS.
When we talk about "cost", it is not only a time cost, but also an expense in dollars.
Imagine that you need to optimize a mass production process in a factory.
Observing the intermediate process incur huge costs (because the manufacturing process must be stopped for observations). This is the big difference between a simple simulator and a real problem.
In short : The more data you observe, the higher the expense is (in dollar). In this case, it is very important to observe only the final score which has the valuable information.
We updated additional explanation of the observation cost problem in the supplement.
I think it's also a good idea to use BO or PSO instead of ES.
However, these cases will be very different from the original idea of GEO, and we will need to devise a new mehtod for the combination strategy of the generative model and BO (or PSO).
For example, the core idea of BO is an exploit & explore strategy through likelihood scores.
(This method can be very efficient, but it has the disadvantage of having non-linear complexity.)
When combining a neural network with BO, we can think of a neural network model that can estimate the maximum likelihood score.
This is can be a good approach, but it's too different from the core idea of GEO.
Meanwhile, the core idea of PSO is a combination of single particle velocity and group velocity.
The velocity is explicitly defined by the score and the variable $x$, so I'm not sure how to combine them with a neural network.
One method might be to alternate between PSO and surrogate-model based optimization (SO).
$PSO$ $\rightarrow$ $SO$ $\rightarrow$ $PSO$ $\rightarrow$ $SO$ $\rightarrow$ $\cdots$
However, this is not a combination method, but this is a simple alternating scheme.
GEO is not a simple alternating method of
$ES$ $\rightarrow$ $SO$ $\rightarrow$ $ES$ $\rightarrow$ $SO$ $\rightarrow$ $\cdots$
Because GEO's evolution and generative models are parts that cannot be used in isolation. (The evolutionary algorithm use generators ($G$) as elements, not variables ($x$).
Meanwhile generators diverge without evolution)
So right now, I haven't figured out how to apply BO or PSO to SO, but we can try them in our next study.
Thanks for the good idea
Literature Review:
Though this paper targets high-dimensional black-box optimization, there is almost no literature review on this topic.
For example, in evolution strategies, which is the basis of the proposed approach, different approaches have been proposed for high-dimensional problems, such as SEP-CMA [1], VD-CMA[2], LM-CMA[3], etc.
[1] A Simple Modification in CMA-ES Achieving Linear Time and Space Complexity, PPSN 2008
[2] Comparison-Based Natural Gradient Optimization in High Dimension, GECCO 2014
[3] Lm-cma: An alternative to l-bfgs for large-scale black box optimization, Evolutionary Computation (2017)
Moreover, though one of the contributions of this paper is a surrogate-assisted evolution strategy, no literature review has been performed on this line. See for example [4, 5] and references therein.
[4] A global surrogate assisted CMA-ES, GECCO 2019.
[5] pysamoo: Surrogate-Assisted Multi-Objective Optimization in Python, arxiv (2022)
Multi-objective optimization has also not been reviewed.
Experimental Evaluation:
The baseline approaches are very limited, as the literature review has not been performed well. NSGA-II is a traditional GA based approach for mainly bi-objective optimization. A lot of improvements are reported. Surprisingly, it has been applied to a single objective optimization in Figure 3, which is not understandable. CMA-ES is basically an approach for single objective optimization, and there are several extensions of CMA-ES to multi-objective optimization, but it has not been explained which algorithm is used.
No surrogate-assisted evolution strategies or other surrogate-assisted approaches are compared in the experiments. No state-of-the-art evolutionary multi-objective optimization approaches such as MOEA/D or NSGA-III are compared. I suggest you to look at BBOB BiObjective Testbed and algorithms reported there (https://numbbo.github.io/data-archive/bbob-biobj/).
Because of the lack of the literature review and comarison with the state-of-the-art approaches, I can not judge the goodness of the proposed approach.
The algorithm description looks strange to me. What do you mean by “while n = N do” and “while m = M do”?
See the comment above. The lack of literature review and the lack of experimental comparison.
No
We really appreciate for your valuable review. The revision has taken your comments into consideration.
We would like to correct a minor misunderstanding of your summary, and explain why we did not test the algorithm you suggest.
A.1 About high-dimensional optimization algorithms [SEP-CMA, VD-CMA, LM-CMA].
I understand they are developed for high-dimensional problems.
However, they are designed to optimize (high-dimensional) convex functions, not (high-dimensional) non-convex functions.
The optimization of convex and non-convex function is totally different, because convex functions can be easily optimized by gradient descent method.
Most of ML algorithms uses modern gradient descent methods (such as Adam optimizer). They can optimize target functions in million, billion, and trillion dimensions.
However, we don’t say the them high-dimensional black-box optimizers.
We should distinguish convex problems and non-convex problems.
A.2 About surrogate assist models
In the section 1 and section 4, we mentioned that it is important to ensure O(N) (linear) computational complexity in high-dimensional optimization problems.
Surrogate assisted models might be good approaches for the next generation optimization strategy, but their computational time is not linear as far as I know. To make sure this, we have recently rerun the experiments in haste.
As a result, we reconfirmed that the models do not have linear complexity and incur enormous costs at high dimension or large function calls.
In the worst case, the time cost is too high, so we cannot optimize at all even in the low dimension (d ~ 100).
In short, computational complexity should be considered first, because in extremely high dimensions such as 10,000, enormous function calls are required.
This is a big difference between classical ES algorithms and surrogate assist models.
A.3. NSGA2, CMAES and the single-objective optimization.
You mentioned as if NSGA2 algorithm should not be used in the 1-objective problem. I understand your comment, as CMAES is known to outperform NSGA2 (in general).
However, it is not always true.
(In our last submission, we showed CMAES results in the supplement.
In this revision, we plotted the CMAES results in figure 3 and compared them with NSGA2.)
As you can see in the single-objective optimization experiment (Figure 3 and supplement), CMAES is not always better than NSGA2.
In the experiment (Styblinski-Tang), NSGA2 is the better one.
If we had used the Ackley function as a black box, CMAES would have performed better.
The performance of optimization algorithm is (very) strongly correlated to initial states, boundary conditions, target functions, and hyper-parameters of algorithm.
For this reason, it is impossible to guarantee that a particular algorithm is a better than others.
Also, in the practical problems (not just a test function), we have experience that NSGA2 is also a good optimization algorithm in 1-objective problems.
This is a reason why we used NSGA2 instead CMAES.
A.4 NSGA3, MOEAD, and others.
From the early stage of study, we already tested [NSGA3, MOEA/D, …], but we concluded that there is no need to show all results in main paper. Let me explain why.
NSGA3 and MOEA/D might be better than NSGA2 in low-dimension because NSGA3 and MOEA/D have reference direction assistances which NSGA2 does not have.
However, in the extremely high-dimensional space, the performance between NSGA3 and NSGA2 becomes indistinguishable.
It is because they are easily trapped in local-optimum (near the initial state) when the dimension increases to [d >> 1,000]
In the revision figure 4 and 5, you can find that NSGA3 and MOEA/D often show worse performance than NSGA2.
This is why we plotted NSGA2 instead NSGA3 and MOEA/D in the first submission.
It seems clear that NSGA2 is better in this experiment, but we also added NSGA3 and MOEA/D for comparison.
(I also agree that more data is better somehow.)
A.5. 2-objective optimization and CMAES
How to optimize 2-object with CMAES : A simple alternating method:
Optimize f1 -> optimize f2 -> optimize f1 -> optimize f2 -> ...
It requires additional hyper param.
But, we agree that NSGA2, NSGA3 and MOEA/D could be better than CMAES in general (especially for low-dimensions).
So, we just corrected it in Figure 4 and 5.
Ps.
Although we explained the reason why we did not test algorithms you suggested, we agree that it is also important to show data as much as we can.
The revision includes more data and related studies based on your comments.
Thank you
1. Additional answer.
About the purpose of the experiments.
It seems to be there is a little misunderstanding of the purpose of this experiment.
The experiments in the paper are NOT intended to compare the performance of algorithms and determine which algorithm is SOTA.
Let me explain the reason.
Let’s consider we have a set of Bayesian optimization (BO), Reinforcement learning (RL), evolution strategy (ES).
The question is follows.
“Is it possible to compare performances of BO and ES?”
“Is it possible to compare performances of ES and RL?”
“Is it possible to compare performances of RL and BO?”
The answer will be “No”
It is because the goal of BO and ES seems similar, but actually different.
BO is designed to optimize black-box for the case of high cost functions.
Meanwhile, ES is designed to optimize black-box for the case of low cost functions.
If the number of function calls N < 100, BO will be better in most cases.
On the contrary, if N > 10,000, ES will be better in most cases.
Therefore, It is not strange idea if we want to see how the performance of BO and ES changes in each dimension.
However, if we try to compare performance between BO and ES, it will be very strange idea. Metaphorically speaking, this is equivalent to a performance comparison between a language model and an image model.
The algorithms of your comments are ES, and they have the goal which is :
“Searching global optimum of black-box in low-dimensions”.
And it is well known that ES optimization becomes very difficult when the dimension exceeds 100. (I think that you already know about it, because in the materials you gave, most of the test dimensions are less than 100.)
In the paper (and supplement), we show that GEO has much worse performance in low-dimension.
It is because we designed GEO to have the goal :
“Searching local optimum in extremely high-dimension (~10,000)” (It is not low-dimension nor global optimum)
And it is totally different from the goal of classical ES algorithms.
Therefore, we thought that comparison between GEO and ES can be inappropriate, just like that of BO and ES.
Also, from the experiment, we wanted show you :
“What goals GEO can achieve?. How they different?”
GEO : Better in high-dimension, worse in low-dimension
ES: Better in low-dimension, worse in high-dimension
In summary, since the goal of GEO and [CMAES, NSGA2, NSGA3, MOEA/D, ...] are different, I see that the experiment to compare performance between them is unsuitable just like that of BO and ES. Because such experiments may give the false message that GEO is a better algorithm.
2.
About experiments
For the above reasons, in the first submission, we thought that it was only important to show the trend of ES and GEO according to the dimensions.
Since it is known that ES rarely works at high dimensions, we thought it was sufficient to show only NSGA2 (the best result found in the experiment).
However, these attempts seem to make misunderstandings.
So, we've detailed the reasons for your comments, and we've incorporated them into the revision as well.
3.
About SOTA of ES.
The performance of optimization algorithm is (very) strongly correlated to initial states, boundary conditions, target functions, and hyper-parameters of algorithm.
In the test function optimizations, researchers already have found the best combination to make SOTA.
However, it is impossible in the real-world problems to find the best combination for an algorithm, because the real-world problems are very expensive in general.
For this reason, there is a risk that the (so-called) SOTA of optimization studies can be biased toward the test function.
I think this is why the performance of the black box algorithm is not alway as we expected.
In my opinion, we have to be careful in saying what SOTA is.
4.
About summary
In the paper, we suggested that input seed z can be random variable or constant. In the short summary of yours, you summarized that we have used latent z, but we did not told that z is a latent vector.
It is not make sense that z is both a latent variable and a constant. In the paper, we explained that most of GAN algorithms feed random latent variable, but some GAN algorithms do not.
Likewise, we also do not see z as a latent vector, we see it a simple input feed.
5. Typo correction: I found [while n=N], but it is [while n<N] or [for n in range(N)]. I’m really sorry for confusing you
Originality:
The paper seems to be original, though I'm not extremely familiar with the SOTA on multi-objective high-dimensional blackbox optimization.
Quality:
The paper has interesting experiments.
However, there are many simple baselines that seem relevant for blackbox optimization of high-dimensional algorithms.
In particular, some well-known works include:
Salimans, Tim, et al. "Evolution strategies as a scalable alternative to reinforcement learning." arXiv preprint arXiv:1703.03864 (2017).
Such, Felipe Petroski, et al. "Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning." arXiv preprint arXiv:1712.06567 (2017).
Sehnke, Frank, et al. "Policy gradients with parameter-based exploration for control." International conference on artificial neural networks. Springer, Berlin, Heidelberg, 2008.
In particular, OpenAI ES and PGPE are similar to this approach in that they estimate a gradient. These papers could be a more fair comparison since the baselines used in the paper are notoriously poor at high-dimensional optimization.
Furthermore, the methods are fairly unsophisticated, making them easy to compare to.
Furthermore, I'm not sure what the purpose of the LeNet-5 experiments is, seeing as there is no baseline comparison.
Clarity:
The paper is poorly structured and written to the point where it is quite difficult to understand. See the "Questions" section for the many points of uncertainty.
There are numerous grammatical errors that make it difficult to read.
In particular, the paper often either misuses or is missing articles (words like "the" and "a").
The pseudocode uses "while n=N", which is not at all clear. I'm assuming that they mean "for n $\in$ N".
The authors often use imprecise language, which can cause confusion.
For example, in Line 119 they write about "accidentally-high-scoring elements". I'm assuming that they are referring to points that are high due to stochasticity -- however, calling those "accidental" is imprecise/incorrect.
Significance:
The approach is interesting and seems to perform well -- however, with the missing baselines and lack of clarity on the details, it is very difficult to evaluate its significance.
1. Line 107 the authors write: "We experiment with both random variables and pseudo-random constants as input feeds z". Where are the results? If it is in a plot or figure, please reference it (I could not find which one it is referring to). If they are not, please at least mention what the results of the experiments were. What do you mean by "pseudo-random constants"? How is it different from a "random variable?" Are the random variables somehow not generated by a "pseudo-random" number generator? If $z$ is constant throughout training, then is it really a "generative" method? How would optimizing $G$ be any different than optimizing "x" directly?
2. Line 140 the authors write: "We experimented with both stochastic and non-stochastic generators". Again, I'm not sure where in the paper the results are.
In particular, this is an interesting experiment: If non-stochastic generators worked, then this wouldn't really be a generative process. Can you get away with not using generators and instead just having a large population?
3. How do you measure pareto efficiency in the paper? In the pseudo-code "ParetoEfficiency" is written. In the paper, on Line 117 the authors write that data are "sorted and selected by Pareto efficiency". How exactly is this calculated?
4. Similar to question 3, there are simply many parts of the paper that are poorly explained, so much so that I cannot list out every question that I have. Here are a few:
A. Line 19: I understand that there are multiple objectives to optimize, but how are these objectives combined? The authors simply write "Optimize" without explanation. Do you take the min? Do you take the sum? If they're combined, how is it different from a single objective? Or instead, are you looking for multiple $X$ / trying to find the best $X$ for each of the $m$ functions?  I assume you take the min, but this is not made clear in the paper.
B. Line 95: "GEO uses only searched points without prepared data". I think I have a good idea of what this means, but it is not made explicitly clear at all. What do you mean by "prepared data?"
C. Line 113: "The critic network learns variable $x$ in a global region. Global training is essential". What does this mean? What would be the alternative approach? Again, while I think I have a decent guess of what the authors are trying to convey, it is not completely clear.
D. In Cartpole-V1 is the random seed fixed? My understanding is that cartpole's initialization is random, so observing the state is required.
The authors seem aware of the limitations of the work. They mention the "Pareto front collapsing problem" and the "GPU memory consumption problem".
No
I really appreciate your valuable questions and advices
Typo correction:
I found [while n=N], but it is [while n<N] or [for n in range(N)].
I’m really sorry for confusing you
Q1.
A1.1. Where random z and constant z used?
Figure 4 is non-stochastic black-box experiments. We used (pseudo random) constant seed z in it.
Figure 5 is stochastic black-box experiments. We used random seed z in it.
A1.2.
About pseudo random “constant“ :
The seed z is initialized with random, after the initialization, we regard z as a constant.
In the most of GAN algorithms, the seed z is random variable. (StyleGAN uses constant z as its seed of CNN)
It is because the generator of GAN must generate various images with a single generator. They regard seed z as a latent space, and create multi images using the latent space.
However, GEO has a pool of generators. There is no need for generators to have latent space. Because the parameter of generator always changed by critic, and the mutated generator always creates a new output X.
Also, the goal of optimization is not generating various outputs. The goal is finding Pareto-fronts.
Therefore, we don’t have to regard z as latent space, and the seed doesn’t have to be random.
A1.3. The reason why we used random & constant seeds.
We can always use random seed z.
Instead, we have to refresh the pool periodically. The pool refresh process makes computational time almost doubled. Age-evolution is another option but, it makes the performance degraded.
Therefore, we used constant z when we can.
However, in the stochastic black-box, we have to refresh the pool regardless of z.
So we used random z in Figure 5.
A1.4. How would optimizing G be any different than optimizing "x" directly?
In the paper, we show that the depth of the generator is important.
We also proved it by the experiment "1-layer generator test".
In the related works section, we introduced GLOnet to emphasize the importance of the deep generator.
Also, we explained the reason.
We guess that the good result comes from the manifold searching ability of generative models.
Q2.
A2. This question is the same as Q1.
If we use random z : stochastic generator (Figure 5)
If we use constant z : non-stochastic generator (Figure 4)
As I mentioned in A1.2, we don’t need latent space. It is because the goal of optimization and GAN are different. (see A1.2)
The constant z method will have trouble only when we need “inference”.
But I don’t find any reason to have inference.
Q3. How to measure Pareto efficiency
A3.
A simple example of Pareto-efficiency (a case of maximization of a 2-objective function).
Let ($f_i$, $g_i$) ∈ S
If there is no ($f_j$, $g_j$) in S that satisfies $f_j$ > $f_i$ or $g_j$ > $g_i$
Then, ($f_i$, $g_i$) ∈ P, where P is the Pareto front.
And how to sort :
Def : P = Pareto(S), where S is an original set, P is Pareto front subset of S.
P1 = Pareto(S)
P2 = Pareto(S-P1)
P3 = Pareto(S-P1-P2)
Sort : subsets in order [P1, P2, P3, …]
FYI.
Pareto efficiency is very famous concept, so you can easily find codes in Google.
The python code that I recommend you : https://stackoverflow.com/questions/32791911/fast-calculation-of-pareto-front-in-python
It will be easy to understand.
I also recommend to read very simple explanations:
https://www.statisticshowto.com/pareto-efficiency/
https://www.economicshelp.org/blog/glossary/pareto-efficiency/
Q4.A.
"Pareto-efficiency" is the method. The Pareto-efficiency is a widely used method for evaluating and optimizing multi-objective targets.
In the paper, we mentioned that we used Pareto-efficiency for sorting of scores.
A simple maximize or minimize in 1-object is also a Pareto-efficiency of a 1-object.
"Take min, or sum" is not a multi-objective optimization.
Q4.B.
In the related work EGAN, they have dataset. The dataset is called true data in GAN concept.
However, optimization problems does not have true data.
In other words, GAN has true data from the start point, while optimization algorithms does not. It is the meaning of “without prepared data”.
Q4.C.
The related work L-GSO is an optimization method for 1-object. In the paper, they used a local-sampling method. The center of local sampling is determined by “current point” (currently optimized point), and it is exactly same with the Pareto front of 1-object.
The dimension of Pareto front = N-1, where N-object blackbox
Therefore, the local sampling method has only one sampling region.
However, if N >=2, there will be numerous sampling centers since the dimension of Pareto front is N-1.
This is why we cannot use the local sampling method when N>=2, and this is why training data should be in global region. (Pareto front is not a single point anymore)
Q4.D.
Random seed is NOT fixed.
We tried to use a stochastic environment, because we thought that a stochastic environment is better to evaluate performance of optimizers. (by assuming a very difficult problem)
The purpose of the LeNet-5 experiments.
This experiment is not for baseline comparisons.
In the related work L-GSO, authors mentioned that the performance of L-GSO is better in sub-manifolds.
Because GEO's generator-critic work flow is similar to L-GSO, we guessed that the performance of GEO would be better in a low-dimensional manifold.
Also, we thought that image generating problem is the simplest way to show manifold structures.
Therefore, we expected that GEO will generates blurred image of numbers (because a smooth image is a low-dimensional manifold structure).
However, results seem to be noise shape, not manifolds. There must be a difference between GEO and L-GSO, but we do not clearly find what makes the difference.
About RL papers
I'm reading the paper you gave me, and it's interesting. And I see that the concept of evolution is partially related.
But as far as I understand, it is about reinforcement learning.
RL and black-box optimization have completely different problem definitions. (ex. I/O structure of environments)
Therefore, it is inappropriate to compare RL and black box optimizations.
To be clear, black box optimization and RL have completely different problem structures and should be clearly distinguished.
FYI, why did we used Cartpole-v1 toy model?
Let me explain the reason for the Cartpole-v1 experiment.
The real-world problem where GEO is used is the evaluation of electronic device performance.
The target function is a Verilog device simulation.
Since this simulation has a time-sequential I/O structure, it looks like it could be optimized with RL.
The problem, however, is that the cost of the simulation is too high to observe all of the time-sequential data.
In practice, it is only possible to observe the final score, which is the sum of the rewards.
Then, the I/O structure is not time-sequential structure anymore.
Therefore, this problem should be solved by black box optimization techniques, not RL.
The Cartpole-v1 experiment is a toy model for this situation.
There were many changes in the revision.
In particular, the algorithm description has been improved.
Additional explanations were added according to the comments of the reviewer, and unnecessary explanations were removed.
The background of the Cartpole-V1 experiment and the purpose of the LeNet-5 experiment were further explained.
Data has also been added.
thank you
## On Pareto Efficiency
I believe I phrased my question regarding Pareto Efficiency poorly. I'm familiar with Pareto Efficiency, but not with the author's usage of the term in the paper. In the code and examples given (and in its most well-known form) Pareto Efficiency/the Pareto Front is a binary notion. A point either is or isn't Pareto Efficient (on the Pareto Front).
However, when the authors mention "sorting by Pareto Efficiency" and use it in the pseudocode, they are not referring to a binary notion.
In the response here, the authors mention a specific method of sorting the points by iteratively calculating and removing the Pareto front. My understanding is that this is a form of Non-Dominated Sorting. Neither the term "nondominated sorting", nor even a description of this sorting method, are mentioned anywhere in the paper. This seems to be a pretty important implementation detail for describing and re-implementing the algorithm. There are many entire papers dedicated to different ways of doing this sorting, including these fairly recent ones [1, 2].
It seems to me that the author's usage of the term "Pareto Efficiency" is incorrect. If someone were to read the paper and pseudocode and implement this, they would likely take the term "ParetoEfficiency" in the pseudocode and just apply a binary fitness of whether the point is on the Pareto Front (since that is what Pareto Efficiency is) rather than the sorting method the authors mentioned in this response.
Furthermore, because the authors do not mention any details of this sorting in the paper or supplementary material (or even use the term "nondominated sorting"), it would not be possible to properly reproduce the algorithm and results with confidence.
[1] Long, Qiang, Xue Wu, and Changzhi Wu. "Non-dominated sorting methods for multi-objective optimization: review and numerical comparison." Journal of Industrial & Management Optimization 17.2 (2021): 1001.
[2] Tian, Ye, et al. "Effectiveness and efficiency of non-dominated sorting for evolutionary multi-and many-objective optimization." Complex & Intelligent Systems 3.4 (2017): 247-263.
Thank you for your comments
Your comment is that we should have been more detailed about the Pareto efficiency sorting method, for a better understanding of readers.
Of course our intention was the non-dominated sorting.
Since the concept of Pareto-efficiency is widely used even in autoML, we thought that it would give a simple and clear explanation of our intention to readers.
In general, (rank n of) Pareto-front, Pareto-efficiency, Pareto-frontier … are used as the same term of (rank n of) non-dominated sorting.
A more precise explanation is that "Pareto-front" means the result, and "non-dominated sorting" means the method to get Pareto-fronts. The difference between non-dominated sorting methods is the computational complexity.
However, the result of methods should be exactly same : Pareto-fronts.
In short, whichever method is used, the Pareto-front result must be the same, so we skipped the description of the method.
LEMONADE [1] is an early study of autoML.
In the pseudo code, they simply describe calculation of Pareto-front
P <- ParetoFront(P U N)
with a simple definition of Pareto-front.
(And they did not describe what method they used.)
“Non-dominated sorting methods for multi-objective optimization” [2] is a method for a fast calculation of non-dominated sorting. In this paper, they also simply use the term Pareto front as the result of non-dominated sorting.
However, in the “Effectiveness and efficiency of non-dominated sorting for evolutionary multi-and many-objective optimization” [3], they often use the term Pareto-front as a ground state of the target function. It is because the goal of this study is the performance comparison of W/ non-dominated soring and W/O non-dominated sorting. (The conclusion is that non-dominated sorting is very important.)
We agree that it would be much better to explain details of method for a better understanding.
We will add more explanations about pareto efficiency and sorting techniques in the revision (and supplement) as soon as we can.
Thank you again for comments.
[1] Elsken, T., Metzen, J. H., & Hutter, F. (2018). Efficient multi-objective neural architecture search via lamarckian evolution. arXiv preprint arXiv:1804.09081.
[2] Long, Qiang, Xue Wu, and Changzhi Wu. "Non-dominated sorting methods for multi-objective optimization: review and numerical comparison." Journal of Industrial & Management Optimization 17.2 (2021): 1001.
[3] Tian, Ye, et al. "Effectiveness and efficiency of non-dominated sorting for evolutionary multi-and many-objective optimization." Complex & Intelligent Systems 3.4 (2017): 247-263.
Based on my reading of the LEMONADE paper, they are using the term correctly.
When they say "Pareto Front" they are referring to the strict definition of "Pareto Front" and are not referring to any type of sorting.
They are not sorting/ranking, but directly filtering everything that's not on the pareto front.
Hence, they do not mention non-dominated sorting, nor do they use the terminology regarding "rank n" that the authors bring up. They precisely define exactly what a pareto front is, and do not describe sorting or ranking because they do not perform sorting or ranking.
>LEMONADE maintains a population P of parent networks, which we choose to comprise all non-dominated networks
In fact, they explicitly say that they don't consider any points that are dominated
>One could also include some dominated architectures in the population to increase diversity, but we do not consider this in this work
Indeed Long et al. mention the concept of ranks of Pareto fronts.
However, note that:
1. They define precisely what they mean when they write that.
2. In the original version of this paper, the authors did not mention "ranks" of pareto fronts
Furthermore, the fact that, as the authors mention, Tian et al. us the term differently, means that it is quite important to specify exactly what is meant by "sorted and selected by Pareto Efficiency".
We recognize that there may be confusion because we provided too little information in the first submission.
In the revised version (and supplement), we specified the definition of Pareto efficiency, the description of ranks and sorting.
Thank you again for your help.
Thanks for the responses and updates. Most of my specific concerns were addressed. I believe the method is actually quite interesting and achieves good results;
however, the writing is still extremely poor.
While the authors made some modifications during the rebuttal period, there are still a very large number of imprecise statements, non-academic language and grammatical errors in the manuscript. I would highly recommend giving the paper a *thorough proofreading*. As-is, the writing is still so difficult to parse that it significantly affects my understanding of the paper.
I believe many of the other reviewers's concerns can be addressed with better writing. This includes things like clarity on experimental details (this is particularly important), placing / framing / motivating the work more precisely with respect to related work, and improved grammar and general writing (which would make it *much* easier to read).
I will update my score to reflect the fact that the underlying method seems interesting enough.
A brief note on the RL papers:
Although the papers evaluate simple genetic/evolution algorithms in reinforcement learning, they are still treating the reinforcement learning as a blackbox problem, where they output policy parameters. Their methods should still be able to apply to these simpler settings just as well.
Thanks to your valuable review, we were able to improve the revision a lot.
In the first submission, their were a lot of insufficient explanation.
In particular, it seems that the purpose of each experiment was not sufficiently explained. (Catpole-V1, LeNet-5)
In the revision, we tried to write a detailed description, according to the opinions of the reviewers.
Additional experiments have also been added.
Regarding sentence structures and statements, we are still working on making the paper easier to read.
But, we agree that we need additional proofreading.
Thank you.
The strengths of the paper include:
1) It proposes an interesting method that utilizes generative neural networks and evolution strategy. Evolution helps to stabilize the training of critic networks while the critic networks provide efficient mutations.
2) Experiment results are better than baseline methods in many cases.
The weaknesses of the paper include:
1) For generative neural networks, stabilizing training often is very challenging, and training can be expensive.
2) Due to the complexity of generative neural networks, it will be difficult for the proposed method to scale to handle high dimension problems.
3) The organization of the paper can be better. The pictures and algorithm should be close to their description pages. The pictures' quality is not high. I failed to print the paper using two different printers. There are something wrong with the format to this paper.
I had a hard time to understand the Algorithm 1 on page 3. What is N and M? Should "while n = N " be "while n < N"?
In Figure 3, what is BO package used?
In Figure 5, what does The noise is restored for visualization mean?
Why does GEO fail to find the Pareto front for ZDT2?
Why the generated images can not be recognized but still have high scores?
I think the authors adequately addressed the limitations and potential negative societal impact of their work.
No
I really appreciate your valuable questions and advices
Q1. I had a hard time to understand the Algorithm 1 on page 3. What is N and M? Should "while n = N " be "while n < N"?
A1.1
Typo correction: I found [while n=N], but it is [while n<N] or [for n in range(N)].
I’m really sorry for confusing you
A.1.2
Meaning of N, M :
N: The number of critic networks == The number of target objects
M: The number of mutation per 1 target object (1 critic network)
M is a hyper-parameter of the evolution strategy, you can choose how many mutation will be.
Q2. In Figure 3, what is BO package used
A2.
It is an unpublished package of my company.
Basically, BO is a Gaussuian process based Bayesian Optimization method. The special trick of the package is using evolution algorithms for fast estimation of acquisition function. By tuning the acquisition function searching policy, we can control performance of BO.
Still, it is a Gaussian process based BO, so it has O(N^3) computational complexity as other GP-BO does.
Since the Guassian process is the core algorithm, the auxiliary evolutionary algorithm does not decisively affect the computational complexity and performance of GP-BO.
Q3. In Figure 5, what does The noise is restored for visualization mean?
A3.
In figure 5,
G : generator
F : Black-box (non-stochastic)
X = G(z), where z is a seed.
We defined non-stochastic experiment as
Score = F(X)
And we defined stochastic experiment as
Score = F(X) + N, where N is random normal distribution.
Figure 5-b) is plot of F(X), where X
5-a) is plot of F(X) + N
Meaning of restored :
Because the pool size is too small, it does not fully illustrates overall distribution of noise N. Because we wanted show the range of X that GEO see in the several iterations, we visualized it by adding more noises onto F(X).
Therefore, 5-b) has meaningful information, 5-a) is an auxiliary figure for understanding.
Q4. Why does GEO fail to find the Pareto front for ZDT2?
We guess the collapsing problem comes from the shape of Pareto front.
ZDT1 has a convex Pareto front shape. ZDT3 has a partially concave, partially convex Pareto front shape.
However, ZDT2 has only a concave Pareto front shape.
In the evolution stage, the sorting algorithm (sorting by pareto efficiency) tends to sample edge state first if the Pareto front has concave shape.
Therefore, only edge states survive after few iterations.
We tried to solve this problem, but we do not find any efficient method that do not harm optimization performance.
Q5. Why the generated images cannot be recognized but still have high scores?
We guess it is almost like the situation where we can see in “Adversarial attack”.
Adversarial attack : https://pytorch.org/tutorials/beginner/fgsm_tutorial.html
In the adversarial attack, we can change the prediction of neural network by adding noise-like image onto original image.
We think that the generated image of Figure-6 is almost like an adversarial attack on white back ground image.
C1. For generative neural networks, stabilizing training often is very challenging, and training can be expensive.
Yes, they can be.
However, black-box optimization studies have a different point of view.
We assume that making O(N) computational complexity is one of the most important goals.
Also, in the high-dimensional optimization problems, making O(N) complexity is much more important since they require a lot of black-box function calls.
The training cost is the 2nd point to be considered.
Also, what we have done is the development of a stabilizing method. Training is not that expensive because we found a practical method (evolution) to stabilize generative model in optimization.
C2. Due to the complexity of generative neural networks, it will be difficult for the proposed method to scale to handle high dimension problems.
Yes. GEO is strongly depends on performance of GPU. We mentioned that this problem mainly comes from memory consumption of the attention network. We are trying to find other efficient network structures.
Still, GEO has much stronger performance in searching extremely high-dimensional space than previous methods.
C3. Picture organization and printing problems.
Paper printing problem :
I wrote this paper in Overleaf, a verified paper writing site, so it is difficult to answer why you are having printing problems.
I have experienced inconsistency between PDF readers (Adobe, Chrome, …) and between OS (Mac, Windows, …) many times. I guess OS or APP inconsistency is the problem.
Picture positions :
In other science field, I have read so many papers that arrange 1-Figure / 1-page style.
So I thought it is okay to follow 1-Figure / 1-Page style.
I would be better to change writing style next time. Thank you for your advice.
There were many changes in the revision.
In particular, the algorithm description has been improved.
Additional explanations were added according to the comments of the reviewer, and unnecessary explanations were removed.
The background of the Cartpole-V1 experiment and the purpose of the LeNet-5 experiment were further explained.
Data has also been added.
thank you

Meta-review:
role: AC
data: metareview: While the topic of the paper and the reported experimental results appeared to be of interest of the reviewing team, a number of limitations were put to the fore by the reviewers, who graded the paper with scores between 2 and 5, and often emphasized various issues such as a lacunary literature review (See in particular comments of Reviewer wE4h), insufficient comparison with state-of-the-art approaches and in particular on realistic test-casses (bPMJ), as well as on the lack of clarity (See for instance in Reviever uzBL’s review: „The paper is poorly structured and written to the point where it is quite difficult to understand“ ). For all these reasons I recommend rejection.     
role: PC

Decision: Reject