Model-based Optimization with
Normalizing FlowsS a ra Ka rami

I F T S e mi nars
April 2024

● Introduction and Background● Proposed approach

Outline● Experiments and Results● Conclusion and Future work2

Optimization
General form:

Problem difficulty*

Selecting method according to 𝑓(𝑥)•Blackbox Optimization (BBO)•

Derivative Free Optimization (DFO)•

Gradient Descent•

Analytical Solvers*Problem difficulty: Less information / assumptions available

3Optimization
General form:

Problem difficulty*

Selecting method according to 𝑓(𝑥)•Blackbox Optimization (BBO)

Only gives output values when gets some input queries•

Derivative Free Optimization (DFO)

No direct access to derivatives but they exist and there is a level of smoothness•

Gradient Descent

No regular form but there is a formulation to calculate the derivatives•

Analytical Solvers

Formulating the problem and its constraints in specific forms*Problem difficulty: Less information / assumptions available

4

Optimization
General form:
Selecting method according to 𝑓(𝑥)Problem difficulty•Blackbox Optimization (BBO)
Only if derivatives do not exist or are too costly to compute•

Derivative Free Optimization (DFO)•

Gradient Descent•

Analytical Solvers

5Black Box Optimization
Function characteristics:●

Complexity: non-smooth, discontinuous, highly multimodal, noisy●

Dimensionality: large search space●

Separability: dependence between objective variables

Real-world examples:●

Computer simulation●

Laboratory experiment

Schematic of a Black Box [1]Goal: Find a good enough solution with minimum evaluations[1] C. Audet, W. Hare, Derivative-Free and Blackbox Optimization, 1st ed. 2017.

6

Black Box Optimization
Function characteristics:●

Complexity: non-smooth, discontinuous, highly multimodal, noisy●

Dimensionality: large search space●

Separability: dependence between objective variables

Real-world examples:●

Computer simulation●

Design of experiments

Schematic of a Black Box [1]Goal: Find a good enough solution with minimum evaluations[1] C. Audet, W. Hare, Derivative-Free and Blackbox Optimization, 1st ed. 2017.

7

Black Box Optimization
Function characteristics:●

Complexity: non-smooth, discontinuous, highly multimodal, noisy●

Dimensionality: large search space●

Separability: dependence between objective variables

Real-world examples:●

Computer simulation●

Design of experiments

Schematic of a Black Box [1]Goal: Find a good enough solution with minimum number of sample evaluations[1] C. Audet, W. Hare, Derivative-Free and Blackbox Optimization, 1st ed. 2017.

8

BBO vs DFO
Black-box optimizationoTypically, no assumptions of any form of continuity, differentiability or smoothness on the functiono More on the heuristic side without mathematical supporto Main approaches are Evolutionary Algorithms (EA) and Randomized based methods

Derivative free optimizationo More mathematically supported: prove of convergence and/or stopping criteriono More on the deterministic sideo Direct search and surrogate-based

9

BBO Methods
Deciding BBO method● Complexity and characteristics of 𝑓(𝑥)Model-based BBO● View optimization as a series of incremental updates of a modeloGaussian processoProbabilistic distribution

Problem difficulty● Cost of evaluation or evaluation budget•

Genetic Algorithms•

Estimation of Distribution
Algorithm (EDA)•

Density model-based
Evolutionary Strategy (ES)•Bayesian Optimization● Sample candidate solutions•

Simulated annealing● Evaluate the samples and update the model•

Hill climbing● Initialize the model

10

BBO Methods
Deciding BBO method● Complexity and characteristics of 𝑓(𝑥)Model-based BBO● View optimization as a series of incremental updates of a modeloGaussian processoProbabilistic distribution

Problem difficulty● Cost of evaluation or evaluation budget•

Genetic Algorithms•

Estimation of Distribution
Algorithm (EDA)•

Density model-based
Evolutionary Strategy (ES)•Bayesian Optimization● Sample candidate solutions•

Simulated annealing● Evaluate the samples and update the model•

Hill climbing● Initialize the model

11

BBO Methods
Deciding BBO method● Complexity and characteristics of 𝑓(𝑥)Model-based BBO● View optimization as a series of incremental updates of a modeloGaussian processoProbabilistic distribution

Problem difficulty● Cost of evaluation or evaluation budget•

Genetic Algorithms•

Estimation of Distribution
Algorithm (EDA)•

Density model-based
Evolutionary Strategy (ES)•Bayesian Optimization● Sample candidate solutions•

Simulated annealing● Evaluate the samples and update the model•

Hill climbing● Initialize the model

12

Hill Climbing
Deciding BBO method● Complexity and characteristics of 𝑓(𝑥)Model-based BBO● View optimization as a series of incremental updates of a model••o Gaussian process
Starts from a random point
Randomlyoselects
a neighbor
Probabilistic
distribution
with better value● Initialize the model● Sample candidate solutions

Problem difficulty● Cost of evaluation or evaluation budget•

Genetic Algorithms•

Estimation of Distribution
Algorithm (EDA)•

Density model-based
Evolutionary Strategy (ES)•Simulated annealing•

Hill climbing● Evaluate the samples and update the model

https://umu.to/blog/2018/06/29/hill-climbing-irl
https://www.slideshare.net/JismyKJose/heuristc-search-70129411

13

Hill Climbing
Deciding BBO method● Complexity and characteristics of 𝑓(𝑥)Model-based BBO● View optimization as a series of incremental updates of a model••o Gaussian process
Starts from a random point
Randomlyoselects
a neighbor
Probabilistic
distribution
with better value● Initialize the model•

Highlyto get candidate
stuck in● proneSamplesolutions
local optima or plateaus!

Problem difficulty● Cost of evaluation or evaluation budget•

Genetic Algorithms•

Estimation of Distribution
Algorithm (EDA)•

Density model-based
Evolutionary Strategy (ES)•Simulated annealing•

Hill climbing● Evaluate the samples and update the model

https://umu.to/blog/2018/06/29/hill-climbing-irl
https://www.slideshare.net/JismyKJose/heuristc-search-70129411

14

Hill Climbing
Deciding BBO method● Complexity and characteristics of 𝑓(𝑥)Model-based BBO● View optimization as a series of incremental updates of a model••o Gaussian process
Starts from a random point
Randomlyoselects
a neighbor
Probabilistic
distribution
with better value● Initialize the model••

Highlyto get candidate
stuck in● proneSamplesolutions
local optima or plateaus!

Problem difficulty● Cost of evaluation or evaluation budget•

Genetic Algorithms•

Estimation of Distribution
Algorithm (EDA)•

Density model-based
Evolutionary Strategy (ES)•Simulated annealing•

Hill climbing● Evaluate the samples and update the model
Hill climbing vs Gradient descent?

https://umu.to/blog/2018/06/29/hill-climbing-irl
https://www.slideshare.net/JismyKJose/heuristc-search-70129411

15

Hill Climbing
Deciding BBO method● Complexity and characteristics of 𝑓(𝑥)Model-based BBO● View optimization as a series of incremental updates of a model••o Gaussian process
Starts from a random point
Randomlyoselects
a neighbor
Probabilistic
distribution
with better value● Initialize the model••

Highlyto get candidate
stuck in● proneSamplesolutions
local optima or plateaus!

Problem difficulty● Cost of evaluation or evaluation budget•

Genetic Algorithms•

Estimation of Distribution
Algorithm (EDA)•

Density model-based
Evolutionary Strategy (ES)•Simulated annealing•

Hill climbing● Evaluate the samples and update the model
Hill climbing vs Gradient descent? Greedy choice the next point based on immediate local
information regardless of gradient amount or direction

https://umu.to/blog/2018/06/29/hill-climbing-irl
https://www.slideshare.net/JismyKJose/heuristc-search-70129411

16Simulated Annealing
Deciding BBO method● Complexity and characteristics of 𝑓(𝑥)Model-based BBO● View optimization as a series of incremental updates of a model••oGaussian processoProbabilistic distribution● suboptimal
Initializesolutions
the model
Accepts
with a probability relative toa temperature parameter and has an adaptive step size
● Sample candidate solutions

Problem difficulty● Cost of evaluation or evaluation budget•

Genetic Algorithms•

Estimation of Distribution
Algorithm (EDA)•

Density model-based
Evolutionary Strategy (ES)•Simulated annealing•

Hill climbing

Provides better exploration and possibility of jumping outof local●optima
Evaluate the samples and update the model

https://www.researchgate.net/figure/Principle-of-the-simulated-annealing-algorithm_fig5_360434054

17

Simulated Annealing
Deciding BBO method● Complexity and characteristics of 𝑓(𝑥)Model-based BBO● View optimization as a series of incremental updates of a model•oGaussian processoProbabilistic distribution● suboptimal
Initializesolutions
the model
Accepts
with a probability relative toa temperature parameter and has an adaptive step size
● Sample candidate solutions

Problem difficulty● Cost of evaluation or evaluation budget•

Genetic Algorithms•

Estimation of Distribution
Algorithm (EDA)•

Density model-based
Evolutionary Strategy (ES)•Simulated annealing•

Hill climbing•

Provides better exploration and possibility of jumping outof local●optima
Evaluate the samples and update the model•

One sample at a time gives a very limited view of the optimization landscape and scheduled temperature leads to premature convergence

https://www.researchgate.net/figure/Principle-of-the-simulated-annealing-algorithm_fig5_360434054
https://www.researchgate.net/figure/Simulated-Annealing-if-the-temperature-is-very-low-wrt-the-jump-size-SA-risks-a_fig4_30531067

18

Density Model-Based ES
Deciding BBO method● Complexity and characteristics of 𝑓(𝑥)Model-based BBO•● View optimization as a series of incremental updates of a model

Randomly Sampling a number of points fromo Gaussian process
a distributionoProbabilistic distribution

Problem difficulty● Cost of evaluation or evaluation budget•

Genetic Algorithms•

Estimation of Distribution
Algorithm (EDA)•

Density model-based
Evolutionary Strategy(ES)● Initialize the model•

Simulated annealing● Sample candidate solutions•

Hill climbing● Evaluate the samples and update the model"CMA-ES." wikipedia.org. https ://en.wikipedia.org/wiki/CMAES (accessed Sep. 25,2022)

19

Density Model-Based ES
Deciding BBO method● Complexity and characteristics of 𝑓(𝑥)Model-based BBO•● View optimization as a series of incremental updates of a model

Randomly Sampling a number of points fromo Generation
Gaussian process
a distribution:and IndividualsoProbabilistic distribution

Problem difficulty● Cost of evaluation or evaluation budget•

Genetic Algorithms•

Estimation of Distribution
Algorithm (EDA)•

Density model-based
Evolutionary Strategy(ES)● Initialize the model•

Simulated annealing● Sample candidate solutions•

Hill climbing● Evaluate the samples and update the model"CMA-ES." wikipedia.org. https ://en.wikipedia.org/wiki/CMAES (accessed Sep. 25,2022)

20

Density Model-Based ES
Deciding BBO method● Complexity and characteristics of 𝑓(𝑥)Model-based BBO••● View optimization as a series of incremental updates of a model

Randomly Sampling a number of points fromo Generation
Gaussian process
a distribution:and Individualso Probabilistic distribution
Evaluation: Calculating the function value(Fitness)
each individual● forInitialize
the model● Sample candidate solutions

Problem difficulty● Cost of evaluation or evaluation budget•

Genetic Algorithms•

Estimation of Distribution
Algorithm (EDA)•

Density model-based
Evolutionary Strategy(ES)•

Simulated annealing•

Hill climbing● Evaluate the samples and update the model"CMA-ES." wikipedia.org. https ://en.wikipedia.org/wiki/CMAES (accessed Sep. 25,2022)

21

Density Model-Based ES
Deciding BBO method● Complexity and characteristics of 𝑓(𝑥)Model-based BBO•••● View optimization as a series of incremental updates of a model

Randomly Sampling a number of points fromo Generation
Gaussian process
a distribution:and Individualso Probabilistic distribution
Evaluation: Calculating the function value(Fitness)
each individual● forInitialize
the model

Selection:
fitter individuals● selecting
Samplethecandidatesolutions

Problem difficulty● Cost of evaluation or evaluation budget•

Genetic Algorithms•

Estimation of Distribution
Algorithm (EDA)•

Density model-based
Evolutionary Strategy(ES)•

Simulated annealing•

Hill climbing● Evaluate the samples and update the model"CMA-ES." wikipedia.org. https ://en.wikipedia.org/wiki/CMAES (accessed Sep. 25,2022)

22

Density Model-Based ES
Deciding BBO method● Complexity and characteristics of 𝑓(𝑥)Model-based BBO••● View optimization as a series of incremental updates of a model

Randomly Sampling a number of points fromo Generation
Gaussian process
a distribution:and Individualso Probabilistic distribution
Evaluation: Calculating the function value(Fitness)
each individual● forInitialize
the model•

Selection:
fitter individuals● selecting
Samplethecandidatesolutions•

Updatingmodel● density
Evaluate
the samples and update the model"CMA-ES." wikipedia.org. https ://en.wikipedia.org/wiki/CMAES (accessed Sep. 25,2022)

Problem difficulty● Cost of evaluation or evaluation budget•

Genetic Algorithms•

Estimation of Distribution
Algorithm (EDA)•

Density model-based
Evolutionary Strategy(ES)•

Simulated annealing•

Hill climbing

23

Density Model-Based ES
Deciding BBO method● Complexity and characteristics of 𝑓(𝑥)Model-based BBO••● View optimization as a series of incremental updates of a model

Randomly Sampling a number of points fromo Generation
Gaussian process
a distribution:and Individualso Probabilistic distribution
Evaluation: Calculating the function value(Fitness)
each individual● forInitialize
the model•

Selection:
fitter individuals● selecting
Samplethecandidatesolutions•

Updatingmodel● density
Evaluate
the samples and update the model•

Creating next generation"CMA-ES." wikipedia.org. https ://en.wikipedia.org/wiki/CMAES (accessed Sep. 25,2022)

Problem difficulty● Cost of evaluation or evaluation budget•

Genetic Algorithms•

Estimation of Distribution
Algorithm (EDA)•

Density model-based
Evolutionary Strategy(ES)•

Simulated annealing•

Hill climbing

24

Density Model-Based ES
Deciding BBO method● Complexity and characteristics of 𝑓(𝑥)Model-based BBO••● View optimization as a series of incremental updates of a model

Randomly Sampling a number of points fromo Generation
Gaussian process
a distribution:and Individualso Probabilistic distribution
Evaluation: Calculating the function value(Fitness)
each individual● forInitialize
the model•

Selection:
fitter individuals● selecting
Samplethecandidatesolutions•

Updatingmodel● density
Evaluate
the samples and update the model•

Creating next generation"CMA-ES." wikipedia.org. https ://en.wikipedia.org/wiki/CMAES (accessed Sep. 25,2022)

Problem difficulty● Cost of evaluation or evaluation budget•

Genetic Algorithms•

Estimation of Distribution
Algorithm (EDA)•

Density model-based
Evolutionary Strategy(ES)•

Simulated annealing•

Hill climbing25

● Introduction and Background● Proposed approach

Outline● Experiments and Results● Conclusion and Future steps

26

CMA-ES
Covariance Matrix Adaptation Evolution Strategy (CMA-ES), a popular Density Model-Based ES

Our focus:● Flexible density model: more efficiently lead the search to the optima● Meta-optimization: reuse the costly information gained in previous runs
"CMA-ES." wikipedia.org. https ://en.wikipedia.org/wiki/CMAES (accessed Sep. 25,2022)

27

CMA-ES
Covariance Matrix Adaptation Evolution Strategy (CMA-ES), a popular Density Model-Based ES•

Randomly sampling the 1st generation from a standard
Gaussian distribution•

Evaluation: calculating fitness for each individual•

Our focus:
Selection:
selecting the fitter individuals•

Updating covariance matrix and mean of the Gaussian•

Creating next generation from the updated Gaussian● Flexible density model: more efficiently lead the search to the optima● Meta-optimization: reuse the costly information gained in previous runs

"CMA-ES." wikipedia.org. https ://en.wikipedia.org/wiki/CMAES (accessed Sep. 25,2022)

28

CMA-ES
Covariance Matrix Adaptation Evolution Strategy (CMA-ES), a popular Density Model-Based ES•

Randomly sampling the 1st generation from a standard
Gaussian distribution•

Evaluation: calculating fitness for each individual•

Our focus:
Selection:
selecting the fitter individuals•

Updating covariance matrix and mean of the Gaussian•

Creating next generation from the updated Gaussian● Flexible density model: more efficiently lead the search to the optima● Meta-optimization: reuse the costly information gained in previous runs

"CMA-ES." wikipedia.org. https ://en.wikipedia.org/wiki/CMAES (accessed Sep. 25,2022)

29

CMA-ES
Covariance Matrix Adaptation Evolution Strategy (CMA-ES), a popular Density Model-Based ES

Drawbacks of CMA-ES:●

Arbitrary choice of distribution limits performance

Our focus:● Flexible density model: more efficiently lead the search to the optima● Meta-optimization: reuse the costly information gained in previous runs
"CMA-ES." wikipedia.org. https ://en.wikipedia.org/wiki/CMAES (accessed Sep. 25,2022)

30

CMA-ES
Covariance Matrix Adaptation Evolution Strategy (CMA-ES), a popular Density Model-Based ES

Drawbacks of CMA-ES:●

Arbitrary choice of distribution limits performance"CMA-ES." wikipedia.org. https ://en.wikipedia.org/wiki/CMAES (accessed Sep. 25,2022).,"cmaes 0.3.1." https://pypi.org/project/cmaes/0.3.1/ (accessed Apr. 10, 2024)

31

CMA-ES
Covariance Matrix Adaptation Evolution Strategy (CMA-ES), a popular Density Model-Based ES

Drawbacks of CMA-ES:●

Arbitrary choice of distribution limits performance

Flexible Distribution

Ideal samples"CMA-ES." wikipedia.org. https ://en.wikipedia.org/wiki/CMAES (accessed Sep. 25,2022).,"cmaes 0.3.1." https://pypi.org/project/cmaes/0.3.1/ (accessed Apr. 10, 2024)
L. Faury, C. Calauzenes, O. Fercoq, and S. Krichen, "Improving Evolutionary Strategies with Generative Neural Networks," arXiv preprint arXiv :1901.11271v1, 2019.

32

CMA-ES
Covariance Matrix Adaptation Evolution Strategy (CMA-ES), a popular Density Model-Based ES

Drawbacks of CMA-ES:●

Arbitrary choice of distribution limits performance●

Starting from scratch and withdrawing gained knowledge"CMA-ES." wikipedia.org. https ://en.wikipedia.org/wiki/CMAES (accessed Sep. 25,2022)

33

CMA-ES
Covariance Matrix Adaptation Evolution Strategy (CMA-ES), a popular Density Model-Based ES

Drawbacks of CMA-ES:●

Arbitrary choice of distribution limits performance●

Starting from scratch and withdrawing gained knowledge"CMA-ES." wikipedia.org. https ://en.wikipedia.org/wiki/CMAES (accessed Sep. 25,2022).,"cmaes 0.3.1." https://pypi.org/project/cmaes/0.3.1/ (accessed Apr. 10, 2024)

34

Proposed Approach
Covariance Matrix Adaptation Evolution Strategy (CMA-ES), a popular Density Model-Based ES

Drawbacks of CMA-ES:●

Arbitrary choice of distribution limits performance●

Starting from scratch and withdrawing gained knowledge

We propose using Normalizing Flows (NF) instead of Gaussian to model the density

Our focus:● Flexible density model: more efficiently lead the search to the optima● Meta-optimization: reuse the costly information gained in previous runs
"CMA-ES." wikipedia.org. https ://en.wikipedia.org/wiki/CMAES (accessed Sep. 25,2022)
L. Faury, C. Calauzenes, O. Fercoq, and S. Krichen, "Improving Evolutionary Strategies with Generative Neural Networks," arXiv preprint arXiv :1901.11271v1, 2019.

35

Proposed Approach
Covariance Matrix Adaptation Evolution Strategy (CMA-ES), a popular Density Model-Based ES

Drawbacks of CMA-ES:●

Arbitrary choice of distribution limits performance●

Starting from scratch and withdrawing gained knowledge

We propose using Normalizing Flows (NF) instead of Gaussian to model the density

Our● focus:
Flexible enough to adapt to the shape of the landscape●● Flexible
densitymoreefficiently lead the search to the optima
Simultaneous
searchmodel:
of multiple
modes●● Meta-optimization:
reuse the costly information gained in previous runs
Fine tuning instead of starting from scratch"CMA-ES." wikipedia.org. https ://en.wikipedia.org/wiki/CMAES (accessed Sep. 25,2022)
L. Faury, C. Calauzenes, O. Fercoq, and S. Krichen, "Improving Evolutionary Strategies with Generative Neural Networks," arXiv preprint arXiv :1901.11271v1, 2019.

36

Normalizing Flows
Structurally restricted Generative Neural Networks (bijective GNNs):
Learn probability distribution 𝑝𝑥 (𝑥) over a random variable 𝑋 from observations

A family of Probabilistic Generative Models: learn 𝑝𝑥 (𝑥) over 𝑋 from observations● Structurally restricted Generative Neural Network: bijective Generative NN● Highly expressive● Efficient sampling

Volume correction● Exact and efficient density evaluation● Based on change of variable formula

Flow / Bijection
Invertible
Differentiable
Efficiently computable 𝑑𝑒𝑡𝐷ℎ(𝑥)37Normalizing Flows
Structurally restricted Generative Neural Networks (bijective GNNs):
Learn probability distribution 𝑝𝑥 (𝑥) over a random variable 𝑋 from observations

A family of Probabilistic Generative Models: learn 𝑝𝑥 (𝑥) over 𝑋 from observations● Structurally restricted Generative Neural Network: bijective Generative NN● Highly expressive
Flow / Bijection●

Invertible
Efficient sampling Differentiable

Fixed latent distribution

Volume correction● Exact and efficient density evaluation● Based on change of variable formula"Tutorial on normalizing flows, part 2." https://engineering.papercup.com/posts/normalizing-flows-part-2/ (accessed Apr. 10, 2024)

Flow / Bijection
Invertible
Differentiable
Efficiently computable 𝑑𝑒𝑡𝐷ℎ(𝑥)38Normalizing Flows
Structurally restricted Generative Neural Networks (bijective GNNs):
Learn probability distribution 𝑝𝑥 (𝑥) over a random variable 𝑋 from observations

A family of Probabilistic Generative Models: learn 𝑝𝑥 (𝑥) over 𝑋 from observations● Structurally restricted Generative Neural Network: bijective Generative NN● Highly expressive
Flow / Bijection●

Invertible
Efficient sampling Differentiable

Fixed latent distribution

Volume correction● Exactflows
and efficientdensityevaluation
Designingis the coreresearch
problem in NFs●● Based
onlayerchangeformulaEach flow
in NFofis avariable
simple invertible,
differentiable function●

The overall flow network is a composition of simple flow layers"Tutorial on normalizing flows, part 2." https://engineering.papercup.com/posts/normalizing-flows-part-2/ (accessed Apr. 10, 2024)

Flow / Bijection
Invertible
Differentiable
Efficiently computable 𝑑𝑒𝑡𝐷ℎ(𝑥)39Normalizing Flows
Structurally restricted Generative Neural Networks (bijective GNNs):
Learn probability distribution 𝑝𝑥 (𝑥) over a random variable 𝑋 from observations

A family of Probabilistic Generative Models: learn 𝑝𝑥 (𝑥) over 𝑋 from observations● Structurally restricted Generative Neural Network: bijective Generative NN● Highly expressive
Fixed latent distribution● Efficient sampling

Volume correction● Exactflows
and efficientdensityevaluation
Designingis the coreresearch
problem in NFs●● Based
onlayerchangeformulaEach flow
in NFofis avariable
simple invertible,
differentiable function●

The overall flow network is a composition of simple flow layers"Tutorial on normalizing flows, part 2." https://engineering.papercup.com/posts/normalizing-flows-part-2/ (accessed Apr. 10, 2024)

Flow / Bijection
Invertible
Differentiable
Efficiently computable 𝑑𝑒𝑡𝐷ℎ(𝑥)40Normalizing Flows
Structurally restricted Generative Neural Networks (bijective GNNs):
Learn probability distribution 𝑝𝑥 (𝑥) over a random variable 𝑋 from observations

A family of Probabilistic Generative Models: learn 𝑝𝑥 (𝑥) over 𝑋 from observations● Structurally restricted Generative Neural Network: bijective Generative NN● Highly expressive
Fixed latent distribution● Efficient sampling

Volume correction● Exact and efficient density evaluation● Based on change of variable formula"Tutorial on normalizing flows, part 2." https://engineering.papercup.com/posts/normalizing-flows-part-2/ (accessed Apr. 10, 2024)

Flow / Bijection
Invertible
Differentiable
Efficiently computable 𝑑𝑒𝑡𝐷ℎ(𝑥)41Normalizing Flows
Structurally restricted Generative Neural Networks (bijective GNNs):
Learn probability distribution 𝑝𝑥 (𝑥) over a random variable 𝑋 from observations

A family of Probabilistic Generative Models: learn 𝑝𝑥 (𝑥) over 𝑋 from observations● Structurally restricted Generative Neural Network: bijective Generative NN● Highly expressive
Fixed latent distribution● Efficient sampling

Volume correction● Exact and efficient density evaluation● Based on change of variable formula"Tutorial on normalizing flows, part 2." https://engineering.papercup.com/posts/normalizing-flows-part-2/ (accessed Apr. 10, 2024)

Flow / Bijection
Invertible
Differentiable
Efficiently computable 𝑑𝑒𝑡𝐷ℎ(𝑥)42Normalizing Flows
Structurally restricted Generative Neural Networks (bijective GNNs):
Learn probability distribution 𝑝𝑥 (𝑥) over a random variable 𝑋 from observations

Designing flows is the core research problem in NFs but NOT our research goal

"Tutorial on normalizing flows, part 2." https://engineering.papercup.com/posts/normalizing-flows-part-2/ (accessed Apr. 10, 2024)

43

Normalizing Flows
Structurally restricted Generative Neural Networks (bijective GNNs):
Learn probability distribution 𝑝𝑥 (𝑥) over a random variable 𝑋 from observations

Designing flows is the core research problem in NFs but NOT our research goal●

NF being highly expressive●

Sampling and density evaluation is efficient●

Exact forward and backward mapping"Tutorial on normalizing flows, part 2." https://engineering.papercup.com/posts/normalizing-flows-part-2/ (accessed Apr. 10, 2024)

44

Normalizing Flows
Structurally restricted Generative Neural Networks (bijective GNNs):
Learn probability distribution 𝑝𝑥 (𝑥) over a random variable 𝑋 from observations

Designing flows is the core research problem in NFs but NOT our research goal●

NF being highly expressive●

Sampling and density evaluation is efficient●

Exact forward and backward mapping"Tutorial on normalizing flows, part 2." https://engineering.papercup.com/posts/normalizing-flows-part-2/ (accessed Apr. 10, 2024)

45

Normalizing Flows
Structurally restricted Generative Neural Networks (bijective GNNs):
Learn probability distribution 𝑝𝑥 (𝑥) over a random variable 𝑋 from observations

Designing flows is the core research problem in NFs but NOT our research goal●

NF being highly expressive●

Sampling and density evaluation is efficient●

Exact forward and backward mapping

We adopted RealNVP Normalizing Flows for our work
"Tutorial on normalizing flows, part 2." https://engineering.papercup.com/posts/normalizing-flows-part-2/ (accessed Apr. 10, 2024)

46

● Introduction and Background● Proposed approach

Outline● Experiments and Results● Conclusion and Future steps● Conclusion / PhD Plan

47

Recap
CMA-ES, a popular Density Model-Based ES•

Randomly sampling the 1st generation from a standard
Gaussian distribution•

Evaluation: calculating fitness for each individual•

Our focus:
Selection:
selecting the fitter individuals•

Updating covariance matrix and mean of the Gaussian•

Creating next generation from the updated Gaussian● Flexible density model: more efficiently lead the search to the optima● Meta-optimization: reuse the costly information gained in previous runs

"CMA-ES." wikipedia.org. https ://en.wikipedia.org/wiki/CMAES (accessed Sep. 25,2022)

48

Recap
CMA-ES, a popular Density Model-Based ES•

Randomly sampling the 1st generation from a standard
Gaussian distribution•

Evaluation: calculating fitness for each individual•

Our focus:
Selection:
selecting the fitter individuals•

Updating covariance matrix and mean of the Gaussian•

Creating next generation from the updated Gaussian● Flexible density model: more efficiently lead the search to the optima● Meta-optimization: reuse the costly information gained in previous runs

"CMA-ES." wikipedia.org. https ://en.wikipedia.org/wiki/CMAES (accessed Sep. 25,2022)
L. Faury, C. Calauzenes, O. Fercoq, and S. Krichen, "Improving Evolutionary Strategies with Generative Neural Networks," arXiv preprint arXiv :1901.11271v1, 2019.

49

Proposed Approach
We proposed using Normalizing Flows (NF) instead of Gaussian to model the density●

Flexible enough to adapt to the shape of the landscape●

Simultaneous search of multiple modes●

Fine tuning instead of starting from scratch

L. Faury, C. Calauzenes, O. Fercoq, and S. Krichen, "Improving Evolutionary Strategies with Generative Neural Networks," arXiv preprint arXiv :1901.11271v1, 2019.

50

Proposed Approach
We proposed using Normalizing Flows (NF) instead of Gaussian to model the density●

Flexible enough to adapt to the shape of the landscape●

Simultaneous search of multiple modes●

Fine tuning instead of starting from scratch

Integrating RealNVP in Evolution Sterategy

L. Faury, C. Calauzenes, O. Fercoq, and S. Krichen, "Improving Evolutionary Strategies with Generative Neural Networks," arXiv preprint arXiv :1901.11271v1, 2019.

51

Proposed Approach
We proposed using Normalizing Flows (NF) instead of Gaussian to model the density●

Flexible enough to adapt to the shape of the landscape●

Simultaneous search of multiple modes●

Fine tuning instead of starting from scratch

Integrating RealNVP in EDA (Estimation of Distribution Alg.)●

Separate the impact of ES●

Investigate the full potential of NF

L. Faury, C. Calauzenes, O. Fercoq, and S. Krichen, "Improving Evolutionary Strategies with Generative Neural Networks," arXiv preprint arXiv :1901.11271v1, 2019.

52

NF + EDA
We proposed using Normalizing Flows (NF) instead of Gaussian to model the density

Integrating RealNVP in EDA (Estimation of Distribution Alg.)●

Separate the impact of ES●

Investigate the full potential of NF•

Sampling the 1st generation from a randomly initialized
RealNVP with a GMM base•Evaluation: calculating fitness for each individual•

Selection: selecting the fitter individuals•

Fine-tuning RealNVP to focus the distribution on the fitter
samples•

Sampling the next generation from the fine-tuned RealNVP

L. Faury, C. Calauzenes, O. Fercoq, and S. Krichen, "Improving Evolutionary Strategies with Generative Neural Networks," arXiv preprint arXiv :1901.11271v1, 2019.

53

NF + EDA
We proposed using Normalizing Flows (NF) instead of Gaussian to model the density

Integrating RealNVP in EDA (Estimation of Distribution Alg.)●

Separate the impact of ES●

Investigate the full potential of NF•

Sampling the 1st generation from a randomly initialized
RealNVP with a GMM base•Evaluation: calculating fitness for each individual•

Selection: selecting the fitter individuals•

Fine-tuning RealNVP to focus the distribution on the fitter
samples•

Sampling the next generation from the fine-tuned RealNVP

L. Faury, C. Calauzenes, O. Fercoq, and S. Krichen, "Improving Evolutionary Strategies with Generative Neural Networks," arXiv preprint arXiv :1901.11271v1, 2019.

54

NF + ES
Integrating RealNVP in Evolution Sterategy

Integrating RealNVP in EDA•

Sampling the 1st generation from a randomly initialized
RealNVP with a GMM base•

Sampling the 1st generation from a randomly initialized
RealNVP with a GMM base•Evaluation: calculating fitness for each individual in theoriginal space•

Evaluation: calculating fitness for each individual•

Selection: selecting the fitter individuals in the original space•

Selection: selecting the fitter individuals•

Fine-tuning RealNVP to focus the distribution on the fitter
samples•Fine-tuning RealNVP to focus the distribution on the fitter
samples•Remapping the selected samples to the latent space•

Sampling the next generation from the fine-tuned RealNVP•

Taking ES step in the latent space to generate the next
generation•

Mapping the new generation to the original space with thefine-tuned RealNVP
55

NF + ES
Integrating RealNVP in Evolution Sterategy

Integrating RealNVP in EDA•

Sampling the 1st generation from a randomly initialized
RealNVP with a GMM base•

Sampling the 1st generation from a randomly initialized
RealNVP with a GMM base•Evaluation: calculating fitness for each individual in theoriginal space•

Evaluation: calculating fitness for each individual•

Selection: selecting the fitter individuals in the original space•

Selection: selecting the fitter individuals•

Fine-tuning RealNVP to focus the distribution on the fitter
samples•Fine-tuning RealNVP to focus the distribution on the fitter
samples•Remapping the selected samples to the latent space•

Sampling the next generation from the fine-tuned RealNVP•

Taking ES step in the latent space to generate the next
generation•

Mapping the new generation to the original space with thefine-tuned RealNVP

NF+ES is expected to outperform NF+EDA

56

NF + EDA (Estimation of Distribution Alg.)
best score observed

Best Score on
Rastrigin
Model the density in EDA using Normalizing
Flows

●

Separates the impact of ES●

Investigate the full potential of NF# object evaluations

57

NF + EDA (Estimation of Distribution Alg.)
best score observed

Best
BestScore
ScoreononRastrigin
Rastrigin
Model the density in EDA using Normalizing
Flows

●

Separates the impact of ES●

Investigate the full potential of NF# object evaluations

58

NF + ES (Evolutionary Strategy)
NF + ES model-based BBO vs NF + EDA Model-based BBO
Best Score on Rastrigin
Best Score on Rastrigin
Best Score on Rastrigin

59

● Introduction and Background● Proposed approach

Outline● Experiments and Results● Conclusion and Future steps

60

Conclusion and Future Steps● Preliminary Results:
o NF has the required capability to enhance model-based BBO
o NF+EDA results are promising despite the simplicity of the method and with minimum hyperparametertuningo NF+ES results outperform NF+EDA and are competent with CMA-ES even in the preliminary experiments
without hyperparameter tuning● Challenges ahead:o Model convergence and Exploration/Exploitation balanceo Hyperparameter tuningo Conducting experiments on real-world data

● Next phase:o Applying Meta-optimization to further improve performance and sample efficiency

61