## 3 Previous Work  
Simulation-based dynamic optimization consists of finding the configuration of the simulation model that is most suitable for a given situation--e.g., the configuration that reduces latency, increases throughput, etc.--as time advances, represented through various states of the simulated model. In [10], the authors show that an adequate framework to deal with this kind of problem is to model them as Markov decision processes.  
Dynamic programming and reinforcement learning techniques are proposed as the main methods for solving this type of problem, to which deep reinforcement learning is later added as the means to efficiently control large and complex models.  
In [32], the use of DRL is proposed for the optimization of the quality of experience (QoE) for applications of the DASH video streaming standard. In [11], Q-learning is used to optimize the allocation of network resources and communication resources in response to requests for access division to 5G networks. In [33], the authors use Q-learning to prioritize and manage rewards to minimize task response times and optimize the use of cloud computing resources. DRL applications in economics can be found in [12]. Multiple DRL applications in communication networks are reviewed in [34].  
Reinforcement learning has also been applied in several works on production scheduling for manufacturing systems, as shown in [35; 36]; in fluid mechanics [37]; and in health applications [38; 39]. In [4; 5] the application of evolutionary algorithms, originally developed for static optimization, for solving dynamic optimization problems is reviewed. The methods are based on a metaheuristic to which components are added to deal with the dynamic characteristics of the problem and adapt to various changes in the environment, making the pertinent modifications and updates in the populations. The paper proposes a taxonomy of dynamic optimization problems and then lists the benchmarks present in the literature. The authors claim that most popular components of evolutionary algorithms for dynamic optimization are particle swarm optimizers (PSOs) and differential evolution (DE) algorithms.  
Some applications of evolutionary algorithms for dynamic optimization include training neural networks for classification problems with drift of concepts, hyperparameter optimization of Support Vector Machines (SVMs), training time series predictors with neural networks, adaptive agriculture strategies, the locations of chemical odor sources through robots, cost reduction in electrical energy systems, and the identification of polluting sources in water distribution networks.  
In [40], the authors compare evolutionary algorithms for dynamic optimization with Q-learning. The results obtained on popular benchmarks prove that Q-learning performs competitively against previous algorithms. The authors in [41] propose using multi-state Markov decision processes combined with multi-trajectory Least-Squares Temporal Difference to decide whether one or more machines have to be replaced. However, as far as we know, no previous work attempted to address simulation-based dynamic optimization problems with metric spaces. Moreover, different from previous work using RL, our proposal does not require a time-consuming training phase and allows for continuous assessment of diverse scenarios, simultaneously achieving multiple objectives.