Bayesian optimization builds a model of the search space and optimizes the error over the course of iterations, efficiently tuning hyperparameters and selecting the best models.