# A Literature Survey on Offline Automatic Algorithm Configuration
## 4 Other Studies
### Studies on Different Aspects of the Algorithm Configuration Problem  
An important barrier to the improvements in algorithm configuration (AC) is a scarcity of empirical studies and reproducible experiments. The hyper-parameter optimization library HPOlib [121] and the algorithm configuration library ACIib [122] was developed to handle this obstacle. However, even with such benchmark libraries available and easy to use to compare different methods, testing new parameter tuning techniques is hard because AC benchmarks cannot be easily set up and the evaluation of the AC method's performance is computationally expensive. [PERSON] et al. [123] proposed using surrogate benchmarks that consider overall important features of the algorithm configuration cases instead of the expensive original algorithm configuration problem. They generate a new set of eleven surrogate benchmarks, nine of which are for optimizing running time, and the rest are for optimizing solution quality. All procedures are publicly available at ([[http://www.ml4](http://www.ml4) aad.org/algorithm-analysis/epms/]([http://www.ml4](http://www.ml4) aad.org/algorithm-analysis/epms/) (accessed on 10 June 2022)). Additionally, these surrogate scenarios can substantially speed up configurators, and they facilitate the comparison and evaluation of the performance of algorithm configuration methods. [PERSON] et al. [124] claimed that default parameter settings of optimization algorithms involve important information because default parameter values are commonly chosen based on deep insights into the algorithm controlled by the parameters, and they can be used in parameter tuning. They compared state-of-the-art configuration methods with and without access to meaningful default parameter vectors to explore the impact of the default configuration on these configuration methods. Their first results showed that widely used automated algorithm configuration methods moderately use the information produced by default parameter settings. Although SMAC rarely benefits from good defaults, default values are very important for irace. Secondly, they studied whether the configuration process could benefit from the information provided in the default by reducing the range of the parameters. They suggested two simple methods, which are guided by default values, for reducing the ranges of the target algorithm's parameters. They improved the performance of automatic configuration methods by decreasing the range of possible values and, thus, the size of the search space. In particular, their reduction techniques increase the efficacy of SMAC for 15 out of the 20 configuration scenarios, that of GGA++ for 7 out of the 9 scenarios, and that of irace for a scenario on which they previously obtained only minor improvements. They also compared their results with the recently suggested warm-starting SMAC [125] and they concluded that default-guided search space reduction provides similar and complementary benefits. Additionally, this method can be implemented in a wider range of automatic configuration methods in contrast to warm-starting. [PERSON] et al. [126] discussed the pitfalls that have been encountered in algorithm configuration experiments (such as over tuning) and suggested the best ways to discard these pitfalls by developing a tool called GenericWrapper4 AC (available at [[https://github.com/autom1/](https://github.com/autom1/)]([https://github.com/autom1/](https://github.com/autom1/)) GenericWrapper4 AC (accessed on 10 June 2022)). This package provides an interface between algorithm configuration procedures and target algorithms to obtain reliable, reproducible, and robust algorithm configuration experiments. At the end of their study, they gave general recommendations for effective configuration. In [PERSON] et al. [127], the authors studied the effect of transformation of the parameter settings for parameter tuning. They considered five different transformations of a single parameter and examined their effects under various configuration budgets.