# A Literature Survey on Offline Automatic Algorithm Configuration
## 3 Comparative Studies  
There are some studies conducted to compare the performance of automatic algorithm configuration methods. These comparative studies are scarce in the literature, and [PERSON] (2019) summarized the reasons as follows: such a representative study requires a set of different benchmark problems and algorithms; it must consider different cases, such as different performance metrics to be optimized; multi- and per-instance setups; benchmarks comprising different parameter types. Additionally, the computational cost of such a study is the most significant difficulty that researchers may face. In this section, we will summarize a few comparative studies of automatic algorithm configuration techniques. [PERSON] et al. (2018) described and compared four automated algorithm configuration procedures: F-Race, Revac, ParamILS, SPO, and the blind random search (BRS) method. Additionally, they analyzed their advantages and disadvantages in terms of quality of results, the effort required, information delivered, usability, user-friendliness, and behavior in different scenarios. A standard genetic algorithm (SGA) used for solving classical continuous test functions was used for the evaluation of these tuning methods. They discussed the requirements of each method, summarized the main global conclusions from their study, and established some guidelines for users that can help to select the most suitable configuration method. For details of the numerical results, we refer to the original paper. [PERSON] et al. (2019) suggested a new algorithm configuration technique that is a combination of a recently developed method for comparing and ranking evolutionary algorithms Chess Rating System for Evolutionary Algorithms (CR5 AEAs) and meta-evolution methods. Another important objective of this study is to compare the proposed configuration method with two other procedures: F-Race and Revac. According to the results of this study, all three methods have strengths and weaknesses in different domains (which satisfies the No Free Lunch theorem), and the CRS-Tuning method did not perform worse than the compared techniques. [PERSON] et al. (2019) proposed MOCRS-Tuning (Multiobjective CRS-Tuning), an improved version of CRS tuning, to answer the question of \"what must be a sufficient number of problems used in the tuning process to obtain robust enough parameters?\" Their results showed that while finding better configurations is not guaranteed when using a bigger subset of problem instances, the best parameter values can also be obtained by using a small problem instance subset. This also essentially reduces the required time for the tuning process. [PERSON] and [PERSON] (2019) discussed the most crucial issues about configuring evolutionary algorithm parameters, and they executed a modest experimental comparison among selected algorithm configuration methods. Their main goal is to demonstrate the feasibility of using algorithm configuration methods and motivate their usage. They considered three algorithm configuration methods (meta-EA (CMA-ES algorithm is selected as a meta-EA), meta-EDA (Revac), and Sequential Parameter Optimization) and discussed their advantages and disadvantages when configuring evolutionary algorithm parameters. Add-ons, such as racing, sharpening, and a combination of them, are used to increase the search efficiency of tuning methods. For details of these add-on methods used in the algorithm configuration literature, we refer to (2019; 2020). Revac's performance improved with racing and sharpening, resulting in Revac++. The performance of EA is tested on the Rastrigin function. Although their experiment was conducted on one single test function and one EA, they obtained some interesting conclusions. The most important one is that all of the tested algorithms significantly outperform the human experience-based parameter values. With respect to their conclusion, the rank of the tested three methods is different when regarding different tasks of tuning methods.