# A Literature Survey on Offline Automatic Algorithm Configuration
## 6 Multi-Objective META-EAs
### Racing Methods  
#### 2.2.1 F-Race  
[PERSON] et al. [57] introduced an automatic racing procedure for automatic algorithm configuration problem based on the methods from the machine learning literature for model selection [58; 59] and [PERSON]'s nonparametric two-way analysis of variance by ranks [60]. The main difference between the racing methods and the system selection methods is that the racing methods deal with a larger set of competing systems when compared with the system selection methods. This procedure aims to find the best-performing parameter vector of a target algorithm from a finite configuration set by eliminating bad-performing ones whenever they are proven to be poor statistically, and the procedure is iterated over the remaining ones. The F-Race process starts with performing n runs for each configuration to acquire sufficient data before elimination. Obtained data are recorded for each parameter configuration. At each iteration, a problem instance is randomly selected for the performance comparison of candidate parameter configurations. After evaluating configurations on every instance, costs (or utilities) of each are added to the stored values. After each iteration, [PERSON]'s two-way analysis of variance by ranks, a nonparametric statistical test, is used to control if there are significant differences among the configurations. If the null hypothesis is rejected with a significance level, it can be inferred that at least one parameter configuration is better than at least one other parameter configuration. After that situation, a pairwise comparison between parameter configurations is performed. Statistically poor configurations are discarded from the candidate's configuration set. The termination condition is either a remaining one parameter vector or reaching a previously defined time budget, and it gives a best performing configuration set as an output. A block design based on different problem \begin{table}
\begin{tabular}{c c c} \hline \hline
**Methods** & **Advantages** & **Disadvantages** \\ \hline GGA++ & GGA++ is among the most competitive and robust tuners suggested. & \\  & The main advances of GGA++ are its tree-based representation and used operators that can handle any type of parameters (both numerical and categorical parameters without discretization) & \(\bullet\) GGA++ could not deal with the complex parameter types (such as conditional parameters) that can occur in some solvers [56]. & \\  & GGA++ uses an adaptive capping mechanism like ParamILS, and it combines it with a parallelization mechanism to use multiple processing units & \(\bullet\) GGA++ could not deal with the complex parameter types (such as conditional parameters) that can occur in some solvers [56]. & \\ \hline ParamILS & It is a sophisticated method and can be used for tuning various algorithms, even if they have many parameters (categorical and numerical), and always demonstrated a very good performance in the vast majority of cases studied [47]. & \(\bullet\) Because ParamILS is based on stochastic local search, the selected random seed can affect its performance. & \(\bullet\) Major drawback of ParamILS is that it can only deal with discrete parameters and so it can be uninformative in terms of alternative parameter vectors and the parameter’s space structure. Furthermore, it is difficult to analyze interactions between parameters, specific instances, and parameter configuration correlations [14; 47]. & \(\bullet\) Capping mechanism cannot be efficient when the performance criterion is solution quality [13]. \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Cont. instances (one instance considered as a block) is applied by F-Race. The null hypothesis of the Friedman test claims that all possible rankings of the parameter vectors within each block are equal.