# A Literature Survey on Offline Automatic Algorithm Configuration
## 5 Concluding Remarks  
Although automatic algorithm configuration techniques have been developed and used for more than a decade, an effective solution to the parameter tuning problem of highly-parameterized algorithms and commercial solvers has recently become possible. This success is thanks to methodological advances in recent configuration procedures. We anticipate that the algorithm configuration topic will be crucial for the development and implementation of metaheuristics and will also be extensively used in industry and academia.  
This topic is an abundant research domain, and there are many questions that should be researched and there are open directions for the future, such as:  
* An additional effort is essential to reduce the computational cost of the tuning process. Developing different performance evaluation techniques and using efficient sampling methods in the initial phase of the tuning process will facilitate achieving this goal.
* With the fact that different configuration procedures can be successful for various types of algorithm configuration problems, it is still unclear which algorithm configuration method should be chosen for a specific algorithm configuration scenario. [PERSON] et al. [12] suggested developing an automatic selection strategy based on the target algorithm and target problem.
* Additionally, although the open-source codes of these automatic configuration methods are proposed, they require practitioners to have satisfactory programming knowledge. A well-documented and easily-available toolbox, allowing integration of newly developed algorithm configuration techniques, is also an important requirement for end-users and practitioners. There are some toolboxes (such as irace and SPOT) out there, but their feature richness, documentation, and usability could be improved.
* Studies on developing benchmark libraries (such as ACLlib), which also enable the integration of tuning methods, will help the extension of empirical studies in this area.  
Until now, a large collection of independent configuration procedures were suggested and summarized in our paper. However, it is expected that algorithm configuration methods will be improved in a more structured research area [135]. We anticipate that this study will be a guide for understanding the general perspective of the studies in this area.  
As a final point, in this paper we provided a short introduction to the framework of the algorithm configuration problem, and also provided a survey about offline algorithm configuration approaches and related works with these methods. We used a modified classification of tuning methods based on the structure of configuration methods, such as model-free tuning methods, model-based tuning methods, DOE-based tuning methods, and racing methods. After explaining the logic of these methods, we also argued about their main advantages and disadvantages to help researchers or practitioners select the best method for their specific problem. Additionally, then, we mentioned comparative studies of these methods, related research topics with the algorithm configuration problem, and other studies that consider different aspects of this problem. We provided a comprehensive review of automatic algorithm configuration methods for researchers or practitioners. Moreover, some recommendations and possible future directions for this topic are provided as a conclusion.  
Conceptualization, Y.E.; methodology, Y.E.; validation, A.D.; formal analysis, A.D.; investigation, Y.E.; resources, Y.E. and A.D.; writing--original draft preparation, Y.E.; writing--review and editing, Y.E. and A.D.; visualization, Y.E. and A.D.; supervision, A.D. All authors have read and agreed to the published version of the manuscript.  
This research received no external funding.  
Not applicable.  
The authors declare no conflict of interest.