# Combinatorial Optimization Problems and Metaheuristics: Review, Challenges, Design, and Development
## 4 Results: Standardization, Methodology, and Framework for COP and Metaheuristics
### Framework Structure
#### 4.1.1 Repository (Local and Global)  
The repository is designed to maintain custom components, widgets, and templates developed to take advantage of framework preexisting services. These components must follow the software design pattern (standards) defined by the framework. By way of illustration, these components can be (1) combinatorial optimization problem templates, (2) metaheuristics, (3) subordinated heuristic, (4) external environment, (5) benchmark test (problem instances and a particular search configuration), etc. The repository was designed into two levels, (1) global and (2) local. Global repo is accessible to everyone, and the community sustains it; they can download or upload components. The local repo represents local storage, where a person who works in a COP experiment downloaded components from the global repo or kept his/her developed new \"pieces of software\". Furthermore, the repository must be supported by a database that maintains information about the components. The concrete component in the repository must have metadata to describe them, and it should be stored in the database. Additionally, COP or metaheuristics could consider the preliminary classification system presented in Sections 2.3 and 2.6 as a primary reference to form their metadata. Consequently, each type of component should be split into categories, and each type of component defines its own metadata. It should also consider the component author/owner (code), algorithm creator, paper that proposed the metaheuristics (if it is sharable--copyright), etc. Therefore, these components can be tagged properly according to their type. This metadata system also can support a search engine, enabling the filtering of the components by some criteria. Additionally, this database could also supply the main concepts of metaheuristics and COP (analogous to a wiki). Moreover, the repository must provide a benchmarking process to compare metaheuristic performance. Thus, the database also should keep benchmarks of metaheuristic performance. This section of the database, in a summarized way, needs to keep two typesof information. The first type of information is the benchmark test, and they can be seen as templates that the community will reuse. A benchmark test template comprises several problem instances (data) designed to evaluate metaheuristic performance in investigating these search spaces (data). The second category of information results from metaheuristic performance after performing a specific benchmark test (template). These tests must follow an experimental design and provide statistical information about their performances. Figure 6 shows the basic relationship between metaheuristics and benchmark tests. The critical success factor is to develop wide and challenging tests by covering several aspects to defy the metaheuristics capacity. Consequently, a good test could consider different levels of difficulty and search space landscapes. These tests also could consider different types of problems such as routing, planning, scheduling, etc. Additionally, they can consider problems with a huge number of dimensions and so on so forth. The objective behind this is to generate _reference tests_ that can be used as scientific standards to appraise and certify metaheuristics' performance. Therefore, from a community perspective, it demands proposals of challenging benchmark tests by covering several aspects to defy the metaheuristics capacity. We believe it is an open question, and it could be a good research line to develop the field. Research questions such as the following could be addressed. (1) What tests are capable of evaluating the capacity of a metaheuristic as a searcher method? Or still, (2) what are the dimensions that should be explored and considered in these tests? Thus, several metaheuristics can be subjected to the same batch of tests and their performance can be compared. Additionally, the tests already performed do not need to be executed again because they are already stored in the database.