## 3 Methodology and Criteria for a Comparative Evaluation of the Guidelines
### Survey Evaluation
#### 3.3.1 AHP and Weight Calculation  
_Analytic Hierarchy Process_ (AHP) is a method for systematic decision support between alternatives [60]. It turns abstract priorities of a decision maker into quantitative weights and thus helps to make objective decisions. One of the advantages of the process is that it can be used by a single decision maker but is also suitable for group decisions as well. A disadvantage is that criteria and possible features should be well known. Furthermore, priorities can still be biased and subjective. Nevertheless, the quality of a decision is tremendously increasing compared to a free decision without any supporting process. The decision making process is composed of the following steps, as described by [60]:  
1. Define the problem and determine the kind of knowledge sought. 2. Structure the decision hierarchy from the top with the goal of the decision, then the objectives from a broad perspective, through the intermediate levels (criteria on which subsequent elements depend) to the lowest level (which usually is a set of the alternatives). 3. Construct a set of pairwise comparison matrices. Each element in an upper level is compared with the elements in the level immediately below it. 4. Use the priorities obtained from the comparisons to weight the priorities in the level immediately below. Do this for every element. Then for each element in the level below add its weighted values and obtain its overall or global priority. Continue this process of weighing and adding until the final priorities of the alternatives in the bottom level are obtained. The pairwise comparison is based on a two sided scale. Each criterion is compared to every other criterion. Two criteria are compared by putting them on the opposite sites of the scale. The original scale features nine levels of intensity of importance. The center of the scale represents level 1 that represents equal importance for both criteria. The other eight levels extend to both sides towards the two criteria. Figure 6 shows a graphical example of such scale for better understanding. If a criterion x is assigned with one of the nine values when compared to criterion y, then y receives the reciprocal value when compared to x [60]. Figure 6: Graphical example scale for category comparison. The results are entered into a matrix. If only one decision maker is performing the process, it is suitable to directly fill in the matrix without using a graphical scale. If the answers are collected using a survey, a graphical scale is more appropriate because participants do not need to understand the complete process and a matrix might be too demanding. From this matrix the weights (priorities) for criteria (alternatives) can be derived. This is conducted by normalizing the matrix, adding up each row and dividing them by their totals. This method has been proven to be close to actual statistical results [60]. In order to fill in the matrix, answers of the participants are aggregated by averaging. Since only half of the matrix can be filled directly via the answers, the corresponding fields are completed with the reciprocal values. Table 1 is established through this process. Table 2 shows the matching of short names used in the tables for the criteria. Criteria are from here on also called categories or main categories. For simplicity the scale is reduced to five intensity levels. In general the scale is kept as suggested by [PERSON]. Only a small number of categories are chosen to keep the number of necessary comparisons as small as possible, since the number of comparisons grows exponentially with the number of categories. There are also subcategories for each category. AHP was not used on the subcategories. The process for the subcategories is explained in Section 3.3.2. The categories and subcategories are based on the IPv6 topics which are covered by the RFCs of IETF and are possible content for a secure deployment guide.