## Materials and Methods
### Ranking of Features Based on Expert Opinions
#### 2.6.6 Determining the Ranks of the Attributes in Decision Making  
The preference rates \(p_{i}\) can be represented as rankings in a rather straightforward way for each of the \(m\) experts. First, the preference rates should be ordered in decreasing order \(p_{i_{1}}\), \(p_{i_{2}}\),..., \(p_{i_{n}}\) for each of the experts. For expert \(j\), the attribute having the highest preference rate \(p_{i_{1}}\) should have the 1 st rank, i.e., \(r_{i_{1}j}=1\), the one with 2 nd highest preference rate \(p_{i_{2}}\) should have the 2 nd rank, etc. This results in each of the experts \(j\) having their own ranking list \(r_{ij}\). \begin{table}
\begin{tabular}{l l l l l l l l} \hline \hline  & **Size** & **Shape** & **Other** & **Contour** & **Density** & **Homogeneity** & \(a\) \\ \hline Size & & & & & & & 0 \\ Shape & 1 & & & 1 & & 1 & 3 \\ Other & 1 & 1 & & 1 & & 1 & 4 \\ Contour & 1 & & & & & 1 \\ Density & 1 & 1 & 1 & 1 & & 1 & 5 \\ Homogeneit & 1 & & & 1 & & 2 \\ \hline Sum & 5 & 2 & 1 & 4 & 0 & 3 & 15 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The preference table of Expert 2 is used as an example. The number of attributes is \(n=6\). The first column, as well as the first row, shows the attributes, the 6 \(\times\) 6 matrix shows 1 for those cases when the attribute in the row is preferred to the attribute in the column (the diagonal is of course empty). Column \(a\) is the number of cases when the given attribute is preferred (this is shown in the figures later in Section 3.1). 2.6.7. Calculating the Kendall Agreement Index  
The Kendall agreement index [44]\(W\) is a well-known quantity for determining the grade of agreement between experts, it quantifies the inter-expert reliability on a scale between 0 (total disagreement) and 1 (total agreement) according to the following formula,  
\[W=\frac{12\cdot\sum_{i=1}^{n}\left(R_{i}-\overline{R}\right)^{2}}{m^{2}\cdot(n^{ 3}-n)}, \tag{4}\]  
where  
\[R_{i}=\sum_{j=1}^{m}r_{ij}\qquad\text{ for }i=1,\ldots,n \tag{5}\]  
is the total ranking of the \(i\)th attribute,  
\[\overline{R}=\frac{1}{n}\sum_{i=1}^{n}R_{i} \tag{6}\]  
is the average ranking, \(m\) the number of experts, and \(n\) the number of attributes. 2.6.8. Building the Aggregated Preference Table  
If the preference tables of all the \(m\) consistent experts are summed, then an aggregated preference table can be prepared. The aggregated preference table has the value \(a_{i}\) in its last, summarizing column, similarly to Table 1, but in this case, variable \(a_{i}\) contains the number of all cases when any of the consistent experts prefers the \(i\)th attribute to any other attributes. For calculating the aggregated preference rate for the \(i\)th attribute, the formula  
\[P_{i}=\frac{a_{i}+\frac{m}{2}}{m\cdot n},\ \text{ for }i=1,\ldots,n \tag{7}\]  
should be used, with \(n\) being the number of attributes, and being \(m\) the number of consistent experts. 2.6.9. Calculating the Weights of Each Attribute  
From these values the weights [45]\(Z\) can be calculated by  
\[Z_{i}=\frac{u_{i}-\min(u)}{\max(u)-\min(u)}\cdot 100\text{ in }\%\qquad\text{ for }i=1,\ldots,n, \tag{8}\]  
where \(u_{i}\) is the standard normal distribution value corresponding to \(P_{i}\). This means that the weight \(Z_{i}\) that was proposed by [PERSON] and [PERSON] is a simple normalization of the standard normal distribution values \(u_{i}\) between their minimum and maximum. These weights can be used in fuzzy signatures [10,11] and other expert systems. 3.1.1. Selecting the Set of Attributes  
Size, shape, contour, density, and homogeneity were considered the most important features based on the textbooks for CT liver diagnosis [38, 39, 40]. All other properties were summarized in an attribute, called \"other\", which might need to be elaborated more in a later phase of the study. 3.1.2. Generating a Pairwise Comparison Questionnaire  
The questionnaires were anonymous, and 28 experts filled and sent them back. The text of the questionnaire can be found in Appendix B.