# Algorithm for Producing Rankings Based on Expert Surveys
## 2 Basics of SSRR  
The purpose of the SSRR methodology is to create rankings based on input from multiple experts while avoiding distortions due to differing standards among the experts. Rather than first generating grades and then translating them into a ranking, SSRR starts by ranking small groups of units relative to each other and subsequently merges these groups to create one comprehensive ranking. Thus, SSRR is a pure approach to ranking, starting with small ranking clusters and ending with an aggregate ranking.  
The basic principle of SSRR is as follows. Experts compare and rank units in relation to other units. Each expert ranks only those units they are familiar with, creating a small ranking referred to as a segment. Subsequently, these segments are strung together into a combined ranking. For example, Expert I may say that she is familiar with units \(P\), \(Q\), and \(Y\) and rank them; whereas Expert II may say that he is familiar with units \(Y\), \(H\), and \(K\) and rank them. Combining the input from the two experts, we can create the larger ranking as shown in Figure 1.  
\begin{table}
\begin{tabular}{c p{284.5 pt}} \hline \hline
**Grade** & **Description** \\ \hline \multirow{3}{*}{A} & Excellent performance: strong evidence of original thinking; good organization; capacity to analyze and synthesize; superior grasp of subject matter with sound critical evaluations; evidence of extensive knowledge base; high level of motivation. \\ \hline \multirow{3}{*}{B} & Good performance: evidence of grasp of subject matter; some evidence of critical capacity and analytical ability; reasonable understanding of relevant issues; regular attendances of classes; productive contribution to the discussion by sharing thoughts and questions that demonstrate familiarity with the material; interest in other students’ contributions. \\ \hline \multirow{3}{*}{C} & Average performance: understanding of the subject matter; ability to develop solutions to simple problems in the material; acceptable but uninspired work, not seriously faulty but lacking style and vigor; meeting the basic requirements of preparedness and regular attendance; rare participation in class discussion. \\ \hline \multirow{3}{*}{D} & Poor performance: inconsistency in attendance and preparation for classes; lack of participation in class discussions; absence of respect for the contributions of other students. \\ \hline \multirow{3}{*}{E} & Inadequate performance: little or no evidence of understanding of the subject matter; weakness in critical and analytic skills; limited or irrelevant use of the literature. \\ \hline \hline \end{tabular}
\end{table}
Table 1: An example of detailed grading guidance: grading system of the Academy of the Organization for Security and Cooperation in Europe (OSCE).  
One way of thinking about SSRR rankings is that they are based on inference. In Figure 1, combining the input from Experts I and II makes it possible to infer that \(Q\) and \(P\) should be ranked above \(H\) and \(K\).  
One premise for stringing together segments from different experts is that there is always some overlap between the segments, i.e., that at least one unit from each segment is ranked by two or more experts. This ensures that the segments can be strung together. In Figure 1, \(Y\) is ranked by both Experts I and II, providing a linkage between their two ranking segments and making it possible to string them together. If an expert ranks only units that nobody else has ranked, the input from this expert becomes an isolated island in the data and cannot be included in the ranking. It is therefore important to ensure that there is always some overlap between the input from experts.