# Algorithm for Producing Rankings Based on Expert Surveys
## 1 Introduction  
Expert opinion can be useful for the creation of rankings when more objective data are not available [1]. An example of a ranking based on expert opinion is the Corruption Perception Index (CPI) published by Transparency International [2]. It is difficult to obtain objective data on corruption, so instead, CPI is created on the basis of expert opinion [3, 4]. Although the views of the experts whose input is used for the CPI are based on imperfect information and carry a risk of bias [5, 6, 7], the resulting index is nonetheless a useful tool to inform the political choices of citizens, guide companies in their investment decisions, and help governments understand how their level of corruption is seen [2, 8].  
Most expert-based rankings involve experts grading units in some way or other [9]. However, experts are liable to apply grades with different degrees of strictness, and this can distort a ranking [10]. For example, if experts are asked to grade countries from 5 to 1, where 5 is best and 1 is worst, some experts will liberally dish out 5s, while others will strive to use the whole grading scale evenly, and yet others will be very strict and mostly give low grades. Individual experts may also vary in their strictness over time depending on their mood and other subjective factors.  
The problem with grading is particularly severe when different clusters of units are assessed by different experts [11]. For example, if one expert assesses European countries and another assesses Latin American countries and one of the experts is generally stricter than the other, this may lead to countries from one region getting an unfair advantage.  
In order to overcome this problem, some systems include elaborate criteria to ensure that all experts do their grading according to the same standards [11]. For example, the Academy of the Organization for Security and Cooperation in Europe (OSCE) uses the descriptions in Table 1 to guideexperts--that is to say, professors grading student papers. This helps even out some differences among professors, but it is still up to each professor to define what counts as _good_ organization, _superior_ grasp of the subject matter, etc. Even if a professor manages to make fair judgements among the papers submitted by students taking his or her course, it is difficult to know whether a professor on a different course might be generally more strict or lenient. Especially if the courses areectives with some students taking one course and others taking another, this can distort the overall grade point averages (GPAs) of students.  
The existing literature includes several techniques for addressing the weaknesses of the traditional way of dealing with criteria-based input e.g., [12; 13]. Based on these improvements, criteria-based input has been effectively processed in several cases e.g., [14; 15].  
The segmented string relative ranking (SSRR) approach, first proposed by [PERSON] [16], goes one step further and omits criteria altogether. However, SSRR remains unwieldy when applied manually, demanding a lot of work and carrying a risk of human error in calculations. There is thus a gap in the literature when it comes to increasing the efficiency of SSRRs, and this article seeks to fill that gap by developing an automated algorithm so that SSRRs can be produced using computers.