Complex tasks and simple solutions: The use of heuristics in the evaluation of research

Sir, When we approach the complex task of evaluating the scientific contributions of a unit of research (e.g., a manuscript, research proposal, or scientist), we tend (or rather, it is almost unanimously recommended) to seek a complex solution. Scientific contributions are regarded as a multidimensional phenomenon that research evaluation can only approach in a complex way. So, for example, the U-Multirank was developed as a new ranking of universities in which the performance of an institution is measured by a large number of indicators (van Vught & Ziegele, 2012). This large number is not only used to determine the scientific impact of an institution on quite different sectors of society but also to measure the impact multidimensionally within a sector (e.g., within science). Here the aspects involved in science may include productivity, citation impact, cooperations, and external funding. The assessment of research achievements in science is mainly conducted using peer review (Bornmann, 2011). In many areas where this procedure is applied it is used in an elaborate way. For example, institutional evaluations involve an internal evaluation, in which an institution evaluates itself and writes a self-evaluation report, followed by an external evaluation, in which a group of experts visits an institution for several days and compiles a report on the quality of the institution on the basis of their own impressions and the self-evaluation. The procedures in the evaluation of research are complex, mainly because recommendations and decisions in this area are associated with a high degree of uncertainty. It is generally regarded as necessary to counter the uncertainty with a complex evaluation process. But as the results of Gigerenzer and Gaissmaier (2011) show, in an uncertain world a complex measurement or a complex procedure is not always better than a simple measurement or a simple procedure. In a complex measurement, details are often taken into account, many of which are not relevant for the assessment. In addition, in many cases it is hardly possible to take into account all aspects of a situation. For example, the expert assessment of scientific work often has to take place under severe time pressure. Thus, for instance, an institutional evaluation may involve a complete university department being assessed by an external evaluation group within a few days. In such situations, according to Gigerenzer and Gaissmaier (2011), it is better to avoid a complex measurement and trusting the intuition of those making the decision. We can assume that scientists actually apply heuristics when conducting evaluation tasks (although they are probably not aware of applying them). Intuition is a form of unconscious intelligence that may be almost impossible to articulate but that can work relatively precisely in forming a judgment. According to Marewski and Gigerenzer (2012), intuition is based on simple, intelligent heuristics that only take into account some of the available information. “A heuristic is a simple decision strategy that ignores part of the available information and focuses on the few relevant predictors” (p. 78). So, for example, with the “take-the-first” heuristic, decision makers choose the first alternative that occurs to them. The “fluency” heuristic is defined as follows: “If both alternatives are recognized but one is recognized faster, then infer that this alternative has the higher value with respect to the criterion” (Gigerenzer & Gaissmaier, 2011, p. 462). Further heuristics are the “take-the-best” heuristic and the recognition heuristic (Marewski & Gigerenzer, 2012), which is discussed later because the recognition heuristic is regarded as especially relevant to research evaluation. Whereas the complex model that may be applied to a decision process (e.g., statistical models) contains all the available information, heuristics only take into account part of the information. With heuristics, people can relatively quickly and simply arrive at a sound judgment (Gigerenzer & Gaissmaier, 2011). A range of studies showed that intuitions, which are applied in the right circumstances and are based on heuristics, are not only economical but often accurate as well. This is particularly true in those situations that are complex and hard to forecast and in which the predictors are correlated with one another (Marewski & Gigerenzer, 2012). Therefore, the use of heuristics seems particularly appropriate in the evaluation of research because (a) the judgments of quality of research are associated with a high degree of uncertainty, and (b) a good (or bad) scientific performance often—in line with the Anna Karenina principle (Bornmann & Marx, 2012)—tends to show up similarly in different areas (such as publication success, external funding, and lecturing activity), that is, the predictors are correlated with one another. Because the use of heuristics in the evaluation of research has hardly been (empirically) researched, it is not clear how frequently they are applied: there is neither a breakdown of the heuristics frequently or infrequently applied nor empirical investigations into the validity of the judgments based on certain heuristics. There is a lack of formal models of the heuristics in this area: “Formal models of heuristics allow asking two questions: © 2014 ASIS&T