## 1 Introduction
### Validity of the Three Metrics Used in External R&D Evaluations
For that reason, the decision of R&D policy actors at the Slovenian Research Agency to use the citation index in such a restrictive way cannot be based on any kind of rational argument. It would be expected that the shortcomings of the Web of Science citation database with regard to the social sciences and humanities would lead to a more flexible counting of citations, _i.e._, all citations in Web of Science, not just citations from scientific articles that have a full bibliographic record in Web of Science. This is completely contrary to recent trends that set aside the classical bibliometric indicators, such as the crude measures of a journal's impact factor, number of citations _etc._, in favor of more sophisticated bibliometric indicators. For example, in the recent period the Hirsh index has been built to measure both the actual scientific productivity and scientific impact of a scientist [37]. The index is based on a set of a scientist's most quoted papers and the number of citations they have received in other scientists' publications. The advantages of this index include its simplicity, the fact that it can combine the citation impact with publication activity and that is also not affected by single papers that have many citations. Concerning the third parameter, namely the data about funding from non-Agency sources (_i.e._, third-party funding) calculated in the full-time equivalent of employment of scientists (FTE), which is also regarded as an important indicator for selecting a grant recipient in Slovenia, we can also offer a series of critical remarks. The most serious problem with this indicator in Slovenia is that it is difficult to achieve a consensus among R&D policy actors and scientists concerning the role funding from various non-Agency sources plays in the context of measuring R&D performance. Thus, while the data concerning the amount of money received by applicants in the last 5 years from various non-Agency sources may represent the ability of individual scientists to successfully commercialize research results, it is also possible, if not indeed likely, that this crude financial indicator induces some type of double counting. Although it is quite easy to measure financial streams, there is much criticism from bibliometric experts regarding the use of both indicators for the same purpose. As many bibliometric experts note, the validity of third-party funding as a measure of R&D performance has not yet been comprehensively proven [38, 39, 40]. In that sense, the validity and reliability of this indicator in R&D ex-ante evaluations in Slovenia can only be ascertained if the data concerning third-party funding is supported by additional information. Without this additional information, the crude data about financial streams could even result in misleading information. If we demand that metrics need to be developed in context, then any kind of ambiguity in the interpretation of the data must be avoided. In the Slovenian case, it would be reasonable to upgrade the crude indicator of third-party funding as a measure of R&D performance with additional bibliometric indicators, _i.e._, co-authored publication networks. The knowledge flows between researchers from the academic scientific sector and various societal sectors outside the academic scientific sector (business-enterprise sector, civil society, _etc._) through common publications have become a more important bibliometric indicator since late 1990. This pervasive trend reflects the increasing orientation of academic researchers towards joint knowledge created with partners from outside of academic science [41]. In the Slovenian case, all the conditions for a more objective analysis of the patterns of collaboration through co-authored publications are already presented. Namely, in the SICRIS database the author names of co-authored publications are normalized and disambiguated.