Source document: https://docs-lawep.s3.us-east-2.amazonaws.com/1698060033665.pdf (Liberia)
May 2022 Impact Evaluation of the Icelandic Research Fund Government of Iceland Ministry of Higher Education, Science and Innovation Publisher: Ministry of Higher Education, Science and Innovation Impact Evaluation of the Icelandic Research Fund Quality Enhancement Framework for Icelandic Higher Education May 2022 hvin@hvin.is www.hvin.is Design: Ministry of Higher Education, Science and Innovation ©2022 Ministry of Higher Education, Science and Innovation ISBN 978-9935-9678-1-7 2 Table of Contents List of Figures .
.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

4 List of Table .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.


4 Samantekt __________________________________________________________________________________________________________ 5 Executive Summary _______________________________________________________________________________________________ 6 Introduction ________________________________________________________________________________________________________ 7 Icelandic Research Fund .

.

.

.

.

.

.

.

.

.


.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

7 Description of the IRF .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.


7 Methodology ______________________________________________________________________________________________________ 10 Timeframe .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

10 Data Sourcing and Analysis .

.

.

.

.

.

.


.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

10 Data Limitations .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.


10 Survey .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

11 Interviews .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.


.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

11 Results _____________________________________________________________________________________________________________ 12 1.

Funding Patterns of IRF .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.


.

.

.

12 1.1.

Application Process .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

12 1.2.

Success Rate by all IRF Grants .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.


.

.

.

.

.

.

.

.

.

.

.

.

15 1.3.

Success Rate by Gender .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

16 1.4.

Success Rate by Discipline .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.


17 2.

Springboard to Other Activities and Funding .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

19 2.1.

Springboard to Other Funding .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.


19 2.2.

Effect on Other Activities .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

21 2.3.

Consequences of non-funding .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.


22 2.4.

Development of a Scientific Idea .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

23 3.

Scientific Outputs .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.


24 4.

Educational Engagement .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

29 5.

Social, Cultural and Economic Impact .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.


30 Conclusions _______________________________________________________________________________________________________ 34 Appendix A ________________________________________________________________________________________________________ 36 3 List of Figures Figure 1: IRF Logic Model .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.


.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

9 Figure 2: Number of applications by grant type (2011-2022) .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

15 Figure 3: Application success rate by grant type (2011-2022) .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.


.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

16 Figure 4: Success rate by gender (2011-2022) .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

17 Figure 5: Number of applications by gender (2011-2022) .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.


.

.

.

.

.

.

.

.

17 Figure 6: Success rate, natural sciences v. social sciences and humanities (2011-2022) .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

18 Figure 7: Number of years it took PIs to receive a grant, after having their first application rejected .

.

.

.

.

.

.

20 Figure 8: Number of applications to success .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.


.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

21 Figure 9: The proportion of peer-reviewed journal articles published each year out of the total peer-reviewed journal articles published in the six-year period from receiving an IRF grant .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.


27 Figure 10: The proportion of researchers’ citations out of their total citations in the six-year period from receiving and IRF project grant .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.


28 Figure 11: Number of Masters and Doctoral student in IRF-funded projects (2011-2015) .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

29 List of Tables Table 1: Survey response on RANNIS’ communication .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.


13 Table 2: Sample responses to the question “How did you learn how to write a grant?” .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

14 Table 3: Sample responses to the question “What happens once a study is funded?” .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

19 Table 4: Sample responses to the question “What is IRF’s impact on scientists?” .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.


.

.

.

.

22 Table 5: Sample responses to the question “What are the consequences of non-funding?” .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

22 Table 6: Sample responses to the question “How does a scientific idea emerge?” .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

24 Table 7: Sample responses to the question “What is IRF’s impact on society – scientists’ visibility in society?” .
.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

30 Table 8: Sample responses to the question “What is IRF’s impact on society – trainees as potential employees?” .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.


.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

32 Table 9: Sample responses to the question “What is IRF’s impact on society – importance of Icelandic universities?” .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.


.

.

.

.

.

.

.

.

.

.

32 Table 10: Sample responses to the question “What is IRF’s impact on society – education and training of scientists?” .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.


33 4 Impact Evaluation of the Icelandic Research Fund May 2022 Samantekt 1.

Bakgrunnur Tilgangur matsins var að framkvæma megindlega greiningu á styrkveitingum Rannsóknasjóðs Íslands, sýnileika hans og áhrifum.
Skoðað var hvernig styrkveitingar dreifðust, hvort styrkveitingar leiddu til frekari rannsókna og annarra styrkja, hvernig styrkir studdu við framlegð í vísindastarfi og menntun nýrra vísindamanna, auk þess sem félagsleg, menningarleg og efnahagsleg áhrif styrkja voru greind.

Verkefnastyrkir, öndvegisstyrkir, rannsóknastöðustyrkir og styrkir fyrir doktorsnema (frá 2016) voru metnir.

Greiningin einskorðaðist við styrki sem veittir voru 2011-2015 en gögn fyrir 2011 voru ekki aðgengileg.
Gögn sem lágu til grundvallar greiningum komu frá Rannsóknamiðstöð Íslands (RANNÍS), einstökum háskólum, matskerfi opinberu háskólanna, Hagstofunni og matskerfi Háskólans í Reykjavík.

Að auki var send út könnun og viðtöl tekin.

2.

Helstu niðurstöður Fjármögnun: Umsóknum hefur farið fjölgandi, árangurshlutfall er að meðaltali rúmlega 20% og mikil sókn er í styrki.

Engan mun er að finna á árangurshlutfalli milli kynja en hlutfall kvenna í umsóknum er örlítið lægra en karla.
Engan mun er heldur að finna á milli sviða (náttúruvísindi annars vegar og félags- og hugvísindi hins vegar).

Umsækjendur voru almennt mjög ánægðir með hvernig RANNÍS stendur að umsókna- og úthlutunarferlinu en samskipti um hluta ferlisins þarf að bæta.

Áhrif á frekari rannsóknir: Þeir sem ekki fá styrki eru mjög líklegir til að sækja um aftur og Rannsóknasjóður þykir ómissandi á meðal vísindamanna.
Niðurstöður viðtala og spurningakanana leiddu í ljós að styrkveitingar úr Rannsóknasjóði eru mikivægt fyrsta skref til að hefja rannsóknir, styrkja nýliðun í vísindum og til að auka möguleika á frekari erlendum styrkjum í kjölfarið.

Áhrif á vísindi: Ekki var mögulegt að tengja vísindagreinar og aðrar afurðir rannsókna beint við einstaka styrki þar sem of fáir vitna í styrki í sínum greinaskrifum.
Birting greina er nokkuð jöfn frá ári til árs, og eins og búast mátti við aukast tilvitnanir í verk styrkþega ár frá ári.

Áhrif á menntun: Doktors- og meistaranemar njóta góðs af styrkjunum, bæði beint og óbeint.

Þjálfun nýrra kynslóða vísindamanna er mjög mikilvæg fyrir styrkþega.

Áhrif á menningu, samfélag og efnahag: Ekki var hægt að framkvæma greiningu á nákvæmum áhrifum á samfélag, menningu, efnahag og stefnumörkun, því ekki voru fyrir hendi gögn sem gerðu það kleift.
Könnunin sýndi þó ekki verður um villst að vísindamennirnir telja sig hafa áhrif á með þátttöku í opinberum nefndum og þátttöku í umræðu í fjölmiðlum.

Hátæknifyrirtæki meta mikils að geta ráðið einstaklinga sem hafa fengið þjálfun í rannsóknum og líta á Rannsóknasjóð sem afar mikilvægan hlekk í þeirri keðju.

3.

Mikilvægur fyrirvari I. Gögnin sem matið byggði á skorti samræmi og skýrleika.
Ósamræmi var oftar en ekki á milli gagnasafna og ekki var mögulegt að framkvæma ýmsar greiningar vegna lélegra gagna eða algjörrar vöntunar á gögnum.

Niðurstöðurnar sem hér koma fram eru réttmætar og áreiðanlegar, en þær eru grófkenndar.

Gagnabanki um íslenskar rannsókir (IRIS), sem nú er í þróun, mun bjóða upp á getu til að framkvæma betri slíkar greiningar í framtíðinni.

II.
II.

Greining á bæði akademískum og samfélagslegum áhrifum væri stórlega bætt með betri gögnum, skilgreiningu gagna, söfnun gagna og vörslu gagna.

Slík greining yrði afar mikilvæg fyrir vísindamenn, háskólana, RANNÍS og stjórnvöld til að fylgjast vel með árangri sínum og áhrifum Rannsóknasjóðs.

5 Impact Evaluation of the Icelandic Research Fund May 2022 Executive Summary 1.
Background Terms of Reference: to provide a quantitative analysis of the Icelandic Research Fund, its visibility and impact, covering funding patterns, springboard to other activities and funding, scientific outputs, educational engagement & social, cultural and economic impact.

Project Grants, Grants of Excellence, Postdoctoral Awards were examined 2011-2015 (so as to capture outcomes) and Doctoral Student Awards from inception in 2016.
Data sources: the Icelandic Centre for Research (RANNIS), the universities, University of Iceland’s coordinators of the Evaluation System of Icelandic Public Universities, Statistics Iceland (Hagstofan) and the Reykjavik University evaluation system.

These were supplemented by qualitative survey.

2.

Principal Findings Funding patterns: Applications are robust with a success rate across grant types of just over 20%.

Demand is high and applications have increased over time.
There is no evidence of gender bias, though women make slightly fewer applications, nor evidence of differentiation between disciplines (Natural Sciences / Social Sciences & Humanities).

There were positive views of how RANNIS organises programmes, with caveats about some communications.

Springboard to other activities and funding: Rejected applicants persist with applications: the IRF is clearly valued by researchers.
Survey and interview evidence suggests that the IRF is a key first step in grant-getting, important for early-career scientists and a step towards larger international grant awards.

Scientific outputs: It was not possible to relate specific research outputs to specific grants: acknowledgment of grant awards in publications is not universal.

Publication of journal articles is consistent across years and citations rise year-on-year, as expected.
Educational engagement: Doctoral and Masters students benefit directly and indirectly from IRF awards.

Training next generation scientists is important for grant holders.

Social, cultural and economic impact: Public engagement and cultural, economic and policy impact with demonstrable change is not captured, evidenced, evaluated or curated systematically.

Survey data show engagement of award holders outside academia, from media appearances to company formation.
Companies value the pipeline of trained scientists; the role of the IRF is regarded as important for this.

3.

Important Notes I.

Existing data lack clarity and coherence.

Data was often inconsistent and some analyses were not possible because of poor quality or no data.

The conclusions are valid but lack granularity: more could be done with a clear and coherent data set using the developing Icelandic Research Information System (IRIS).

II.
II.

Analysis of academic and societal impact would be improved by more robust data definitions and collection.

Analysis is important for individual researchers to be able to mark their achievements, as well as for universities, funding bodies and government.
6 Impact Evaluation of the Icelandic Research Fund May 2022 Introduction Icelandic Research Fund In the fall of 2020 the Ministry of Education, Science & Culture (the Ministry) announced the intent to evaluate the impact of the Icelandic Research Fund (IRF).

The Quality Board for Icelandic Higher Education was commissioned to conduct the review according to terms of reference accepted in October 2020.

The terms of reference specified the following areas of evaluation: 1.

Funding Patterns 2.
Funding Patterns 2.

Springboard to Other Activities and Funding 3.

Scientific Outputs 4.

Educational Engagement 5.

Social, Cultural and Economic Impact Katrin Frímannsdóttir, PhD, was hired to oversee the evaluation.

Fróði G. Jónsson, M.S., was hired as a project assistant.

The project was expected to last one year with the final report presented in March 2022.
Description of the IRF “The Icelandic Research Fund (IRF) is an open competitive fund which provides research grants according to the general priorities of the Icelandic Science and Technology Policy Council and based on peer review of proposals.

The role of the IRF is to enhance scientific research (basic and applied) and research education in Iceland.
For this purpose, the IRF awards funding to research projects led by individuals, research teams, universities, research institutes, and companies and to research students (cf.

Act 3/2003 with later amendments)1.” The Icelandic Centre for Research (RANNIS) is the government agency that manages domestic and international competitive funding for research, innovation, education, and culture.

This includes the IRF.


In 2020, the annual grant budget for all Icelandic funds at RANNIS was just over ISK 15.800 M, of which the annual budget for the IRF was ISK 2.800 M (18%); just over ISK 1.000 M was granted for new applications2.

The main goals of the IRF are to enhance basic and applied research, as well as science education.

The logic model for IRF is shown in Figure 1.

It covers input, activities, output, outcome and impact.
Input: The Ministry provides funds for the IRF and the Board of Directors, which is appointed by the Minister, issues rules and guidelines for IRF.

RANNIS is the operational arm that manages the fund and reports to the Ministry.

Other input groups are grant applicants, which includes all individuals included on an application, and grant recipients´ institutions.

1 https://en.rannis.is/funding/research/icelandic-research-fund/ 2 RANNIS.

(2021).

Ársskýrsla RANNÍS 2020.

Reykjavík: RANNIS.


7 Impact Evaluation of the Icelandic Research Fund May 2022 Activities: Members of IRF´s Board of Directors make the final funding decision based on the evaluations made by expert panel members, who are both international and Icelandic.

RANNIS manages the IRF, including writing and publishing handbooks, announcements, the application cycle, communication, data management and reporting.
Grant recipients manage the science research process by organising the research, data collection, data management, data analysis, reporting, and communication of results.

Recipients´ institutions manage the financial part of the grant, science education, and training scientists.

Output: RANNIS takes care of annual distribution of funds, communication with recipients, and communication and publication of guidelines and information for the IRF.
The PIs oversee communication of research outcomes whether they be articles, presentations, posters, podcast, interviews, books, book chapters, teaching or other media.

Training of new scientists is also part of PI output.

The PI´s institutions are in charge of science education as well as admitting students and trainees.

Outcome: The short-term outcomes are, for example, presentations, articles submitted, job offers, and graduation of grant participants.
Long-term outcomes include published articles and other scientific communication channels as well as citations, individual indices, innovation, the formation of science- based companies, and hiring of graduates for jobs.

Impact: The intended impact of the IRF is to “Enhance basic and applied scientific research and enhance and support research education in Iceland.” Basic research is meant to be independent of whether it has the potential of being useful beyond the extension of knowledge and scholarship.
However, it is clear that there is no line that separates basic and applied research into two unconnected categories.

The thread from basic to applied is more of a continuum: applied research is undertaken with an intended outcome of practical value beyond academia while basic research is not.

However, basic research can lead to significant practical benefit with demonstrable social, cultural or economic impact on society.
The research projects that IRF supports span this spectrum but RANNIS does not categorise applications as basic or applied.

We therefore did not evaluate the distribution of grants as basic or applied research but we did attempt to understand what societal impact IRF supported research had.

8 Impact Evaluation of the Icelandic Research Fund May 2022 Fig.
1: IRF Logic Model 9 Impact Evaluation of the Icelandic Research Fund May 2022 Methodology Timeframe Because this review is examining the outcomes of grant awards, we defined the closest time we could analyse.

Grant awards in 2015 would complete in 2018 or 2019.

It was estimated that 2-3 years post grant would be the least amount of time we could expect scientists to have academic or social impact.
At the other end of the spectrum, very little usable data to assess impact is available from RANNIS prior to 2011.

Therefore, we evaluated recipients of IRF grants, 2011-2015.

Data Sourcing and Analysis This evaluation used quantitative and qualitative data, both existing data and new.

Qualitative data was collected by interviews and from open-ended comments in a survey.

New quantitative data was collected via survey.


Existing data was retrieved from RANNIS, individual universities, University of Iceland´s coordinators of the Evaluation System of Icelandic public Universities, Statistics Iceland (Hagstofan) and Reykjavík University´s evaluation system.

Because the Icelandic Research Information System (IRIS) is still in development we had to rely on data from several separate and disconnected sources.
Analysis was driven by the data available which was often disorganised and incomplete.

We therefore created datasets by linking data from different sources.

This is not ideal because different sources might have used different methods to collect data.

At times, the same information could be found across different sources, but with different values.

We had no way of knowing which of the sources was right and could therefore not trust the data entirely.
There is patently a need for valid and reliable data in this area.

The current development of IRIS can address this.

We performed descriptive statistical analyses, inferential statistical analyses, and causal analyses on the quantitative data that we had access to, consistent with other studies done on impact of research.3 Data Limitations Data was not available that would permit analysis of other grants received.

The same data would be needed to calculate grant application success rate outside the IRF.
We asked questions about extra-IRF grants in the survey but this is self-reported and less useful for detailed analysis.

We intended to perform a social network analysis to show domestic and international networking of scientists, as well as the connection of sciences among IRF recipients and their collaborators.

To do that, 3 Anderson, P. S., Odom, A. R., Gray, H. M., Christensen, W. F., Hollingshead, T., & Seeley, M. K. (2020).
A case study exploring associations between popular media attention of scientific research and scientific citations.

PLoS, 15(7), e0234912.

Arnold, E. (2012).

Understanding long-term impacts of R&D funding: The EU framework programme.

Research Evaluation, 21(5), 332-343.

Bol, T., de Vaan, M., & van de Rijt, A.

(2018).

The Matthew effect in science funding.

Proceedings of the National Academy of Sciences of the United States of America, 115(19), 4887-4890.

Heyard, R., & Hottentrott, H. (2021).


The value of research funding for knowledge creation and dissemination: A study of SNSF Research grants.

Humanities & Social Sciences Communications, 8(1), 1-16.

Fortunato, S., Bergstrom, C. T., Börner, K., Evans, J.

A., Helbing, D., Milojevic, S., .

.

.

Barabasi, A.-L. (2018).

Science of science.

Science, 359(6379), eaao0185.


10 Impact Evaluation of the Icelandic Research Fund May 2022 we would have needed standardised data set of publications with a unique author identifier such as ORCID.

The data needed was not available because too few scientists in Iceland have registered in ORCID.

The survey included a question regarding research groups and international involvement, but the information could not be used for social network analysis.
We intended to perform employment analysis on students and trainees participating in research supported by the IRF and contacted some universities for the information but without sufficient success.

We contacted Statistics Iceland for the data but also without success.

However, they received a request from the Ministry of Finance for a similar analysis and were in the very beginning stages of that process.
Finally, it was not possible to gather which subsequent publications of grant recipients were made possible by an earlier IRF grant.

This was due to the fact that IRF grant recipients do not consistently list funding sources in their journal articles, book chapters, etc.

It was also noted that grant recipients did not consistently report publications derived from a given grant in their progress reports and final reports to RANNIS during this period.
Survey The survey was created by Dr. Frímannsdóttir using SoGo Survey software® and tested for validity and reliability by 8 volunteers.

The survey had 4 main focus areas: experience with RANNIS; impact and experience of receiving a grant; education and training of students; and view of the IRF.

The total number of Principal Investigators (PI) in receipt of an IRF grant was 283 during the evaluation period, of which 238 were unique individuals.
We successfully sent out the survey to 215 individuals; 142 participated (66%).

A descriptive statistical analysis, inferential statistical analysis, and causal analysis was performed on the quantitative data and thematic analysis on open-ended comments.

We had intended to send out a survey to non-recipients of grant, but as there was almost 70% overlap between the groups of recipients and non-recipients we did not.
Interviews A random sample of PI recipients was invited to participate in one-on-one interviews with Dr. Frímannsdóttir.

Invitations were sent to 35 individuals; 17 agreed to be interviewed.

The interviews were conducted both in-person and via video conferencing.

During the first half of the interviews, it became clear that input from companies who hire graduates from Icelandic universities and seek out graduates who have been trained in doing research was needed.
Eight companies within engineering, biotechnology, and pharmacology were contacted and 7 agreed to be interviewed.

All of them were conducted via video conferencing.

A total of 24 interviews were completed.

The interviews had 4 main focus areas: description of the research process, impact of receiving a grant on them as a PI, societal impact of their research, and improvement of the IRF.

A thematic analysis was performed on all interviews based on questions asked and the responses we received.
11 Impact Evaluation of the Icelandic Research Fund May 2022 Results 1.

Funding Patterns of IRF 1.1.

Application Process A call for application is published annually in May with a deadline around June 15.

Applications are reviewed by RANNIS staff for accuracy and guidelines; those that do not meet set standards are dismissed.

The remaining applications are categorised and divided among the expert panels.

The role of RANNIS´ staff is administrative and procedural.
The role of the expert panel and the board is to be evaluative and make a judgment.

“Expert Panel members are appointed by the Science Committee of the Icelandic Science and Technology Policy Council.

Up to seven individuals with qualifications at associate professor level or higher, who have extensive experience of research, are appointed to each Expert Panel.
At least two members of each Expert Panel shall be predominantly active professionally outside of Iceland.”4 The chair of each of the expert panels is in charge of the reviewing process and is appointed by the Science Committee of the Icelandic Science & Technology Policy Council.

The “first reader” of the expert panel is responsible for locating the external evaluators, who are almost exclusively international.

Each application is reviewed by the 3 members of the expert panel and 2-3 external evaluators.
Following review by the external evaluators, members of the expert panel assess the applications, using the external evaluations for reference.

After this, a grade is assigned to the application.

Once the grades have been determined the results are presented to the Board and they ultimately decide who receives funding.
In addition to the scientific evaluation that the expert panel completes, members of the Board must consider the Icelandic Science & Technology Policy Council research emphasis for that year, which is approved by the Science Committee.

Final decisions of funding are announced in mid-January each year.

Description of the evaluation process is published in Icelandic on the RANNIS website where all steps are explained.

A timeline is provided as well.


RANNIS publishes the annual IRF handbook on their website where all pertinent guidelines for a successful application are defined.

The handbook is updated annually.

5 Additionally, RANNIS publishes on their website statistical analysis of applications and funding along with the fund’s official allocation policy.

6 Some changes have been made through the years on the types of grants that are supported.
Grants of excellence, project grants, and postdoctoral grants have been available since 2011.

In 2013 a grant (START) was added for Icelandic postdoctoral students who graduated abroad: the intent was to encourage them to return to Iceland.

This grant was co-founded by FP7 Marie Curie Actions and IRF but 4 RANNIS.

(2022, 1 14).

RANNIS.

Retrieved from https://en.rannis.is/funding/research/the-icelandic-reasearch-fund/expert- panels/ 5 RANNIS.

(2022, 1 26).

RANNIS.


Retrieved from The Icelandic Research FUnd Handbook: https://www.rannis.is/media/rannsoknasjodur/IRF_Handbook.pdf 6 RANNIS.

(2022, 1 26).

RANNIS.

Retrieved from IRF Statistics and preious AWARDS: https://en.rannis.is/funding/research/the- icelandic-reasearch-fund/statistics/ 12 Impact Evaluation of the Icelandic Research Fund May 2022 was only offered once for new applications.

In 2016 a grant dedicated to doctoral students was added to the portfolio.

Villa!
Villa!

Uppruni tilvísunar finnst ekki.

presents results of participants´ responses to questions regarding RANNIS´ communication pattern.

Almost all (95%) of survey respondents say that RANNIS communicates information about grant deadlines either very well or well, and over 90% say that RANNIS communicates application guidelines (length and format) very well or well.
Communication about the timeline of the review process received the least satisfaction: 45% found RANNIS to communicate the timeline either very well or well while 25% found that to either be done poorly or very poorly.

Communication of RANNIS´ distribution of funds, process of annual reports and process of final report is well received: more than 65% of respondents say that is either done very well or well, while less than 10% say that is done poorly or very poorly.
Communication of timeline of review process and IRF’s policy is on the other hand less satisfactory: 45% say that communication of timeline is done very well/well while 26% say that it is done poorly/very poorly.

IRF policy is communicated either very well/well in the opinion of 58% of respondents but 14% say that it is either done poorly or very poorly.
RANNIS communication Application deadline Length of application Format of application Content that must be included Content that should not be included Timeline of review process Distribution of funds Process of annual progress reports Process of final report Policy of the fund Very well/well Neither well nor poorly Poorly/very poorly Don’t know Table 1: Survey response on RANNIS’ communication.
We asked survey respondents how much time they, as PIs, spent on the application and how much the research team spent on the application: 27% of respondents report spending more than 160 hrs on the application while 59% spent 80-160 hrs and 14% spent less than 80 hrs on the application.
It is clear from the replies that the PIs carry the biggest load of the application process: 40% of respondents say that the team spends less than 80 hrs on the application, 55% say that the team spends 80-160 hrs, and 15% say the team spends more than 160 hrs on the application.

According to these results, applying for a grant is time-consuming and requires a considerable amount of commitment, dedication, and organisation of the team.
When asked how they learned how to apply for grants, the more experienced PIs all said that they learned by doing and following guidelines (Villa!

Uppruni tilvísunar finnst ekki.).

Some of the younger 13 Impact Evaluation of the Icelandic Research Fund May 2022 PIs said that scientific writing had been part of their graduate studies.
Some PIs said that they trained their students in grant writing either by making sure they took courses offered by their university in scientific writing or grant writing specifically, while others had it as part of courses they taught.

Some respondents said that they had their students and trainees participate in all grant writing processes.

All agreed that knowing how to write a grant application was a crucial skill in the toolkit of a scientist.
Interviewee Learning how to write a grant 1 “I just learned by trying because my advisors encouraged me to try to apply” 2 3 6 8 “We begin at the BSc stage to practice research by writing a research protocol that will then be used to apply for a grant” “It was required in my graduate studies to take a course in scientific writing and I at least learned how to interpret a call for applications and it really helped me to gather the courage to apply, in real, later on.” “I encourage my students to take courses in scientific writing, but it’s not required.” “In the beginning, years ago, I just went into this process without knowing anything about how to actually apply and I learned by doing, and I’m actually still learning after all these years.” “I don’t require my students to take courses on scientific writing but I help them when they apply for a grant and they participate in my applications.” “It was a requirement in my graduate work to take a course in scientific writing [...] in hindsight it was very helpful and I learned a great deal, especially to help with first steps.” “I just learned by doing, I did not get any training in grant writing during my doctoral studies or during my postdoc.
I was lucky as a postdoc, my salary was paid by my bosses and I did not have to apply for a grant to do that so when I came back home to Iceland I had very little experience in grant writing.” “Grant writing has become a more formal part of my students’ training, almost all of them take a course or two in scientific writing.
Those who are more advanced in their training do participate in the team’s grant writing processes and get training that way.” 10 “I’m of the generation that just learned by doing, by making mistakes, but we are in a very different environment today than before.
I’m always applying with a group and we manage the application by dividing the load and I learn from the team a lot” “In later years we have great support from an office within the university and I write all grant applications with assistance from them and the students both take courses from them and get training from them as well.” Table 2: Sample responses to the question “How did you learn how to write a grant?” Conclusion There is a high degree of satisfaction with RANNIS and their management of the IRF and only few note areas of possible improvement in their work.
Applying for a grant is time-consuming and demands great 14 Impact Evaluation of the Icelandic Research Fund May 2022 dedication from the PIs as well as their research team.

Interestingly, few recipients have formally learned how to write applications even though all participants agreed that knowing how to write grants is an essential skill for scientists.

Many are requiring that their students and trainees take formal courses in grant writing and participate in a grant writing process.

1.2.
1.2.

Success Rate by all IRF Grants Figure 2 shows the number of applications 2011–2022 for all types of IRF grants.

In 2016 a grant for doctoral students was added to the pool and, even though this is outside of the evaluation period, it is included for better oversight of trends.

Applications for a project grant are by far highest, with annual applications varying from 173–214.
The fall-offs in 2021 and 2022 are perhaps unexpected but might be a product of the COVID-19 pandemic and consequent restrictions on research activity.

As noted above, preparation of grant applications is intensive, and opportunities to collect pilot data for applications will have been restricted.

Applications for Postdoctoral grants quickly increased after their introduction in 2011 and typically number over 50 each year.
Doctoral Student grant applications have more than doubled in number since their introduction in 2016.

The Grant of Excellence has the lowest application numbers but still shows annual growth.

Overall the application rates for all grants are maintained year-on-year.

Fig.

2: Number of applications by grant type (2011-2022).
Number of applications All IRF grants 250 200 150 100 50 0 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 Project grant Grant of excellence Post Doc Doctoral students Figure 3 shows application success rate across all 4 types of grants.

Success rates vary across years but the trend of success rate among grant types follows similar lines: success rates appear to rise and fall more-or-less in synchrony.
Project grant applications are by far the largest category (Figure 3) and have a reasonably stable outcome – an average success rate over years of 22% with low variance.

This is comparable internationally to other research grant funding agencies, such as the National Institutes of Health in the USA, the European Research Council and UKRI.

The other grant awards have much lower numbers of applications and the data are a little noisier, but clearly follow the same trends as the Project Grant awards.
15 Impact Evaluation of the Icelandic Research Fund May 2022 Fig.

3: Application success rate by grant type (2011-2022).
Success rate All IRF grants 35% 30% 25% 20% 15% 10% 5% 0% 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 Project grant Grant of excellence Post doc Doctoral students Conclusion This analysis shows that number of applications has increased steadily through the years and success rate has been fluctuating between 20 and 25%, which for project grants is on par with other well-known grant mechanisms internationally.
The Grant of Excellence has the highest degree of fluctuations for success rate, but with very low numbers of applications.

1.3.

Success Rate by Gender The success rate by gender (Figure 4) shows that men were very slightly more successful between 2011- 2014 (by only 2%) but since 2017 the success rate between the two is very similar.

The fluctuations are relatively small, except in 2015 when women had 40% success rate and men 26%.
That year, the difference between the number of applications between male and female PIs was the largest (F = 83, M = 168) during the 11 years examined (Figure 5), with applications by women being the lowest for any of the years examined.

Additionally, Figure 5 shows that the number of applications where a female was the PI are about one third of all applications, but that difference has been decreasing through the years.

In 2021 46% of all PIs were women.


In 2022 the difference was 10%, as the number of male PIs increased more than the number of women PIs.

There is no statistically significant difference between the success rate between women and men (p=.95).

16 Impact Evaluation of the Icelandic Research Fund May 2022 Fig.

4: Success rate by gender (2011-2022).


Success rate by gender 45% 40% 35% 30% 25% 20% 15% 10% 5% 0% 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 Success rate Female Success rate Male Fig.

5: Number of applications by gender (2011-2022).


Number of applications by gender 400 350 300 250 200 150 100 50 0 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 PI Female applications PI Male applications Conclusion The numbers of applications with a female PI have always been fewer than for a male PI.

Further interrogation of this would require data on the ratio of men and women in each discipline.
However, while the success rate by gender was a little higher for males until 2014, since 2016 there is no apparent difference in success rate by gender.

1.4.

Success Rate by Discipline When analysing success rate among disciplines, we grouped applications into 2 categories: Natural Sciences and Social Sciences & Humanities.

To differentiate between these groups, we used RANNIS´ categories for expert panels.
However, these categories have changed slightly through the years: for example, Public Health was part of social sciences until 2016, when it moved to Natural Sciences.

17 Impact Evaluation of the Icelandic Research Fund May 2022 Additionally, the number of expert panels increased from 4 to 7.

The other major change was in 2017 when all ranking of applications was organized by the score of each application, regardless of the expert panel it was evaluated by.
This means that there are no predetermined quotas for each group of sciences.

We grouped the expert panels for Natural Sciences as Science, Mathematics, Engineering & Technical Sciences, Nature & Environmental Sciences, Biological Sciences, Clinical Research and Public Health.

Social Sciences & Humanities includes Social Sciences, Educational Sciences, Humanities and Art.

Success rates between the 2 groups for the number of applications are presented in Figure 6.
Until 2016 there was only a slight difference in success rate between the 2 groups but since 2017 the average success rate diverges a little more, though with broadly the same annual trends.

Success rates are consistently slightly higher for Natural Sciences.

Fig.

6: Success rate, natural sciences v. social sciences and humanities (2011-2022).
Success rate of # applications Natural sciences vs. social sciences and humanities 35% 30% 25% 20% 15% 10% 5% 0% 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 Natural sciences Social sciences and humanities In late 2021 the Ministry created a new fund dedicated to research in education.

The impact of that fund on IRF application numbers and success rate remains to be seen.
Some interviewees from the Social Sciences & Humanities reported that their colleagues rather frequently said it was not worth their effort to apply for IRF because they never received an award.

Nevertheless, all the interviewees themselves said that because the success rate for IRF is around 20% it was not realistic to expect funding by applying once or twice, so they kept applying until funding was received.
Conclusion The success rate analysis between Natural Sciences and Social Sciences & Humanities shows that there are annual fluctuations between groups, with the Natural Sciences tending to be more successful.

There are signs that there is a difference between success rate of the disciplines, but the difference is insignificant.

18 Impact Evaluation of the Icelandic Research Fund May 2022 2.
Springboard to Other Activities and Funding The intent was to evaluate subsequent grants received in the years following award of an IRF grant, but without valid and reliable data we were unable to perform that evaluation except for other IRF grants.

No data was available on international grants.

We used the survey and interviews to gather information on international grants, but this self-reported data is not suited for statistical analysis.
In addition to grant activities, we asked questions, both in the survey and during the interviews, regarding other activities than grants and publications, such as how the research process unfolds, how a scientific idea comes to life in a grant application, and what happens when a grant application is not funded.

2.1.

Springboard to Other Funding Announcement of grant recipients is published annually by RANNIS in mid-January.
According to interviewees it varies greatly what happens once a grant is approved.

Depending on the research, for example, whether the team is set to go or whether they must hire new team members.

With a team in place the research process can possibly begin right away.

However, if hiring is needed, there were reports that the hiring process was often long and could take several months to complete (Villa!

Uppruni tilvísunar finnst ekki.).


Once the research team is ready to act on the research plan an extensive team management by PI or other dedicated team members ensues.

The management of a research team depends on its size, which varies greatly, from fewer than 3 to more than 10.
The type of research impacts the need for management as some research is conducted only on computer or in a small lab, whereas other projects require fieldwork in (for example) the natural environment, classrooms, hospitals, or other open or public spaces.

Interviewee What happens once a study is funded?


4 6 7 17 “We almost always have an international group of students and trainees and it can take a long time to get going, and sometimes it feels like a lottery how long they stay, whether they can live with Icelandic winter and the isolation the country in reality has.” “When we have hired international students or trainees it’s a lot of work getting them work permits, if they are outside Schengen, and just getting them here.
It can take months” “First step is to find doctoral students to participate in the study, the study will not happen without them.

They can dedicate their time to the research process.” “It varies greatly how long it takes to get going, it depends on the research idea.

Some require quite a few students and a lot of fieldwork while in others we are ready to go with most fieldwork lined up.” “It doesn't take me that long to get going as I’m almost always already under way, at least with preparation work.
Because the application process is so long and demanding I’m almost always well under way just by preparing the application.

The biggest hurdle to get going is the hiring process of team members.” Table 3: Sample responses to the question “What happens once a study is funded?” 19 Impact Evaluation of the Icelandic Research Fund May 2022 We analyzed how long it would take individual researchers to receive a grant after getting a grant rejection.
We had data on every grant proposal during the years 2011-2015 and if it was accepted or rejected, so our sample included everyone who applied for an IRF grant in this period.

Most grant proposals involve a team of investigators, but typically there is one person who heads the proposal as the PI.

In this analysis, we looked at the individual PI, not the team, and if this individual was successful or not in their application for grants.
In this period, 638 individual PIs applied for at least one IRF grant.

Of those individuals, 412 never received a grant while 226 PIs received one or more.

Eighty-eight individuals of the 226 always had their grant applications accepted (most often, they only had a single application during these years), 20 had their first application accepted and then only had rejected proposals after that, and the remaining 118 individuals received a grant after having earlier in the period been rejected at least once.
Those 118 individuals are the focus of this analysis.

As can be seen in Figure 7, 14 individuals received grants the same year they got rejected, having applied for multiple grants that year (which was not uncommon).
Fifty-two individuals received grants the year after having been rejected (possibly by modifying their proposals in line with comments received with the earlier rejection); 28 individuals received grants 2 years after having been rejected; and 12 individuals received grants 3 and 4 years after having been rejected.

The limited timeframe constrains this analysis, but it is nevertheless clear that applicants rejected for funding are not discouraged from further attempts, which are often successful.

Fig.
Fig.

7: Number of years it took PIs to receive a grant, after having their first application rejected.

Because many apply for multiple grants from the IRF each year, we looked at how many applications were needed to receive a grant.

Most applicants succeeded in their first 5 attempts (Figure 8), which is in line with the approximate 20% annual success rate of the IRF.

20 Impact Evaluation of the Icelandic Research Fund May 2022 Fig.8: Number of applications to success.
We also looked at whether applicants whose IRF grant application was rejected then applied for a different kind of IRF grant.

As detailed earlier, the IRF had 3 grant types in the 2011-2015 period: Grant of Excellence, Project Grant, and Postdoctoral Fellowship Grant.

In this period, 286 PIs applied more than once for a grant.

Out of them, 207 (72.38%) only applied for one type of grant and 79 (27.62%) for multiple types.
The success rate of the IRF through the years is around 20% and the PIs we interviewed use that as a guide to evaluate their own success and our data analysis supports that viewpoint.

2.2.

Effect on Other Activities Receiving an IRF grant increases scientists’ visibility both within their scientific community and externally.
Participants also pointed out, both in the survey and interviews, that their reputation as scientists was impacted: receiving the grant was perceived as an honour and approval of their scientific work.

When asked in the survey how much positive impact the grant has had on their academic career, 89% say that it had either a lot or quite a bit of an impact; 83% say that it had either a lot or quite a bit of impact on their participation in international research.
The grant also had an impact on their ability to participate in either administration of Icelandic societies/associations and administration in foreign societies/associations.

When asked whether the grant had a negative impact on their career 99% said “no”, but the one who answered “yes” said that it was maybe not a negative impact but that it solidified their perception that academia was not for them (Villa!

Uppruni tilvísunar finnst ekki.).
21 Impact Evaluation of the Icelandic Research Fund May 2022 Interviewee Impact on scientists 7 11 14 17 “Now that I have been a scientist for quite a while it has become more and more important to me to be able to train a new generation of scientists, in addition to keeping me young and sharp, but these young people have so much to choose from in their lives and we have to be able to get them excited about science.” “The opportunity that the grant has given me is to publish a lot of my own research which would not happen without a grant, and that has opened up opportunities abroad for both international research and networking in general [...] That networking is incredibly important in my field.” “Of course, this has had an impact on me as a scientist, I became a scientist later in life and my first IRF grant was a big deal, and of course it changes the push to academic advancement as the grants really count.
People start paying more attention to your work both here in Iceland and abroad.” “I have been in this for a very long time and my first grant had a major impact on my ability to do research but now that I have grants from NIH, DoD, and Europe the IRF has much smaller impact.

it’s just not that much money in comparison to the others, but for a young scientist an IRF grant changes everything.” Table 4: Sample responses to the question “What is IRF’s impact on scientists?” 2.3.
Consequences of Non-Funding All interviewees had experienced a grant application being denied.

During the interviews we discussed what happens to a scientific idea when it is not funded (Villa!

Uppruni tilvísunar finnst ekki.).

Some said that the idea simply “dies” as they need funds to do their research – most research cannot be done without increasing the number of staff, which is very often students, and that calls for increased funding.
Some said that when they have been denied a grant they try to apply for smaller funds, if available, often within their university or academic unit.

Others said that they responded by decreasing the scope of their study as well as by increasing the timeline to test their idea.

No-one who participated in interviews let one denial stop their research: they kept on applying, either to IRF or other grant funds if available, until they received funding.

Interviewee Consequences of non-funding 1 3 5 “It dies!
I can’t do a study without funding, it’s as simple as that.” “It’s clear that an idea dies if it’s not funded.
I keep on trying to apply a few times with the same idea, but it has happened that an idea does not get funded, even with a few tries and then it just dies.” “When we don’t receive a grant for a study, we try to apply for a university grant, or some other agencies, but they are always smaller and then we just decrease the size, but once we sent a slightly changed IRF application to a European fund and received that one!” 11 “Research does not happen if I don’t get help and help is in the form of funds as that allows me to hire students.” Table 5: Sample responses to the question “What are the consequences of non-funding?” 22 Impact Evaluation of the Icelandic Research Fund May 2022 Everyone agreed that research in their field cannot exist without grant funding.
Sixty five percent of survey respondents have applied for at least 1 international grant; 35% have applied for more than 4 grants; and 72% of those who have applied for these grants have been successful and received 1 or more.

Additionally, almost 90% of survey respondents have been members of an international research team, 59% as either a PI or a Co-PI.
During interviews we discussed the notion of “in the absence of” where we explored the idea of the grant environment not existing or being decreased considerably.

As expected, scientists agreed that research would be decreased or eliminated, but that the biggest impact would be on their ability to train other scientists.
As one scientist said “we would not be able to keep the wheel spinning; there would be very little research, very little training of scientists, and few candidates for teaching.” Others said that in the absence of IRF they would have to apply for international grants to a much greater extent – a different environment with different requirements and more competition.
This discussion created an even stronger reaction from the representatives from the companies we interviewed or as one interviewee said: “Research in Iceland is completely dependent on IRF and basic research in Iceland would almost not exist without it.” Another one said that “we would have to hire foreigners much more than we do today and that is very expensive for us.” All the representatives from companies did state that they do not apply for IRF grants but they gain indirectly from hiring students and graduates who are trained on the grants.
Participants were not averse to changing the fund system.

However, they were not keen on “rocking the boat” too much or too hard and said that all the changes RANNIS has done to the IRF through the years have been for the better; the management is much better and the standards for applications are higher; and transparency of the reviewing process has increased.

The general perception is that grants that are funded are of a high quality.

2.4.
2.4.

Development of a Scientific Idea All interviewees agreed that an application begins with a scientific research idea, which is born out of work that is under way.

However, the degree to which the new idea differs from their ongoing research varies greatly.

Quite often it is a completely new direction with new research group participants, even completely unexpected and cross-sectional.

At other times it is a direct continuation of ongoing work.
Nonetheless the application is always centred around a scientific idea that drives the research process.

Not a single participant mentioned that the drive behind a grant application was money, status, or reputation, but all interviewees were realistic and said that receiving a grant makes their research happen.

Interesting discussions did occur when analyzing how a scientific idea was born.

Most often a scientific idea is born out of ongoing research and systematic exploration is a key component of that.
Most often, ideas are generated via white board analysis and research meetings.

In addition, attending conferences, reading articles, and engaging in water cooler, lunch and coffee discussions all have major significance in that process (Villa!

Uppruni tilvísunar finnst ekki.).

One interviewee talked about the decrease in creativity when their department moved and the new place did not provide small social spaces for incidental meetings such as coffee and lunch breaks.
23 Impact Evaluation of the Icelandic Research Fund May 2022 Interviewee Emergence of a research idea 1 3 8 12 “It all depends [how a research idea is developed], sometimes I have many ideas going on at the same time and the research team sends it back and forth, but that process is always so rewarding.

“I work a lot alone and ideas develop as I’m working on something or another.
Once I had a student who was very creative and helped me a lot during the idea creation.” “We try to be organized and methodical in our preparation phase.

We start quite early to think about the next application, and it depends on the knowledge gaps that we have what we decide to focus on.

We begin with a white board meeting and map out knowledge gaps and what we should focus on; what is new, why is this exciting, until we have a good grasp of what needs to be done.
That’s when we begin writing down specific aims and, in that process, we solidify the research question.” “The development of a research idea is the most important part of this process; what is it that I want to study, what is it that I need to know and what questions do I, and the field, need answers for.
I also need to make sure that I take my research in a new direction, but it can’t be too far away because then I lose continuity and maybe go way beyond my knowledge base.” Table 6: Sample responses to the question “How does a scientific idea emerge?” Conclusion Scientists are aware that they would not be able to work as scientists without external funding that supports their research infrastructure and students’ employment.
They are persistent in their approach to grant applications and those who are successful keep applying until they are rewarded.

Receiving IRF funding is perceived as having impact on scientists’ reputation which often translates into invitations to participate in domestic and international research groups, international scientific associations as well as invitation to take part in governmental committees.
Recipients of IRF funding are very often members of international research teams and frequently take on roles of PI and Co-PI in those teams.

In the view of interviewees, consequences of absence of, or reduction of, IRF would be noticed from science education to high tech companies’ ability to operate in Iceland.

3.
3.

Scientific Outputs It is in principle challenging to evaluate the impact a grant has on researchers and their scientific outputs, let alone when the necessary data is either uncertain or unavailable.

However, we used the data that was available to us to estimate impact of IRF funding.

The most widely used variables to evaluate academic impact in the fields under review are publications in peer-reviewed journals and citation counts.
However, even within Natural Sciences and Social Sciences & Humanities, journal publications are far from the only forms of output from research.

Other important research outputs include inter alia monographs and book chapters, open-source data, software, patents and intellectual property.

Moreover, it is hard to compare journal publications and their citations across academic disciplines.
For example, it has been acknowledged for many years that research outputs from the Social Sciences & Humanities greatly differ from Natural Sciences’ outputs: book publications are more frequent than journal papers, the outputs are often in a national language and/or involve nationally or regionally specific topics that will not gather a wide audience and high 24 Impact Evaluation of the Icelandic Research Fund May 2022 citation count, and they are not as well covered in the large citation bases such as Web of Science and Scopus7.
Some evaluators have resorted to using the journal Impact Factor, a parameter that attempts comparison of publications across journals.

The journal Impact Factor is a measure of the frequency with which the average article in a journal has been cited in a particular year.

It does not exclude self-citations (authors citing their own work in other publications).
The journal Impact Factor was originally intended as a guide for librarians making purchasing decisions but has become a crude proxy measure of quality.

It has limitations and cannot be trusted blindly.

Recommendations for evaluating scientific outputs are plentiful in the literature.

The most prominent ones are listed in the widely adopted San Francisco Declaration on Research Assessment (DORA).
8 Of the 18 recommendations listed, the most compelling are: (1) stop using journal-based metrics in funding, appointment, and promotion considerations; (2) evaluate the merits of research itself instead of the journal it is published in; and (3) capitalize on the opportunities that follow online publications instead of print publications.
Because of our lack of ideal data, we were not able to evaluate scientific outputs in any other way than simply by number of journal publications and individual citations.
We are neither qualified—nor do we have access to experts—to evaluate the merits and quality of the publications under review and therefore we only chose to analyze the raw numbers of publications and citations Demographic information on applicants and detailed information about which outputs are produced using the IRF funding in question is hard to come by, and the evaluation is made harder by the severe lack of data from those who do not receive grant funding.
It makes comparison between recipients and non-recipients very difficult.

In order to quantify the effect on an individual from a grant, one needs data on all the individuals’ grants that overlap the grant period under review,9 which is not always readily available for large groups of individuals.

This also makes it troublesome to assess long-term impacts of grants.
Even though journal publications are not the only research output from grants, the effect of the IRF on science travels a great deal through the channels of publications and therefore our focus was on evaluating publication patterns of IRF recipients during the period under evaluation, as well as the years just after.

We explored the notion of how receiving an IRF grant would influence publications and did that by looking at the number of recipients’ peer-reviewed journal publications and citation counts.
It is essential in examining these to keep in mind that they are measures of activity and are not measures of quality.

The data we analysed came from different sources because there was not a common database that had the information we needed.

The data we used was provided by RANNIS, the Evaluation system for Public Higher Education Institutions, Reykjavík University‘s evaluation system (summaries across individual staff research reports), and Google Scholar®.
In this period, the IRF had 4 types of grants: grant of excellence, project grant, postdoctoral fellowship grant and a form of postdoctoral fellowship grant called the START grant (only granted in 2013).

Only individuals who received a project grant were analysed.

If individuals received multiple IRF project grants in the period, their first one was only used for the analysis.

Total individuals in our sample were 178, of which 55 were excluded (see the detailed 7 Glaser, J. a.

(2019).
(2019).

Persistent Problems for a Bliometrics of Social Sciences and Humanities and How to Overcome Them.

International Conference on Scientometrics & Informetrics (pp.

1056-1057).

Rome: ResearchGate.

8 DORA.

(n.d.).

DORA.

Retrieved from San Francisco Declaration on Research Assessment: https://sfdora.org/read/ 9 Heyard, R., & Hottentrott, H. (2021).

The value of research funding for knowledge creation and dissemination: A study of SNSF Research grants.
Humanities & Social Sciences Communications 8(1), 1-16.

25 Impact Evaluation of the Icelandic Research Fund May 2022 reasons for exclusions below), leaving 123.

As this analysis was only conducted for individuals receiving project grants, the 3 who had received a Grant of Excellence and the 8 who had received a Postdoctoral Fellowship Grant or a START grant, and had not already been excluded, were therefore removed from the data set as well, leaving 112.
Of those, 22 received their first grant in 2011, 20 in 2012, 28 in 2013, 22 in 2014, and 20 in 2015.

As is detailed in the introduction, for evaluating the impact of a grant received, we used a 5-year range for each PI, with the first year being the year of their first IRF grant during the years 2011-2015.

Therefore, the period of analysis is researcher-specific, with its lower limit in 2011 and upper limit in 2020.
We counted the total number of peer-reviewed articles that were published in this 5-year range by each individual, and then counted the number for each individual year.

We then found the proportion of each year‘s publications out of the year total.

We excluded from the analysis individuals who had not published a paper in more than 2 of the 5 years under review and those who did not publish a paper for the last 2 or more years in the timeframe.
These often indicated an individual who had changed position or line of work.

We also excluded those for which we could not find citations (for instance, those that did not have a Google Scholar® profile) to keep the sample the same as in the citation analysis (see below).

The aim was to explore whether there was a collective, visible and significant increase in publications at a particular time in or after the grant period, which usually lasted for 3 years.
Beforehand, we expected this increase to occur during the year the grant period ended or shortly afterwards.

This was not what we found.

Individuals supported by the IRF consistently produce journal articles.

Figure 9 shows constant publications over time with no statistically significant differences between years (Villa!

Uppruni tilvísunar finnst ekki., Appendix A).

There are factors that could confound these results.
For instance, we had almost no data on other grants the researchers might have received; we had no information on how many research groups these researchers were a part of; and different projects have different timelines and outputs.
It is worth noting, however, that an evaluation of project funding from the Swiss National Science Foundation10 found that receiving funding resulted on average in 1 additional journal article in each of the 3 years following funding, suggesting that research outputs are not strongly affected by funding.

10 Heyard, R., & Hottentrott, H. (2021).

The value of research funding for knowledge creation and dissemination: A study of SNSF Research grants.

Humanities & Social Sciences Communications 8(1), 1-16.
26 Impact Evaluation of the Icelandic Research Fund May 2022 Fig.

9: The proportion of peer-reviewed journal articles published each year out of the total peer- reviewed journal articles published in the six-year period from receiving an IRF project grant.

We ran a similar analysis on the researchers’ citations using publicly available data from Google Scholar®.

Each individual citation was counted in their 5-year range, for each individual year and in total.
Then, the number of citations for each year was divided by the total number, giving us a proportion of the total number for each year.

Analysing the citations is less precise than the publications: older publications that are not included in the study period can still be cited within the period, which means that citation numbers typically rise over time.

As for the publications analysis above, we hypothesized that we might see a collective jump or an increase in citations around 3 years post-grant award.
27 Impact Evaluation of the Icelandic Research Fund May 2022 Fig.

10: The proportion of researchers’ citations out of their total citations in the six-year period from receiving an IRF project grant As for the publication analysis above, we analysed statistically researchers’ citations, where each year was compared to its following year.

The difference between the years was significant for all compared years, except for between year 1 and year 2 (Villa!

Uppruni tilvísunar finnst ekki., Appendix A).


This is consistent with the observation that citations accumulate over time but gives no evidence of a jump in citations associated with a grant award.

The original intent was to look at the difference in publications and citations between recipients and non-recipients, but we were unable to do so because of data and time constraints.
Further, it would have been beneficial to analyze patents acquired by researchers while supported by an IRF grant but, unfortunately, we did not have access to the required data.

Conclusion We could not discern any specific increase in peer-reviewed journal publications resulting from an IRF project grant.

Year-on-year increases in citations are consistent with the normal accumulation over time and cannot be specifically attached to grants awarded.
It must be kept in mind that the data we used was neither robust nor complete, so strong conclusions should not be drawn from these results.

28 Impact Evaluation of the Icelandic Research Fund May 2022 4.

Educational Engagement Most of the grants included in the present analysis include training and participation in active research of Masters and/or Doctoral students and postdoctoral researchers (see Figure 11).
During the evaluation period, the participation of Doctoral students increased from 50 in 2011 to 72 in 2013 then decreased to 59 in 2015, while the number of Masters students was lowest in 2011 (42), highest in 2012 (58) and decreased to 52 in 2015.

The total number of Masters students who participated in IRF grants during these 5 years was 255 and there were 291 Doctoral students.

Whether some of them are counted more than once is unknown, as the available data set does not provide that information.
However, what is clear is that, on average, there were over 50 Masters students added annually to the pool of researchers working on projects funded by IRF grants, and over 55 Doctoral students.

Fig.
Fig.

11: Number of Masters and Doctoral students in IRF-funded projects (2011-2015) Participation of Masters and Doctoral students in IRF-funded projects 2011-2015 80 70 60 50 40 30 20 10 0 2011 2012 2013 2014 2015 Masters Doctoral Results from both the survey and interviews show that PIs perceive training scientists to be one of the most important impacts of the IRF.

This perception is borne out by the fact that 291 doctoral and 255 masters students were supported by the grants.
Almost all members of faculty have teaching requirements and/or administrative duties that take up time.

Masters and Doctoral students and postdoctoral researchers most often provide competent and knowledgeable support for research while PIs are covering other duties.

Educational engagement can go beyond university students.

One of the interviewees, Guðbjörg Ásta Ólafsdóttir11, PhD leads the University of Iceland´s Research Centre - West Fjords in rural Iceland.
She talked about how they engage the compulsory schools (1st - 10th grade) in research and work with local science teachers to bring young students into their studies.

They created a program for 5th – 10th grade students, who write a research plan, go out into the natural environment to gather samples/data, learn 11 Name disclosed with Dr. Ólafsdóttir’s permission.

29 Impact Evaluation of the Icelandic Research Fund May 2022 to analyze what they collect, and write about the results.
When asked about the impact on students the researcher said “I don’t know yet whether these kids will end up as scientists but they love this programme and are very engaged during the process.” 5.

Social, Cultural and Economic Impact Impact on society is increasingly recognized worldwide as an important element of research.
It is not simply an economic measure but can take many forms: from simple public engagement through, for instance, talks or media articles through to outcomes in which there is demonstrable change in one form or another.

This might be manifested in many ways, including policy formulation, significant cultural events and product/process development.

The evaluation of social impact is complex and a framework for this does not yet exist in Iceland.
Here, we evaluated the social, cultural, and economic impact of research the IRF supports via the survey and the interviews.

Participants told us that they communicate and apply their science and scientific results in many different venues other than academic outputs.
Out of 42 survey respondents who commented on where, other than academia, they communicated or presented their science, 32 said that they had been interviewed on radio, television or newspapers, 9 had participated in public engagement projects, and 21 mentioned social media presence.

During interviews, many of the participants in educational research talked about their feedback to their stakeholders of research (teachers, administrators, students, etc.)
and found that it was their duty to communicate their results by either offering workshops or special presentations.

Additionally, they perceived it as a “thank you” for allowing them access to their school and classrooms.

Participation in governmental committees was discussed by many and was perceived as both a positive and a negative experience as it was often time-consuming but with the potential of having a major impact on the topic of the committee or workgroup (Villa!
Uppruni tilvísunar finnst ekki.).
Interviewee or survey participant Impact on society - visibility in society 2 38 56 79 3 TV, and there has been extensive social media coverage on one of our papers Documentaries on TV, podcasts, interviews Podcasts, interviews, TV programs, social media, exhibitions Several articles in newspapers, Radio and television, Scientific blogs and social media “I have been on a few governmental committees on environmental topics.” Table 7: Sample responses to the question “What is IRF’s impact on society – scientists’ visibility in society?” There is at least one major new startup company that can be traced, at least in part, to funding from IRF during the evaluation period.
Oculis is an Ophthalmology company: “Oculis is a clinical stage biopharmaceutical company whose mission is to develop novel topical treatments (eye drops) for 30 Impact Evaluation of the Icelandic Research Fund May 2022 ophthalmic diseases for both back- and front-of-the-eye in order to improve the sight and the lives of patients worldwide.” 12 One of the cofounders, Dr. Þorsteinn Loftsson13, was an IRF recipient during the evaluation period (project grant, 2013) where his team did basic research on the science the company was based on.
The study title was “Cyclodextrin microparticles for targeted ocular drug delivery” and the company owns many IP’s related to this research.

14 .
14 .

Another company that has been impacted by the IRF is Rannsóknir og Greining (Icelandic Centre for Social Research & Analysis) a company established in 1999 by Inga Dóra Sigfúsdóttir15, PhD, who received a project grant (Multilevel Analysis of Risk and Protective Factors for Risky Behaviours in Adolescents) in 2012 from IRF that to a great extent helped the company establish itself in the area of health and wellbeing of youth in Iceland.
The results have been used to create preventative measures and programmes for youth in Iceland and have been published widely both in mainstream media and scientific journals.

These results and the programmes created out of the results are used by countries all over the world.16 Dr. Sigfúsdóttir received a large EU grant in 2015, which in large part was based on the study funded by the IRF.
According to her, the IRF grant made room on her schedule to apply for the EU grant as it relieved her from some other duties within Reykjavik University where she works.

Many PIs mentioned during interviews that one of the primary direct impacts the grant has is to train, not just educate, scientists who then are hired by Icelandic companies (Villa!

Uppruni tilvísunar finnst ekki.).

As a consequence of these comments, we decided to interview representatives from 7 well- established Icelandic companies.
All of them agreed that it could not be underestimated how important it is for them to be able to hire Icelandic graduates with scientific backgrounds relevant to their companies.

Some said that they could not operate in Iceland without Icelandic graduates with scientific training, and some even went as far as saying that they would have had to move the company abroad if they did not have access to these graduates.

12 Oculis.

(2022, 1 17).

About us.


Retrieved from https://oculis.com/company/about-us/ 13 Name disclosed with Dr. Loftsson’s permission.

14 Oculis.

(2022, 1 17).

Intellectual Property.

Retrieved from https://oculis.com/science/intellectual-property/ 15 Name disclosed with Dr. Sigfúsdóttir’s permission.

16 The Atlantic.

(2022, 1 17).

How Iceland Got Teens to Say No to Drugs.

Retrieved from https://www.theatlantic.com/health/archive/2017/01/teens-drugs-iceland/513668/ ABC News.

(2022, 1 17).
(2022, 1 17).

Iceland cuts teen drinking with curfews, youth centers.

Retrieved from https://abcnews.go.com/Lifestyle/wireStory/iceland-cuts-teen-drinking-curfews-youth-centers-64679265 Behavioral Scientist.

(2022, 1 17).

To Solve Problems Before They Happen, You Need to Unite the Right People.

Retrieved from https://behavioralscientist.org/dan-heath-iceland-drinking-to-solve-problems-before-they-happen-you-need-to-unite-the- right-people-upstream/ BBC News.

(2022, 1 17).
Retrieved from How one country persuaded teens to give up drink and drugs: https://www.bbc.com/news/av/stories-41973296 31 Impact Evaluation of the Icelandic Research Fund May 2022 Interviewee companies Impact on society - trainees as potential employees 1 2 3 “We need individuals who have knowledge in chemistry and biochemical engineering and who come from the universities.” “Almost everyone in our company has a university degree but more than 80% are at the BSc level but we are at the point of needing to hire MSc and even PhD level individuals.
But it’s not just the degree we look for, we need individuals who are able and willing to retrain and learn new things.

Technology changes fast and the science behind what we do is constantly changing.” “We started out by importing scientists for our company but in later years we prefer to hire Icelandic graduates as the likelihood of them staying with us is a lot greater than the foreigners.

Almost all our staff in the labs are Icelandic and actually, of our staff, Icelanders are about 90%.
” Table 8: Sample responses to the question “What is IRF’s impact on society – trainees as potential employees?” When asked whether it was important that the candidates were Icelandic, speak Icelandic and have lived in Iceland they all said that it was a key to a successful employee (Villa!

Uppruni tilvísunar finnst ekki.).

Some said that they had tried to hire international experts to work in Iceland, but most of the time they did not last more than a year or two.
Because it is very expensive to train someone for these jobs, companies regarded the effort as low return on investment.

The main reason for foreigners to leave after such a short time was not the work or the company, but rather a sense of isolation and the Icelandic winter.
Interviewee Companies Impact on society - importance of Icelandic Universities 1 2 3 4 “We compete with the other companies in Iceland who need to hire individuals with scientific backgrounds and the competition can be at times pretty intense.” “We could not exist without graduates from Icelandic Universities, we need scientists to exist.” “We recruit heavily from STEM departments at Icelandic universities.
We try almost everything to recruit them to us but the competition from other Icelandic companies is fierce.” “We rely on graduates from Icelandic universities to work in our company, they are our workforce.” Table 9: Sample responses to the question “What is IRF’s impact on society – importance of Icelandic universities?
32 Impact Evaluation of the Icelandic Research Fund May 2022 When asked what level of education was the most important to them and their companies, they all said that it depended on the position, and what was needed by the company (Villa!

Uppruni tilvísunar finnst ekki.).


On the other hand, what was important was whether they could think like scientists, could be creative in problem solving situations, innovative in thought, able to test a scientific idea, write a research plan, and communicate orally and in writing their science and purpose.
Interviewee Companies Impact on society - education and training of scientists 1 2 3 5 7 “In my department there are 16 out of 19 with a university degree and within the company there are 35+ with a university degree and of those around 30 have a scientific background.

We have all levels of degrees from BSc to MSc to PhD but the higher the degree the fewer individuals we have.
We do expect individuals with the highest degrees to be very good in communication in Icelandic and at least English and they need to have training in both science and communication.” “We are looking for part-time employees who are students that could potentially become full-time employees later on.
That way we can train them when they are students and then we can hire fully trained individuals as full-time employees.” “In our labs, or on the floor, we have mostly BSc educated individuals who have some sort of clinical laboratory training, but we also need MSc trained individuals who lead in the labs and have more education and training.

Once the specimens have been treated the data moves to individuals who analyze the very complicated data output.
There we need statisticians and informatics educated individuals.

After that it’s clinicians who take the data and create meaning out of this.
All in all, we have highly educated individuals in the whole process.” “What we need are individuals who are trained in doing science, not just with a degree but with training working in a lab with a critical eye, which is based on knowledge of course, with critical thinking when reading and interpreting lab results.” “When hiring I often look for old fashioned scientists; individuals who know how to test an idea from A to Z, know how to write a plan for a science experiment, just know how to do science.” Table 10: Sample responses to the question “What is IRF’s impact on society – education and training of scientists?” Conclusion The impact that the IRF has on Icelandic society, culture, and economy could be better evaluated if there was a clear framework for doing so.
However, it is clear from the survey and anecdotal data that impact on the Icelandic knowledge economy is both primary via companies established on the basis of science the IRF supports and secondary via the graduates who are employed by these companies.
The cultural impact is evidenced by engagement of scientists in governmental committees as experts, their participation in exhibitions, social media, and mainstream media, as well as their use of data-based information in problem solving, discussion points, and recommendations while working as experts.

33 Impact Evaluation of the Icelandic Research Fund May 2022 Conclusions 1.

Background IRF awards were examined, 2011-2022.


Analysis was restricted to grants awarded 2011-2015 and 2011 was the earliest year for which viable data was available; to capture award outcomes (academic and/or societal impact).

The awards include Project Grants (the largest category); Grants of Excellence; Postdoctoral awards; and awards for Doctoral students (since 2016).
Data was made available by the Icelandic Centre for Research (RANNIS), individual universities, University of Iceland´s coordinators of the Evaluation System of Icelandic Public Universities, Statistics Iceland (Hagstofan) and the Reykjavik University evaluation system.

New data was created using a specifically designed survey and personal interviews.

2.
2.

Principal findings, listed under the headings of the brief from the (then) Ministry of Education, Science & Culture Funding patterns: Applications for all awards are robust across years, with a success rate across grant types of just over 20% (similar to equivalent international agencies).

There is no evidence of gender bias in award success rate, though women make slightly fewer applications.

The rate of application for both men and women show an upward trajectory across years.
Academic disciplines were grouped into Natural Sciences and Social Sciences & Humanities.

There are no statistical differences in success rate between these.

Survey data indicated very positive views of the performance of RANNIS in organizing applications and awards; some types of communications from RANNIS were viewed less positively.
Springboard to other activities and funding: Data were scarce but a key finding is that applicants rejected for an award are highly likely to persist in making applications, often by adapting their work in light of reviewers’ comments.

The IRF is clearly valued by the research community.

The survey highlighted anecdotal evidence that the IRF was an important first step in grant-getting, important for early-career scientists and a step towards larger international grant awards.
Scientific outputs: At present it is not possible to track specific research outputs to specific IRF grants, a difficulty made harder when individuals hold multiple awards.

Surprisingly, it is clear that acknowledgment of grant awards in publications is not universal.

Publication of journal articles by IRF grant holders is consistent across years and citations rise year-on-year, as is expected.
Educational engagement: Doctoral and Masters students benefit directly and indirectly from IRF awards, being engaged in sponsored research.

Training next generation scientists was recognized by award holders as an important part of their work.

Social, cultural and economic impact: Societal impact in whatever form – inter alia, public engagement through multiple forms or demonstrable change through cultural, economic and policy impact – is not being captured, evidenced or curated systematically.
However, survey data indicated a strong engagement of IRF award holders outside academia, ranging from media appearances to company formation.

Companies in Iceland report dependence on a pipeline of well-trained scientists; the role of the IRF in supporting research activity is regarded as important for this.

34 Impact Evaluation of the Icelandic Research Fund May 2022 3.

Important Caveat Existing data lacked clarity and coherence.
In some instances, data was inconsistent across sources and some of the analyses suggested by the Ministry of Education, Science & Culture were not possible because of either poor quality or no data.

We believe that the conclusions reached are valid and reliable, but they rely on a level of analysis that lacks granularity: more could be done with a clear and coherent data set properly curated, including that within the developing Icelandic Research Information System (IRIS).

4.
4.

General Observations The IRF provides invaluable support to Icelandic researchers and postgraduate students.

The benefits extend through academia and into the Icelandic knowledge economy.

It is well organized by RANNIS, though with some critique of communications.

At the level of granularity, we have been able to use there is no evidence of gender or discipline bias.

Analysis of both academic and societal impact would be greatly improved by more robust data definitions, collection and curation.
Such analysis is important for individual researchers, universities, funding bodies and government.
35 Impact Evaluation of the Icelandic Research Fund May 2022 Appendix A Years y0-y1 y1-y2 y2-y3 y3-y4 y4-y5 Estimate t 95% CI lower 95% CI higher -0.012 0.017 -0.017 -0.004 -0.001 -0.873 1.221 -1.234 -0.036 -0.085 -0.038 -0.010 -0.044 -0.036 -0.085 0.015 0.044 0.010 -0.244 0.027 df 110 110 110 110 110 p 0.385 0.225 0.220 0.808 0.933 Table 11: Five independent paired t-tests, where each test compared peer-review paper publications from researchers in one year to the following year.
*Significant at α = 0.01 (Bonferroni correction for multiple tests).

Years y0-y1 y1-y2 y2-y3 y3-y4 y4-y5 Estimate t 95% CI lower 95% CI higher df p -0.015 -0.010 -0.021 -0.013 -0.016 -4.420 -2.570 -6.030 -2.868 -3.261 -0.022 -0.018 -0.028 -0.022 -0.026 -0.008 -0.002 -0.014 -0.004 -0.006 111 111 111 111 111 < 0.001* 0.012 < 0.001* 0.005* 0.001* Table 12: Five independent paired t-tests, where each test compared citations to researchers in one year to the following year.
*Significant at α = 0.01 (Bonferroni correction for multiple tests).

36 Impact Evaluation of the Icelandic Research Fund May 2022 37 Impact Evaluation of the Icelandic Research Fund May 2022 38