be better science. Many of the speakers today have already spoken about different aspects of this.

In the last session, we heard about preprints. I am a big fan of preprints. They lead to faster communication. They focus on the content, not the container. Open access, open data and code sharing create a global audience. They maximise a public good within and beyond the academy. They have the potential to increase the scrutiny and reliability of data. Now, there are questions about these innovations. They are not magic solutions, they are not silver bullets, and we have already heard today about questions of reliability and the misuse of preprints. Particularly, in the last session focusing on Diamond Open Access, we heard about the issue of how we need to ensure equitable access for authors. I was very encouraged by what I heard in the previous session.

Generally, and this would be the stance I think of the EC Report and the UNESCO report, open science is better for a changing world. It is making science better and more relevant to more people. In DORA, we are playing a part in that, and increasingly, although our focus is primarily on research assessment, we have to recognise that we are operating in a complex landscape and there are intersections with open science and open scholarship. Although open science is not mentioned in the DORA Declaration, our focus on recognising a broader range of contributions to the academy within research and to society resonates very much with the philosophy of open scholarship. We also have to recognise how biases that operate within research assessment are propagating historical inequities and stereotypes that still to this day exclude women. They exclude people from ethnic minorities and exclude disabled researchers from the academy. Equally, considerations of equity and inclusion interact with open scholarship because it forces us to ask who the academy is open to, and right now it is not sufficiently open to all people.

It is important to have a vision to meet these challenges, but we do have to be realistic, and we must not be naive about the pressures on the system. Pressures that have distorted our incentives and that can lead to disincentives of good practice and the difficulties with regard to time and resource pressure which the major actors need to bring to this party. We do need to interact with funders, with governments, with universities and with our researchers. We have to be very much open to dialogue, to hear their concerns about the change. Change brings many different challenges. However, at the same time, we must not lose sight of our vision, but already I hear in many of the contributions to today's conference the willingness to roll up our sleeves and get stuck into the hard work of doing that.
DORA is very much joining in with that effort. We are a relatively longstanding organisation. We received a significant funding boost in 2017 and that has allowed us to have a much more international profile. Our funding is international, and our steering committee now is absolutely international because we know that this has to be a worldwide conversation. DORA has three priorities in its strategy: to promote the declaration and to extend the global and disciplinary reach of the declaration, but the most important one is to develop and promote best practice. We bring a very practical lens to this effort. It is not just about criticising people for misusing impact factors or other aggregate indicators. We want to be part of finding workable solutions to the problems that we face.

A summary of a few of the resources that we have been developing over the years includes briefings we produce, meetings and workshops we convene, webinars we run and articles we publish. If you look at our website, we are collating and curating a resource library of good practice from universities and organisations around the world and in collaboration with SPARC Europe¹¹ and EUA¹² we launched a series of case studies last year. I am a great believer in the philosophy that nothing succeeds like success. I think in this sector there is a great willingness to embrace open science, but a great hesitation to know how to take the first step, and we need to identify and promote those in the vanguard who are willing to experiment, and learn and share the lessons with the rest of the community so that we can all move forward together.

We are an extremely small organisation. We have two members of staff plus an intern. They are fantastic people and do a really great job for DORA, but we enhance our impact by collaborating with many other organisations, with the Royal Society on developing the narrative CV¹³, for example. We recently ran a workshop looking very critically at narrative CVs, because again there are no silver bullets in this game and we must be wary of any dangers that are introduced by the reform of research assessment practices, even if the vision obviously is to make them much more robust and much more fit for purpose. We have helped develop policy in the Wellcome Trust¹⁴ and worked with the Research on Research Institute¹⁵.

Project TARA¹⁶ is one of our latest and most exciting initiatives. It is funded by Arcadia¹⁷, a charitable fund of Lisbet Rausing and Peter Baldwin. This is largely an infrastructure development project. The most important component of it is the development of an online dashboard with which we aim to track the adoption and implementation of responsible research assessment practices worldwide, and by doing so we hope to do an even better job of discovering and promoting those that are really embracing the reform of research assessment and open science practices to then encourage others. Ideally when this launches, and an initial version will be launched this year, we
hope that this will stimulate an enormous worldwide conversation and I very much
hope that people will be banging on our doors to tell us about the good work going on
in their institutions that they want us to include in the dashboard.

There is lots of work that we are doing, and we are very much open to collaboration
with many other organisations, not least the EC and the good work being done here
through the Paris Call. But we recognise, and we have heard some notes today, that
there is resistance to this change. We have certainly seen that with the announcement
of Plan S, and we have seen that with the announcement of changes and reform at
the University of Utrecht, for example. We have heard that Plan S is too risky, unfair
and a violation of academic freedom¹⁸ and Professors Poot and Mulders are obsessed
with the impact factor and very concerned that it will lead to harm to Dutch science¹⁹.
I think these fears are overstated and misguided, but nevertheless it is important that
we hear them. I think they represent a genuine concern for the quality of science
across Europe and across the world and we must be ready to enter dialogue. I believe,
along with Atul Gawande²⁰, that people talking to people is still how the world's
standards change, and that is very much what we are doing here today.

Coming back to what I started my presentation with, something that represents an
update and the trajectory of history that we are talking about today, a year or two ago
the BBC remade the *Civilisation* series that it had first broadcast 50 years ago, but it
did so in a much more open way. Rather than focusing just on Western Civilisation,
the series embraced all of human civilisation and is titled *Civilisations*²¹ – plural – as
a result, and was presented not just from the perspective of one white man but from
a diverse range of experts, and that created much more value in the documentary
series. It created a series that was richer and much more relevant to a worldwide
audience. We must learn that lesson. That is the promise of open science, which we
need to get to through reform of research assessment, and ultimately what we are
trying to do here and what we hope to do is to create a new ‘new social contract’,
to refer to Michael Gibbons’ original paper. I hope that through the Paris Call that is
being announced at this conference then we can start to write that social contract.

## To go a step further

"Despite its very young age, DORA has already gained such prominence that it is now considered the new standard for research evaluation. The declaration has been signed by over 18,000 individuals and 2,500 organizations."
"Academic freedom is about being able to publish what you want, with freedom of conscience and expression. However, it does not necessarily mean being able to choose the journal in which one publishes, and all publicly funded research carries with it the moral responsibility to make the results of that research available to all, to maximize its impact and act in the most ethical manner possible."

"It is difficult to ask researchers to be innovative (or not) in their work. However, it is fundamental to place the robustness of research at the forefront of evaluation."

"We must remember that even citations based on numbers are subjective, not objective. There are biases within citation practices."

"In the US we have noticed that there are relatively few institutional signatories, certainly universities, that have signed. Some American scholars I have talked to say they have a good system, and it is the Europeans who are obsessed metrics. I am not sure that that is true. That is not necessarily what I hear from young American researchers. That is one of the things that we are going to explore in Project TARA. There is a role for national funding agencies here to think about the leverage that they can operate to promote good practice. Funders should not necessarily be afraid about trying to set the standard for what good research looks like."

"If one is doing research assessment based on numbers it certainly is quick, but it is not effective. It is missing out on evaluating many other important academic activities that get discounted in a system that relies unduly on metrics. We do need to think about the workload involved because researchers are very busy. I think the narrative CV format, which is something that we have been helping to promote, is a very promising way because that provides a structured and concise way to allow individuals to give a good account of the variety of activities that they have contributed to the academy, not just their impact factors."

"The impact factor is a good way to assess journals, but it is not a good way to assess individual papers or individual researchers. We ask for individuals and their work should be assessed and we say that they should not be assessed via a proxy that is a simplistic and often skewed number. We see that there is no real correlation between the importance that people attach to their work and the impact factors of the journals that they publish in."
References

1. *BBC One, Civilisation, 1969*, https://www.bbc.co.uk/programmes/b00dtjbv

2. Michael Gibbons, "Science's New Social Contract with Society", *Nature* 402, n° S6761 (December 1999): C81-84, https://doi.org/10.1038/35011576.

3. American Society for Cell Biology, "Declaration on Research Assessment (DORA)", DORA, 2012, https://sfdora.org/.

4. Diana Hicks et al., "Bibliometrics: The Leiden Manifesto for Research Metrics", *Nature News* 520, n° 7548 (23 April 2015): 429, https://doi.org/10.1038/520429a.

5. James Wilsdon et al., "The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management", July 2015, https://responsiblemetrics.org/wp-content/uploads/2019/02/2015_metrictide.pdf.

6. David Moher et al., "The Hong Kong Principles for Assessing Researchers: Fostering Research Integrity", *PLOS Biology* 18, n° 7 (16 July 2020): e3000737, https://doi.org/10.1371/journal.pbio.3000737.

7. Stephen Curry et al., "The Changing Role of Funders in Responsible Research Assessment: Progress, Obstacles and the Way Ahead", report (Research on Research Institute, 18 November 2020), https://doi.org/10.6084/m9 figshare.13227914.v1.

8. Commission Européenne, *Towards a Reform of the Research Assessment System: Scoping Report* (LU: Publications Office of the European Union, 2021), https://data.europa.eu/doi/10.2777/707440.

9. UNESCO, "UNESCO Recommendation on Open Science", 2021, https://unesdoc.unesco.org/ark:/48223/pf0000379949.

10. Michael Sandel, *A New Politics of Hope*, 2018, https://www.youtube.com/watch?v=LCZhA_1n4E.

11. "Scholarly Publishing and Academic Resources Coalition Europe", accessed 16 March 2022, https://sparceurope.org.

12. "European University Association", accessed 16 March 2022, https://eua.eu.

13. "Résumé for Researchers", The Royal Society, accessed 16 March 2022, https://royalsociety.org/topics-policy/projects/research-culture/tools-for-support/resume-for-researchers/.

14. "Guidance for Research Organisations on How to Implement Responsible and Fair Approaches for Research Assessment", Wellcome Trust, 2021, https://wellcome.org/grant-funding/guidance/open-access-guidance/research-organisations-how-implement-responsible-and-fair-approaches-research.

15. Stephen Curry et al..

16. "Project TARA", DORA (blog), 2021, https://sfdora.org/project-tara/.

17. "Arcadia", accessed 16 March 2022, https://www.arcadiafund.org.uk/.

18. Richard Van Noorden, "Arguments over European Open-Access Plan Heat Up", *Nature*, 12 November 2018, https://doi.org/10.1038/d41586-018-07386-x.

19. Dalmeet Singh Chawla, "Scientists at Odds on Utrecht University Reforms to Hiring and Promotion Criteria", *Nature Index* (blog), 9 August 2021, https://www.natureindex.
com/news-blog/scientists-argue-over-use-of-impact-factors-for-evaluating-research.

20. Atul Gawande, “Spreading Slow Ideas”, *The New Yorker*, 22 July 2013, http://www.newyorker.com/magazine/2013/07/29/slow-ideas.

21. *BBC, Civilisations, 2018*, https://www.bbc.co.uk/iptayer/episodes/p05xxp5j/civilisations.
# Open Science Needs No Martyrs, but We Must Recognize the Need for Reform

Toma Susi
University of Vienna

My perspective is primarily that of a researcher. I am an associate professor, I do research. I run a small research group, but I also do some policy work for the Initiative for Science in Europe¹. One could say the title of my talk is a little bit provocative. It comes from a recent interview I gave for the ERC magazine². As long as the incentives in science are not aligned with the robustness, transparency and reproducibility of science, doing that kind of work actually is not a career benefit. Each individual is in a very competitive system. The numbers vary from country to country and field to field, but let us say 10% of PhDs ever get a permanent position. It is a very tough competition and I think everybody who manages to succeed in that system knows how to play by the system's rules, and so we should not expect individuals to put their own self-interest completely aside and martyr themselves, and in the worst case not have a career. Instead, we have to change the assessment system so that people's self-interests and the interests of science align much better than they do now.

We have already heard a lot about what open science is and why we want to have it. The UNESCO statement³ is a very important milestone on this. My focus as a researcher is on its open scientific knowledge part, and one should note that this is about a diversity of practices, not just publication, but data, code, methods and so forth. All these things serve to make scientific dissemination and scientific knowledge much more robust and reusable. Accordingly, there have been a lot of aspirations and policy statements. In particular, the European Commission has been very actively promoting open science in its funding programmes.

For many years we have had very strong-worded statements saying that open science must be embedded in every level of the scientific endeavour and all stakeholders must take responsibility for this. This is all well and good, but the fact is that researchers still get hired and promoted to a large extent on the names of the journals where they publish. I have personal experience with hiring processes
at four different European universities in the last five years and not even one would
have rewarded me for doing open science. Instead, journal impact metrics and the
number of publications were explicitly mentioned. I managed to succeed in that
system. I have also managed to support open access and open data, but it has
always been a balancing act between that and playing the career game successfully.

There are signs that this is slowly changing. For example, the European University Association said that in 34% of European institutions there were no open science elements⁴, which means at least in the rest there were some elements. I am surprised that it is even that high because, as I said, my personal experience is that journal prestige is still very much the career currency. This is a systemic problem, and the problem is that governments distribute public funding based on rankings and aggregated indices to the research institutes and universities, and that of course drives the strategy of these institutions because they want to receive funding and they do need it. Then those research institutions use metrics like journal impact factor, quartile ranks and h-indices to do their hiring and promotions, and thus these metrics flow all the way down to the individual researchers to drive recruitment and promotion.

It is a very easy to use and globalising system in some sense, but it has several serious flaws that DORA and others have highlighted for a long time. From my personal perspective, and I think DORA also recognised this, the most pernicious problem is the misuse of these journal-based impact metrics and the prestige that they signal to individual researchers. I should clarify here. For the individual, this is not irrational. If I have a prestigious publication that really is valuable to me in my career, it is not just an illusory thing. It is a very rational choice to chase those types of publications. That is what needs to change for the whole system to be able to change.

It is a difficult problem for researchers, especially early-career researchers. Many of us have some level of idealistic thinking. We want to do open data, open methods and software, try different things and collaborate a lot, but we are still hired and promoted based on how fast and how prestigiously we publish. Of course, having incentives so badly misaligned with aspirations is bound to not give us the results that we would want for the science system as a whole. That is why we really need to reform research assessment and change the evaluation culture that comes with that.

I have been working on science policy issues in recent years, especially at the
Young Academy of Europe⁵. We collaborated with other researcher organisations,
namely Eurodoc⁶ and the Marie Curie Alumni Association⁷. Early last year we published an opinion piece⁸, which got quite a lot of attention, where we highlighted this problem that we are locked into this system if we want to have a career. I pointed out that even if I have tenure now, I have trainees, post-doc and PhD students. I must worry about their careers, I cannot make my idealistic choices and have their careers suffer consequently. In this opinion piece we tried to think of ways in which the system could be changed. The very first thing we all need to do is we need to be very realistic and honest about the current reward system and its flaws, regardless of if we got hired and promoted in that system. We must recognize what the system is and actually in some sense how absurd it is.

Beyond examples of evolving practices, and there are good examples to highlight, as DORA importantly does, there needs to be a lot more urgent discussion, an internal dialogue, within the research community and between its different segments on what we want to reward and how we want to evaluate people. The status quo simply is no longer acceptable and now is the time to start being very concrete about how we change the system.

To this end, the Initiative for Science in Europe was running an open science working group. We had a virtual workshop in March 2021 with a very diverse set of stakeholders and we tried to think together about how we should reform the system. The draft report from that work has just been put online⁹. In this report we make a few main points. The main one is that researchers really should be central to how we reform research assessment. The time to act is now. The report explores which routes could be taken, at which levels change could happen and what steps different stakeholders could take to achieve comprehensive change while respecting any valid differences, such as between different disciplines. Evaluation in chemistry and medieval history probably is not and should not be the same, and so those differences should be respected.

We propose several policy options for different stakeholder groups. For research communities themselves, we urge that you need to have these conversations right now. You should have had them years ago, but now is better than never, and you must come up with concrete proposals. The message is: you cannot keep on using impact factors. What will you use instead, concretely? You have to decide, because if you do not decide, these impact metrics will continue to be misused or, as seems increasingly likely in Europe due to the activity of the Commission and other funders, new indicators will be imposed on you without your participation. You really must get involved therefore in a very concrete way, right now.
We also identify four essential principles, the main one being that researchers should be engaged in all decisions regarding changes to research assessment, and all stakeholders should engage with them from the beginning in their decision-making processes. Fortunately, there is already a strong researcher representation in the core drafting group for this European Commission initiative to reform research assessment.

We have to end the use of inappropriate metrics. DORA has said this for 10 years and now we just must do it. Honestly there is no valid reason to delay this any further. Of course, the hitch has been that we have known for a long time what we do not want, but we have not known what we want instead, and that is what we need to decide now. We need to agree on appropriate ways of assessing research and researchers in discipline-specific ways. There needs to be an appropriate balance between qualitative and quantitative evaluation. With metrics and indicators, some of them may be good and some of them may not be, so we have to evaluate them in each discipline and other contexts separately.

Finally – this is more for policymakers, universities, and funders – we have to all recognise that reforming assessment requires resources. We may have to come up with new systems. We may just need more time for evaluation, so perhaps we have to also consider whether all of the current forms of evaluation are needed or whether we could, for example, pool evaluations from different funding programmes and so forth.

Of course, we have seen that change is hard and changing entrenched mindsets is really hard. I have a personal anecdote that illustrates how hard this really is. Last year we achieved a very cool piece of science from my ERC project that we published open access with open data, and we tried to initially submit it to a very prestigious journal that has a glamorous brand. They said it was not novel enough. Fine, that is their subjective judgement, and maybe they are right. So we ended up publishing in a respected journal, a solid journal, that is just not as prestigious. Even I then caught myself thinking it was probably not worth a press release because it was not such an impressive journal. I really had to stop myself and realise I was caught in the same kind of thinking myself, even though I really should know better. Then I said to myself that it was the same research and exactly the same results and that I should make the press release. The university agreed and we got some nice publicity for the work.

Change is hard and it will take time to change the culture. I am not sure if we will be able to bring everybody along with us, but we certainly should try. However, it really is time to stop doing this to ourselves. Unlike publishing, evaluation is largely done by and to the research community itself. Of course, governments play a role,
but where do the governments get their ideas on how to evaluate? That comes from
the scientific community in the end. There is a lot of power within the research
community to change this and it really is now the time to start doing so. I am very
happy to see the Paris Call, the European Commission effort, and statements from
different university groups. There is a lot of momentum and I think we really have a
shot at making this happen.

To go a step further

"Senior researchers who are in policymaking positions, who are heads of scientific societies, who are in academies of science, have a lot of prestige and they really should use that prestige and that power to help change the entire system."

"The best form of government is democracy, so the best form of evaluation is peer review, and of course we should be the experts in our own discipline on what is good science and what is not."

"The point of DORA is not to enforce. It is to inspire and to motivate. The European Commission's proposal to form a coalition to reform research assessment will be based on a set of pledges. Signatories will sign up and there will be some form of monitoring. What form consequences would take is part of what is still being discussed. However, the idea is to take it one step further from an aspiration and from a soft power declaration."

"What might have kept DORA and research assessment reform back is that there was a huge focus on open access. Access to research is important. There are questions to solve, there is work to be done, but I think a lot of people's focus has now shifted to assessment, because changing assessment will also unlock so much more innovation in the publishing sphere. It will probably also trigger kind of a new wave in open access."

"We may need some balance of indicators, but ones that are more responsible, more transparent, and more useful, and then perhaps put increasing emphasis on qualitative peer review. That may take some more time."
References

1. "Initiative for Science in Europe", accessed 16 March 2022, https://initiative-se.eu/.

2. Toma Susi, "Open Science Needs No Martyrs, but We Must Recognize the Need for Reform", *ERC: European Research Council* (blog), 26 October 2021, https://erc.europa.eu/news-events/magazine/open-science-needs-no-martyrs-we-must-recognize-need-reform.

3. UNESCO, "UNESCO Recommendation on Open Science", 2021, https://unesdoc.unesco.org/ark:/48223/pf0000379949.

4. European University Association et al., "Open Science in university approaches to academic assessment", December 2021, https://www.eua.eu/resources/publications/999:open-science-in-university-approaches-to-academic-assessment.html.

5. "Young Academy of Europe", accessed 16 March 2022, https://yacadouro.org/.

6. "Eurodoc", accessed 16 March 2022, https://eurodoc-net.com/.

7. "Marie Curie Alumni Association", accessed 16 March 2022, https://www.mariecuriealumni.eu.

8. Véronique De Herde, Mattias Björnmalm, Toma Susi, "Game over: Empower Early Career Researchers to Improve Research Quality", *Insights* 34, n° 1 (9 juin 2021): 15, https://doi.org/10.1629/uksg.548.

9. Toma Susi et al., "Centrality of researchers in reforming research assessment", January 2022, https://initiative-se.eu/paper-research-assessment/.
# To What extend are Early Career Researchers the Harbingers of Change ?

Cherifa Boukacem

University of Lyon-1

The work presented here is a long-term research, which was initiated thanks to a study commissioned by Europe, in which we investigated the practices of young researchers around social networks¹. This study gave us the opportunity to discover a demography in its own right, designated by the dedicated keyword "early career researchers" in English, which can be used as such for bibliographic research. This demographic category is that of the digital natives who have highly developed digital practices, as well as trajectories and career paths that are at the heart of the attention of various players in the academic world, such as large groups of scientific publishers, funding agencies, but also social networking platforms, for example.

It is a social category in the academic world, which has a particular status and position, always in between: between seniors and the very young, between different communities, in the making in terms of career, but also at the hinge of two worlds. And it is around this community that we launched a research project that was funded by the Sloan Foundation and the Publishing Research Consortium over a long period, between 2016 and 2022, and which includes the current pandemic and looks at several countries around the world: France of course, Great Britain, Spain, the United States, China, Russia, and Malaysia. The study constituted an observatory around which we selected 168 young researchers, with whom we conducted semi-structured interviews throughout this period, every year, and every six months since the pandemic. 23 themes were explored in depth, since the interviews could last up to three hours, by exploiting their CVs and their accounts on academic social networks. These researchers were spread across the fields of life sciences, hard sciences, and social sciences, but not the humanities.

While the results show differences between countries, between researchers and between disciplines, where all the results converge in a particularly significant way is that there is a very significant alignment of researchers' practices and representations around research evaluation issues. This alignment comes from the
fact that there is a cult and a culture of scientific production around the article in high-impact journals, very early in the career of young researchers, from the Master's level onwards. As Toma Susi pointed out, the career starts very early, and it happens that Master 2 students are involved in research work and are authors of articles. From the Master's level onwards, students are involved in this logic of submitting articles, thinking in terms of articles, and producing articles in the best journals, the highest ranked ones, what we call A-rank journals with the highest impact factors, to constitute a form of distinction that will enable them to embark on a research career.

There is a real impact factor culture in the way we train young researchers, and this culture is sometimes negotiated between young researchers and senior researchers: am I going to submit this article to this journal with a high impact factor, or have a safer strategy and submit to a journal with a lower impact factor, or the opposite? These strategies, which are found from one young researcher to another, from one country to another, are really structured around the impact factor and the international databases, I am thinking of the Web of Science, Scopus, which are there to root for and justify this strategy.

We can speak of a culture in terms of scientific production, of "Publish or Perish". This focus on articles in high-impact journals, because there is a common terminology, has absolutely similar configurations and normative principles that really mobilize the same practices, actions and the strategies among all the young researchers that we have been able to interview over all these years, strategies that are moreover exacerbated with time. When we give them the floor, young researchers say that "you are what you publish". This title is only a verbatim of a young researcher interviewed. At the heart of this representation, the paper, the famous article, is an infinite quest since the collaborations built, the visits, the scientific stays, the mobilities, the post-docs that one goes to do in different countries, the methods that one learns, the protocols that one develops, have a common destiny: to be published in a journal with a high impact factor. There is also a particularly interesting phenomenon, which is the zombie paper, the zombie article, which is always late, which we always try to finish, and no matter how long it takes, we carry it around with us. It is possible to finish a contract, to go on to another research funding, and despite that, to keep trying to finish it, because this paper will have to justify a line in the CV.

Another point is the drift that this can lead to, namely the "Publish and be wrong", in the sense that by trying to publish quickly and easily, one sometimes targets a bad journal, which is not necessarily a quality journal, or results that are not
consolidated. In other words, we don't take the time to make sure that the positive
results we had the first time are repeated and confirmed, and we publish too quickly
results that are ultimately invalid.

Another issue concerns what can be called the mythology in CVs, carried by the injunction that young researchers must absolutely demonstrate a long list of publications in impact factor journals. When they all align themselves with this strategy, and then present themselves to a recruitment committee, what makes the difference? When they all have the same CV, having published in the right journals, with the right people, having done postdocs in the right prestigious places, what makes the difference? That's one aspect that really challenges these strategies.

There are also all these tasks, all these activities, all these things that you do around
the work cycle of a researcher that are not mentioned in a CV, that are not taken
into account.

It should also be pointed out that when young researchers develop an
interdisciplinary, multidisciplinary profile and then publish, they often find it difficult
to find journals with a high impact factor that are able to accept research that
conveys this interdisciplinarity. The logic of valuing multidisciplinarity on the one
hand, and not always finding an adequate, prestigious publication venue with a high
impact factor on the other, is sometimes problematic, and it produces epistemic
inequalities between researchers. And then there is the famous Matthieu effect,
the law of cumulative advantages that says that when you have published once in a
prestigious journal with prestigious authors, you have a better chance of publishing
in other prestigious journals and of being integrated into prestigious groups. This
is a mechanism that young researchers integrate very quickly, without particularly
knowing the name or the principle.

How to change the rules? How to confront the peers? How to question these rules,
even though with these peers, there is the image of two roped climbers who climb
a wall and are connected by a certain solidarity. The seniors look after the interests
of the juniors and, for example, recommend that they publish in prestigious journals
because that's how you get into the system. They see themselves as the guardians
of the temple and want to maintain this tradition, this way of working, and they want
the young researchers to reproduce it. And in fact, what the results of the Harbinger
project show us, especially today, is that the revolution will necessarily take place
without the peers, without the seniors, because you have to take risks, you have to
seize opportunities, the preprint servers are proof of that. We have to encourage
activities, work that is not sufficiently valued for the moment, such as the work that is
done around data. The pandemic has also shown how important it is to deal with data, to value it and to share it. I draw a parallel with an 18th century literary work, Carl Von Carlsberg, a novel about a young university assistant in Germany who couldn't get a permanent position, and after a while confided in one of his peers who said, "You just have to marry one of our daughters, since we have many daughters. So if you want to join the circle, you have to join it by marrying one of our girls." The novel is known for this, and I find the parallel particularly interesting.

Another point I would like to quickly go over is that this injunction to publish a lot of articles in high impact factor journals, since there is a very strong focus on articles, can create confusion and misunderstandings, which social networking platforms have been very good at exploiting since the digital practices of young researchers mean that they are browsing the web, on platforms, and they sometimes lose their bearings between these platforms and the differences between them. It is not uncommon to see young researchers who are very happy to use social networks such as ResearchGate to seek visibility, to seek social capital, and to observe how communities function because they find their peers and colleagues on these platforms, to the point of confusing these social network platforms with open archives. Besides, if we take the SciHub phenomenon, some researchers justify it by saying clearly that they participate in a form of opening of scientific publication, since it is closed and it must be opened, and that SciHub contributes to it. So, this leads to these confusions, which are ultimately real obstacles to legitimate and official models of open access and open science.

How to open up the game? First, on an individual and collective scale, these young researchers we've talked to over the years don't believe in the impact factor. They play with it, as they say, they have to deal with it, but they don't believe in it and they don't share the logic of the impact factor. They question it, and are very aware that neither the impact factor nor the new indicators are the solution to the system. Some of the young researchers interviewed are gamers, who say they know very well how to "game" the system, how the indicators work, and how to play with them. In fact, the arguments they put forward are the arguments of DORA, without mentioning DORA, which they do not necessarily know. So we are in a very troubling alignment when we analyse the interviews, because they are already compliant with what we claim, but without wanting to name it.

I will end with an extremely important point, namely the results of the pandemic period of the study, which showed to what extent the pandemic has very significantly modified the practices, representations and strategies of the young researchers we knew until then or who joined the study. New trajectories are emerging. The
pandemic showed this generation how far science was in an ivory tower from society. They were very mobilized, very attentive, and very interested in observing the way in which information about the virus circulated in the media, the way in which the debate about vaccines occupied our daily lives, and the debate they made was that by dint of chasing publications, chasing impact factors, science had finally forgotten to address society, the citizens. The question of the debate on science in society, of the dissemination of knowledge to society, has become for some the absolute emergency, to the point that they have changed their career path and do not wish to join the academic world, but rather the world of knowledge dissemination and scientific mediation. We are facing a generation that is changing its career objectives, and that says that science should have the possibility, the right to be wrong, and to recreate scientific debate because the system as it works today does not allow for debate. It just allows for publication in high impact journals to prove that one is right. Finally, they want the way science communicates to adapt to today's world, to allow more diversity in the forms of knowledge, in the new skills that can be valued today, and in the new forms of value that allow us to take into account the complexity of science today, which is not the same as it was at the beginning of the 20th century, not the same as it was at the time of Pierre and Marie Curie in any case.

To conclude, I would like to quote a researcher who left the system saying that he no longer recognized himself in this game, that he wanted to "stop playing" because the pandemic has shown on a worldwide, globalized scale that the ultimate goal of science is to advance knowledge, not to advance egos, indicators, figures or business figures.

## To go a step further

"Scholarly publication should not be restricted to journals and articles. It is a question of how to bring value to other skill sets and to other areas of expertise which do not necessarily result in the publication of a paper."

"In the change of paradigm, heading towards research assessment which is more quality-based, what you do is you make it possible for researchers to be active participants in this assessment. You ask researchers what fields of research and which research activities they consider to matter most over a period of time. You get indicators based on the opinions of researchers rather than ones provided by very costly databases owned by hegemonic commercial players."
References

1. CIBER Research, "Harbingers", accessed 16 March 2022, http://ciber-research.com/harbingers.html.
Research Assessment
– Second Session
<figure><img src="image_33.png" /></figure>
Research Assessment and Open
Science in a Diverse World

Fernanda Beigel

CONICET National University of Cuyo-Mendoza, Argentina

I would like to start by recalling that the Open Science project has been in existence for decades, although we can say that it has entered a consolidation phase in recent years. The Unesco Recommendation for Open Science¹ is a step in this direction. It provides a general framework, a definition that tends towards a global consensus, a set of principles and values and, finally, a list of actions to face. Among the latter, it is important to mention the need to align the incentives of the research evaluation systems with the progress of open science. How can we promote open science through the evolution of research evaluation modes?

Among the various points concerned, it is fundamental to consider all the missions of universities, beyond publication activities, which we assimilate to research activity. I am referring to teaching, but also to what we in Latin America call extensión universitaria (third mission). When working on the Unesco Recommendation, the experts realised that there was not a general consensus on the instruments of promotion and recognition of open science in research evaluation. It is therefore essential to recognise the plurality of incentives that exist, and to consider them in context.

The Unesco member countries that participated in the making on this recommendation also discussed at length one critical issue in the road to open science: inequalities. There are indeed visible inequalities today between what is commonly called the North and the South, between countries that could be described as peripheral or semi-peripheral and central countries. There is a significant digital divide, particularly in terms of infrastructure. The increase in these inequalities can also be seen in the transition to open access as it is currently taking place in European journals, particularly in connection with the move to the Article Processing Charges (APCs) model. In Latin America, this model represents a huge difficulty, due to the very high amounts involved. The Recommendation also defends multilingualism because English is unfortunately often the only code for interoperability between and across databases.
I would also like to mention another asymmetry that is important for the transformation
towards openness in research assessment, namely the very notion of excellence. I
have tried to represent in a diagram the way in which the differentiated vision of the
North and the South can be seen in the power relations of the academic system.
At the top is academic excellence, also more appropriately called 'research quality',
which has been identified with mainstream science, frequently identified with the
main stream of scientific publishing. At the very bottom of the hierarchy is the
so-called peripheral science, i.e. everything that falls outside the circuit of what is
rewarded in the traditional research evaluation system. Alongside this, there are
various transnational circuits (notably the Latin American system), which were born
directly out of open access, with a desire to open up the academic dialogue, as well
as national circuits, which continue to exist. In these peripheral or semi-peripheral
countries of the South, different levels of prestige conferred by publication have thus
arisen: a national level of prestige (which I am observing here mainly for Latin America),
a regional or international level of prestige, with different evaluation cultures for each.

<figure><img src="image_34.png" /><figcaption>Fig. 7 : Traditional representation of the academic system</figcaption></figure>

Source : Beigel, "Research Evaluation in the Southern Road to Open Science" ( Media used during the presentation (available on the conference's website) and Fernanda Beigel, "Circuits and fields. The multi-scale process of academic prestige-building", in Keim, W. et al., Handbook On The Circulation of Academic Knowledge. Forthcoming.

There is a boundary between the different evaluation cultures, which distinguishes
between what might be called national prestige, but which is in fact only recognised
within the institution of origin and in a particular country, and what is called
international prestige, but which, due to inequalities between countries, is in fact reduced to the scale of prestige specific to the country concerned. In Latin America, there is also a certain type of international prestige, which is truly shared within a group of countries, and which can therefore be described as regional, in connection with a very specific publishing circuit.

The left arrow on my diagram represents the fact that when we ask what academic excellence is, we are in fact asking what the best science is: the one that has the most impact? Given that this notion does not originate in this region of the world this is how academic excellence has been defined in the countries of the South. I have shown it separately from the diagram, because it is partly imposed from outside, due to the internationalisation of the academic research system developed from the 1990s. Open science places a precise challenge to this inequality in terms of the autonomous definition of the research agenda.

Let us mention the case of China, in relation to the notion of imposed excellence. In the reforms currently underway in China, we recognise the perverse effects that this type of heteronomous imposition can have, notably through the very rapid adaptability that Chinese researchers have shown, with the generalisation of the use of English and the advent of a fracture between research concerns and societal concerns, all in the name “impact”, but eventually embodied in the Impact factor. It is therefore important to emphasise that the reform underway in China shows the need to support national policies to assist local scientific publication.

If we now look at the other relevant concerns of experts and Member States when developing the Recommendation for open science, we see that these were essentially related to global inequalities in access to resources and infrastructures.
<figure><img src="image_35.png" /><figcaption>Fig. 8: Open Access Repositories</figcaption></figure>

Source: Beigel, "Research Evaluation in the Southern Road to Open Science".
Media used during the presentation (available on the conference's web-site). Based on data extracted from re3data.org - Registry of Research Data Repositories. https://doi.org/10.17616/R3D, accessed January 2022.

To go further in this study of global inequalities, let us look at the global landscape of repositories with scientific production in Southern countries. It can be seen in Figure 8 that there are not yet many publication repositories in Africa, but that Latin America is among the best equipped countries in the South. On the other hand, if we look at Figure 9, that is open data repositories, we can see a glaring difference in the number of repositories between the North and the South, including in Latin America, even if the contrast is still greatest with Africa.
<figure><img src="image_36.png" /><figcaption>Fig. 9: Open Data Repositories</figcaption></figure>

Source: Beigel, "Research Evaluation in the Southern Road to Open Science".
Based on data extracted from re3data.org - Registry of Research Data Repositories. https://doi.org/10.17616/R3D, accessed January 2022.

Other inequalities can be observed in the current global distribution of research information systems or CRIS (Current Research Information System), which allow for the fully integrated administration of all kinds of information that are currently managed independently within the countries of the South, i.e. interfaces for the management of scientific policies, human resources or research projects on the one hand, and open archives or open data warehouses on the other. CRIS systems link the repositories and information databases, providing highly interconnected and interoperable open science data. There are about 950 CRISs listed in EuroCRIS², most of which are located in Europe and northern countries.

Next is another type of global asymmetry, affecting to publications, which can be seen when looking at the distribution between Diamond Open Access journals and open access journals with APCs. South America has 2,612 diamond journals listed in DOAJ and only 130 APCs journals, which is a peculiarity of this region. Western
Europe has nearly 4,300 journals, of which 1,907 are open access with APCs. The latter are generally the most prestigious, and therefore the most sought-after by scientists in the South. But how will they pay individually for an average 2.000 dollars to publish a paper?

I would now like to focus more on the linguistic face of the global inequalities in science. There are a number of tangible initiatives around the main challenges facing our continent, namely multilingualism and the hypercentrality of English. There are initiatives such as the Helsinki initiative³, the Latin American Forum for Research Assessment⁴, which launched CLACSO, or the Jussieu Call⁵, which insists that to have a real impact on society, research must be multilingual. Let's look at the particular strengths and weaknesses of Latin America in terms of bibliodiversity and open access. There is a strong historical regionalism since the 1950s, with the early development of national information management systems, although, as mentioned earlier, these systems were disconnected from other bases such as open access repositories. Regional indexing systems have also been set up, allowing the professionalisation of thousands of diamond open access journals. Latin America also has a very strong and dynamic publishing industry, with books published since the 1930s, and a very long tradition of non-commercial open access.

In addition to these strengths, there is legislation in several countries in favour of open publications and open data, and a federation of repositories with more than 790 institutions in more than 10 different countries. An initiative led by CLACSO, which lists the different actions carried out in Latin America, gathers interesting data on this theme, and specifies that there are 57 instruments relating to open science policies, the majority of which are based on collaborative infrastructures. I also would like to briefly delve in an important tradition of citizen and participatory science, the so-called extension universitaria (or third mission of universities), which has a long history of co-production of scientific knowledge in collaboration with the social sector. All these interactions must be highlighted in the research assessment systems but also revalued in the transition to open science.

One of the key points in this respect is multilingualism. The hyper-centrality of English is now absolutely clear in the research world. However, although English is a dominant reality, it is not the only reality, and in South America we have a number of channels for the circulation of knowledge that bypass the use of English. For example, if we search Scopus or the Web of Science, we find Spanish and Portuguese articles as marginal shares. In contrast, the OLIVA⁶ database, which combines entries from SciELO and Redalyc and has 345,391 entries in Spanish, more than the Web of Science (270,632 entries) and almost as many as Scopus (373,419 entries). The same observation can
be made in BIBLAT⁷ and LA Referencia⁸, the latter database containing many more references in Portuguese than Scopus and the Web of Science.

Well now, how “aligned” to these strengths are the rewards promoted by the research assessment systems in Latin America? We can observe a contradiction, even a form of “alienation” between these regional potentials for open science and the orientations led by the evaluative policies regarding individual careers, institutional accreditation and research funding. There is indeed a certain dichotomy between robust and highly developed public infrastructures and an evaluation system, which relies largely on the impact factor and other criteria imposed by the North, but which are supposed to represent a universal standard, that of the large mainstream journals. In effect, the national classification for researchers and the national funding agencies use the impact factor or the h-index to assign positions or resources. It is necessary to start changing our perspective. We used to represent our arrow as going from North to South: now let’s go from bottom to top, focusing on the quality of the research and its relevance to society. We can start to see that there are different possible production and circulation channels, which promote research and interaction with society, and can contribute to citizen science. It is this diversity of practices that research evaluation must address.

<figure><img src="image_37.png" /><figcaption>Fig. 10: To invert the path to defining research quality</figcaption></figure>

Source: Beigel, “Research Evaluation in the Southern Road to Open Science” and Beigel, “Circuits and fields. The multi-scale process of academic prestige-building”.

Eventually one of the main challenges in South America is to invert the path towards defining research quality autonomously in accordance with a public ecosystem with a robust infrastructure and highly developed resources, which supports diamond journals, as well as many alternative circuits, especially at the local level. The current
evaluation system pushes researchers to look up to mainstream criteria, and distances them from regional ecosystems of scientific production and communication. This is why we need to set up a regional CRIS, which already exists in Peru, Brazil and Argentina. Datasources are a fundamental issue to broaden up the landscape of scientific production under assessment, as is the need to really stimulate the multilingualism of local productions by giving them more visibility in international databases such as SciELO, Redalyc or BIBLAT.

In South America, there are many small-scale journals published by small research teams or universities. At the opposite end of the spectrum, mainstream bibliographic databases include very few Latin American journals. Latindex, SciELO, Redalyc and BIBLAT show it is possible to publish quality journals, with a level of excellence defined collectively in the academic community with a necessary balance between local standards and international criteria, a balance that must be pursued through horizontal and not vertical imposition.

## References

1. UNESCO, "UNESCO Recommendation on Open Science", 2021, https://unesdoc.unesco.org/ark:/48223/pf0000379949.

2. "euroCRIS | The international organisation for research information", accessed 16 March 2022, https://eurocris.org/.

3. Federation Of Finnish Learned Societies et al., "Helsinki Initiative on Multilingualism in Scholarly Communication", 2019, https://figshare.com/articles/Helsinki_Initiative_on_Multilingualism_in_Scholarly_Comunication/7887059.

4. "FOLEC - Foro Latinoamericano Sobre Evaluación Científica", CLACSO (blog), accessed 16 March 2022, https://www.clacso.org/en/folec/.

5. Serge Bauin et al., "Jussieu Call for Open science and bibliodiversity", 10 October 2017, https://jussieucall.org/jussieu-call/.

6. Fernanda Beigel et al., "OLIVA: una mirada transversal a la producción científica indexada en América Latina. Diversidad disciplinar, colaboración institucional y multilingüismo en SciELO y Redalyc" (SciELO Preprints, 19 July 2021), https://doi.org/10.1590/SciELOPreprints.2653.

7. "Biblat - Bibliografía latinoamericana", accessed 16 March 2022, https://biblat.unam.mx/fr.

8. "LA Referencia", accessed 16 March 2022, https://www.lareferenCia.info/es/.
# Open Science and Research Assessment. Trends and State of Play in Europe

Pastora Martínez Samper
Open University of Catalonia – UOC

I am very pleased to join the discussion on how important it is to change and reform research assessment in order to make open science a reality. However, let me also stress a point the other way around: open science is also the main driver for this new way of assessing researchers and research that is more comprehensive and that pushes us towards better science. The open science movement has brought us here today and we must acknowledge that.

I am going to give you an overview of the current situation of open science in research assessment from an institutional point of view, so I am going to focus on European universities. To do that I am going to tell you the results of the work we have been doing at the Expert Group on Open Science at the European University Association. We have been following the trends and the evolution of open science for about a decade, and our Expert Group has itself been evolving in following those different trends. In fact, we have just published our new vision and new key priorities for the years to come, and research assessment is key within that.

We have been following this evolution through different surveys, asking universities whether they were implementing open science, and these surveys have also evolved with time. We started in 2014, just asking publications about open access. Then when we incorporated FAIR data and in 2017 we included research assessment. What I am going to tell you about is a combination of the two last reports and surveys from the EUA¹, and it is worth noting that those surveys were answered by university leaders, or by responsible open science units, or departments at the universities, so it is not a bottom-up approach and it is not the way academics in universities think about things, but it is a top-down approach from the university leaders. The other thing I want to note is that all these results and all these surveys were done just before the European Commission started this discussion with the stakeholders, so it is going to be very interesting to see what happens after the debate we are currently having.
Let us start with open science. The good news is that open science is more and more present within the strategic priorities of European universities and in fact a huge majority of them have specific open science institutional policies or are developing one. However, when we go deep into the detail of how these open science policies are organised, there is an important difference between the different open science areas. For instance, open access to publications is given the highest level of importance and of implementation, whereas evaluation is not seen as important.

<figure><img src="image_38.png" /><figcaption>Fig. 11: Limited role of OS in university academic assessment approaches</figcaption></figure>

Source: European University Association et al., "Open Science in university approaches to academic assessment", December 2021, https://www.eua.eu/resources/publications/999:open-science-in-university-approaches-to-academic-assessment.html.

This is also reflected in the limited role of open science in the academic approaches that universities have. When we asked universities what open science elements were included in these academic assessments most of the respondents responded with elements that were pretty much related to open access of articles and books. Moreover, nearly one-half of respondents did not incorporate open science aspects in their academic assessments at all. When we asked specifically about research career assessment the results showed that open science and open access are given the least importance among the activities for research career assessment, whereas the most the two most important activities are, first, research publications and, second, attraction of research funding.
When we asked for the assessment methods for a research career we found two main aspects that are followed by most of the institutions. Assessment is therefore a combination of the metrics based on the number of publications and citations and the peer review assessment. The main metrics used are those of journal impact factor and h-index. This is the reality at European universities right now because there is a time lag between the open science strategy and open science being incorporated in the research assessment. Also, the main difficulty is that university leaders and administrators like to use this metrics-based approach for research assessment. When we go deep into the details of this main difficulty, most of them are related to practical implementations, so they have seen that incorporating this reform is a complex issue and they perceive that they have a lack of institutional capacity to do it. There are also concerns about increases in costs. There is also another very important concern which is that universities perceive there to be resistance to research reform from researchers. This is a key point. It is very important to bring researchers into the conversation. We cannot just have a top-down approach. We need things to be bottom-up as well in order to create and to move all the reforms we need.

Let me share with you the struggles we have, and I have myself as a vice president in a university, trying to change things and struggling with the balance between what we want to do and what we can actually achieve. There is always this dilemma. It is difficult to navigate through the context, the environment we are in, the lack of incentives, the lack of guidelines, the lack of different policies, the differences between countries and between disciplines, and the question of how we as academics are we going to do this without damaging researchers, particularly early-career researchers. However, when we ask about the future most of the respondents say that open science is going to be more and more present, including within academic assessment.

There are many organisations rethinking how we can improve academic assessment and I am going to give you some examples of some of them. I am going to start at an institutional level. I am going to give you examples of three universities that started working on these issues some years ago. I am going to start with Ghent University, which published that vision statement in 2016² looking towards a more comprehensive evaluation system. That vision was followed by a rethink and reshape of the evaluation system of academic staff, which is now seen more as a professional development and not just as a control and evaluation process. There is another very good example from Utrecht University³, where the medical school embraced an open science strategy, incorporating a more qualitative and descriptive portfolio of what is going to be taken into account in order to make assess researchers and groups.
I will also explain to you a little bit about we are doing at the Open University of Catalonia with our own Open Knowledge Action Plan that we approved in 2018⁴. We decided to incorporate research assessment as a way to move forward to open science. As a first step, we created a taskforce that did a lot of work to see whether we could accomplish what DORA⁵ advocated in not using journal impact factor for assessing researchers and research. After that, we signed it and we started implementing a more narrative curriculum in our internal evaluations, discarding the journal impact factor in our internal course.

There are also some very good initiatives at the national level. There is the Dutch Universities initiative that has incorporated a very interesting way of including asymmetry, for instance, in the assessment⁶. In Finland, some years ago, the Responsible Research network published a nice recommendation set stressing the need for change in assessment in order to incorporate the new ways of producing and communicating in science⁷. Also, universities in Norway came out with the evolution of the open science career assessment matrix⁸. The way they did it is very interesting, taking into account the different disciplines and a bottom-up approach, and they set up a toolbox giving us some indicators, both quantitative and qualitative, and included a reflection on this.

All those examples are also present in the work that DORA, the EUA and SPARC Europe have been doing⁹. These are the individual initiatives that have led us to be heard by the European Commission and now we are gaining momentum in convincing all the actors that these changes have to be made. It is very important that the new European Research Area Policy Agenda for the coming years has a specific action for reform for research assessment. The Scoping Report is also an important step¹⁰. We are now working all together in this process with the Commission as a facilitator in order to create the critical mass that we need. Let me end by saying that change is just ahead. We have room for creating more and more actions in order to move to a more comprehensive, more transparent and fairer way of assessing academic research for better science and I think the opportunity is here.

## To go a step further

"The European Commission, with a set of stakeholders, are now drafting a global agreement on the reform of research evaluation. They have also opened an expression of interest for those institutions that want to be part of this change. More than a hundred institutions have already volunteered, with the aim of ending up with
an agreement with specific actions to be taken, and hopefully also some support
measures which are needed."

"We know what we do not want, but we are also pretty much aware that open science is not going to be a universal solution for all. It should be adapted to the context, to the country, to the university, to the researchers and to the disciplines. At the UOC, we started small, and we are going step by step and with joint action learning between all the universities, between all the stakeholders. That is why these UNESCO recommendations are so important, because the global scale is very, very important for us in order to give this real momentum there. Open access of research publishing is easier because we have been discussing open access to papers for many years, and we know how to incorporate them, and we know how to open this access. We should now learn how to do the other things."

"Not everybody is aware of the limits of metrics measuring research, such as h-index, within the academic community. We should open this debate more, raise awareness and bring everybody in, in order to know all of these drawbacks."

References

1. European University Association et al., "Open Science in university approaches to academic assessment", December 2021, https://www.eua.eu/resources/publications/999:open-science-in-university-approaches-to-academic-assessment.html.

2. "Vision Statement for Evaluating Research at Ghent University", 8 November 2016, https://www.ugent.be/en/research/research-strategy/evaluation/research-evaluation-principles.pdf.

3. "Recognition and Rewards", Utrecht University, accessed 16 March 2022, https://www.uu.nl/en/research/open-science/tracks/recognition-and-rewards.

4. Universitat Oberta de Catalunya, "Open Knowledge Action Plan", May 2019, https://www.uoc.edu/portal/en/coneixement-obert/pla-accio/index.html.

5. American Society for Cell Biology, "Declaration on Research Assessment (DORA)", DORA, 2012, https://sfdora.org/.

6. VSNU et al., "Room for Everyone's Talent", November 2019, https://www.nwo.nl/en/position-paper-room-everyones-talent.

7. Working group for responsible evaluation of a researcher, "Good Practice in Researcher Evaluation: Recommendation for the Responsible Evaluation of a Researcher in Finland", 2020, https://doi.org/10.23847/islb.9789525995282.

8. Universities Norway, "NOR-CAM - A toolbox for recognition and rewards in academic careers", 1 June 2021, https://www.uhr.no/en/front-page-carousel/nor-cam-a-toolbox-for-recognition-and-rewards-in-academic-careers.5780.aspx; original
matrix: Working Group on Rewards under Open Science, "Evaluation of Research Careers fully acknowledging Open Science Practices Rewards, incentives and/or recognition for researchers practicing Open Science", July 2017, https://ec.europa.eu/research/openscience/pdf/os_rewards_wgreport_final.pdf.

9. Saenen, Bregt, Anna Hatch, Stephen Curry, Vanessa Proudman and Ashley Lakoduk, *Reimagining Academic Career Assessment: Stories of innovation and change*, January 2021, https://eua.eu/resources/publications/952:reimagining-academic-career-assessment-stories-of-innovation-and-change.html.

10. European Commission, "Towards a Reform of the Research Assessment System: Scoping Report" (LU: Publications Office of the European Union, 2021), https://data.europa.eu/doi/10.2777/707440.
# Making Room for Everyone's Talent

Kim Huijpen

Universities of The Netherlands

We had a Recognition & Rewards Festival in the Netherlands yesterday. This was a very inspiring event where all our colleagues involved in our national programme, and some of our international colleagues, I am glad to say, shared good practices. We inspired each other in a plenary programme and 17 workshops co-created with the people I work with every day. I am very happy that I could join you here in Paris and explain a little about how we like to make *Room for everyone's talent* in the Netherlands.

This is our ambition. We aim for a healthy and inspiring environment for our academic staff where all talents are valued – teaching, research, impact, patient care and good leadership in academia – and we hope to see this not only in the Netherlands but all over the world.

What I will be sharing today is why we think we need this change, what we want to change, how we think we will be able to achieve this change in the Netherlands and beyond, and also how you might contribute to this change in academic career assessment.

Why do we think change is needed? We can see in academia that we have a broad mission as universities, research institutes and university hospitals, but we only reward our academics for research output and publications, or even just a small set of indicators.
<figure><img src="image_39.png" /><figcaption>Fig. 12: Open Science Recognition and Rewards</figcaption></figure>

Source: Utrecht University, Open Science Recognition and Rewards, https://www.uu.nl/en/research/open-science/tracks/Recognition-and-rewards. Accessed 15 May 2022. CC BY-SA 4.0.

What we see is a mismatch, that Utrecht University translated in this beautiful visualisation comparing a calliper and a very simpler measuring instrument measuring only research as the primary focus of academic assessment. Our ambition is for this measurement instrument to be a little bit more advanced so that we can measure all aspects of what an academic, scholar or scientist does, and that is not only research. It is also teaching and impact. We also need good quality leaders in academia.

What do we want to change? There are five key aspects in our 'Room for everyone's talent' position paper of 2019:

What we strive for is that there will be more diverse career paths in academia, so that there is a career for you if you are good at doing research and publishing in the right journals, but also if you are good at teaching and at working together with organisations, society, and industry. We hope that we can form teams - and we think there already are teams - based on these different types of talent in academia, but we really want to recognise all these different talents.
We think that there should be a better balance between the individual and the team, so we strive for team spirit and teamwork. And I think we should focus more on the quality of the work. Because good scientific research increases scientific knowledge and contributes to solving societal challenges, and that is what we aim for. Of course, we want to stimulate open science. This is the Open Science European Conference, and we all know that if we want to stimulate open science we really should reward scientists who publish in open access journals, who work on research data management and share their data, software or materials. The fifth element of our approach is that we would like to stimulate good quality leadership in academia, and we think this is also key to make this change in academic career assessment.

How do we think we will be able to achieve this change? We set up a joint programme in the Netherlands and we work together and share good practices. We share experiments and are building an online community to make this possible, which we also already did in joint meetings. Every university, research institute and research funder has set up a recognition and rewards committee. We bring the chairs of these committees together in meetings, including at our Recognition & Rewards festivals, where we share good practices and co-create new work. Mutual learning is also key to our approach. The committees themselves translate our national position paper, 'Room for everyone's talents', into their own visions based on their own context and the missions of their organisations². Our approach is therefore both bottom-up and top-down because we think that it is important to bring researchers into the conversation. We cannot make this change if there is no dialogue with the academic staff themselves. We also, however, involve the important leaders in academia in the Netherlands. That is important, in order to have ambassadors of this change movement.

The guiding principles of our approach are as follows. We think that we are working on a culture change, which is a fundamental change of beliefs and not just a change in the rules of the game. What we need therefore is also patience to make this change. It will not happen overnight. We need this broad dialogue in academia, and we need to involve the academic community to make sure they are able to decide themselves how they are assessed. We need to share good practices. We are striving for a balance between giving room for ideas and then bringing them together, and to make this change, good leadership in academia is really, really necessary.

We need to work together, so we have to follow all the international initiatives (and there are a lot these days – I agree that we are gaining momentum) and share
our own experiences at conferences like this. We all have the same vision and
are striving for culture change, but we need diversity in approaches, and we need
communication and we need to share good practices to make this change possible.

I would like to give a little idea of the lessons we have learned so far in the Netherlands because we are not there yet. We really make sure that academics are involved. We are not advancing too quickly. It is very important that some people speed up the process, but it is also very important to be patient as well because we do not want new tools that lead to the same box-ticking exercise. We really want a change that is fundamental.

To end my talk, how can you get involved? I hope you will start a dialogue in your own context and listen to the concerns of your colleagues and peers. Perhaps you could even start a small experiment yourself, and I am very proud of a couple of PhD students who did this at Utrecht University. They just started at some point with a form they did not like, with a form that did not fit the way they thought about academic career assessment, and they changed that form. I think these small experiments are key to make this change happen.

In conclusion, we think we need a better balance in how we recognise and reward academics to help us achieve excellent education, research, impact and leadership, as well as the highest level of patient care in our university hospitals. However, we cannot change academic career assessment on our own. We need to work together in Europe and beyond to change the way we recognise and reward academics.

## To go a step further

"The Dutch national approach of research assessment is broader than the open science approach because it also includes teaching, leadership and societal relevance or societal impact. It is very important because an academic, a scholar, is more than a researcher. Teaching is key to the mission of universities, so it is fundamental to invest in the quality of teachers and to give them room to grow in academia and have a career path as well."

"In the Netherlands, some professors and even ECRS have written open letters to return to the old assessment system. We see this as the academic community sharing concerns, so I think we should really involve the academic community and start a dialogue. We did this by inviting one of these people, the first author of one of these letters at a round table at our Recognition & Rewards Festival yesterday. And
we had a really, really good dialogue with him and with the board members involved
in our initiative. I really understand that a lot of researchers, a lot of scientists, are
concerned about their careers. And we are experimenting. We do not know what it
will look like in a few years. So I think we should really take these concerns seriously,
start a dialogue, keep this dialogue going and make sure that we all do that on a
national level and on the level of a university, faculty or research unit."

References

1. VSNU et al., "Room for Everyone's Talent", November 2019, https://www.nwo.nl/en/position-paper-room-everyones-talent.

2. See examples at Leiden University (https://www.universiteitleiden.nl/en/news/2021/01/academia-in-motion-a-different-form-of-recognition-and-reward), Maastricht University (https://www.maastrichtuniversity.nl/about-um/recognition-rewards), Tilburg University (https://www.tilburguniversity.edu/current/news/more-news/program-recognition-and-rewards), Rotterdam University (https://www.eur.nl/en/about-eur/vision/recognition-rewards/what-recognition-rewards/eur-programme) or Utrecht University (https://www.uu.nl/en/research/open-science/tracks/recognition-and-rewards).
<figure><img src="image_40.png" /></figure>
Open Science, Research
Assessment and HSS
(Humanities and Social Sciences)

Emanuel Kulczycki
Adam Mickiewicz University in Poznań

In my talk I aim to show how to effectively address one of the key challenges of open science and research evaluation, which is multilingualism. The perspective of social sciences and the humanities helps me to show that supporting multilingualism is the foundation of research impact.

Almost half of Europeans are not able to speak any foreign language well enough to hold a conversation. Only a little more than one-third are able to do so in English, the most widely spoken foreign language. Thus, if research is communicated exclusively in English, it is inherently limited. This is apparent during the COVID-19 pandemic in Europe and globally. We observe a widespread need for scholarly communication, not only between researchers, but to enable research to reach decision-makers, professionals, and citizens. The problem is that we have made the publications open, but we see that our societies are not fully able to use them. This can only be possible if research is communicated in multiple languages, including those used in speech and writing locally.

In the ongoing reform of the research assessment system, the call for multilingualism is the most notable omission. Making science actually open requires promoting diverse, multilingual and relevant research accessible for different audiences. Machine translation is not a substitute for publishing research results in multiple languages to different audiences. This is primarily because major corporations and publishers target languages with the largest number of users and translate mostly from English to other languages. Languages with fewer users are too often overlooked and thus become invisible. Exactly the same is true for publications in languages other than English in the largest citation indexes. Therefore, open science and research evaluation must support researchers in publishing research in multiple languages.

Not all local publications need to be translated into English and not all publications in English need to be translated into local languages. The call for multilingualism is
not a call to prioritise local languages over English. It is a call to recognise that the quality of research does not depend on whether the research is communicated in English or not. In the social sciences and the humanities, we study societies and analyse them using their own words and their own concepts. Thus, local languages are needed to express the entire culture and historical heritage. Moreover, in many of the so-called hard sciences, like physics or chemistry, research can be expressed in a fewer number of words than in history or sociology because a lot of work is done by figures, charts, and formulas. This is not the situation of the humanities, where the linguistic diversity plays a key role.

I build my talk around scholarly communication through the lenses of researchers from the social sciences and the humanities. Talking about open science, we regularly reduce it to open access to publications and forget about the researchers who are the actual creators of science and subjects of open science. Therefore, I focus on various ways of publishing research results and highlight that communicating through scholarly books, publications and local journals is crucial for researchers from the social sciences and the humanities. Nonetheless, researchers publishing books and articles in local languages face different challenges than researchers from the so-called STEM fields, natural and technical sciences, among others. I argue that when we reorient our focus from publications to researchers who publish those publications, we will spot that multilingualism and language bias are crucial challenges for open science and research assessment. That is a key message of the Helsinki Initiative on Multilingualism in Scholarly Communication¹, which aims to make language a nonissue in assessment. I am going to present an evidence-based picture of publication patterns of researchers in Europe. I draw on recent studies of the European Network for Research Evaluation in the Social Sciences and Humanities.

Evidence from eight European countries² shows that the Web of Science database covers only a small part of all peer-reviewed publications in SSH. Scholarly book publications have been an important way of communicating research, but they are invisible in the Web of Science or Scopus databases. For instance, only 15% of all publications of Polish scholars from SSH are indexed in Web of Science and only just over 50% of Danish scholars. Fortunately, for over a decade, various European countries have been using national current research information systems, thanks to which we can obtain a good picture of the whole publishing landscape in academia.

There was a study of the percentage of peer-reviewed publications published in English, local languages and other languages in the social sciences and the humanities in eight European countries. In this study, we have taken into account all publication types and the results are not limited to publications indexed in the largest
citation databases. In the Nordic countries most publications are in English, but in the Central and Eastern European countries the local language is dominant. The size of the country, the diversity of disciplines and the size of the market for academic publications are insufficient explanatory factors. Historical and cultural factors still play a major role.

<figure><img src="image_41.png" /><figcaption>Fig. 13: Percentage of peer reviewed publications in English, local language(s) and other languages in the social sciences and humanities in 2014</figcaption></figure>

Source: Emanuel Kulczycki et al., "Publication Patterns in the Social Sciences and Humanities: Evidence from Eight European Countries", Scientometrics 116, no 1 (July 2018): 463-86, https://doi.org/10.1007/s11192-018-2711-0.

This picture focuses on scholarly communication through the lens of publications. One can say that researchers from Poland or the Czech Republic are less internationalised, that is they publish less in English, than their colleagues from Denmark or Norway. However, as I said earlier, we need to reverse this focus and look at scholarly communication from the perspective of researchers.
<figure><img src="image_42.png" /><figcaption>Fig. 14: Language patterns of article publishing on the researcher level across countries</figcaption></figure>

Source: Emanuel Kulczycki et al., "Multilingual Publishing in the Social Sciences and Humanities: A Seven country European Study", Journal of the Association for Information Science and Technology 71, n° 11 (November 2020): 1371-85, https://doi.org/10.1002/asi.24336.

We turn now to the results of the study focused on multilingual publishing of journal articles in seven European countries. This picture is built around individual researchers, not their publications. In a three-year period, over a half of Polish, Czech and Slovenian researchers published in at least two languages. Some of them in this period published in three, four and more languages. As other studies show, multilingual publishing includes not only articles but also books.

Evidence shows that the Web of Science and Scopus databases cover only one-fourth of all peer-reviewed journal articles published by researchers from these analysed countries, and only a small percent of the articles published in local languages. In this way, intended or unintended language priorities in assessment may lead to a systemic undervaluation of SSH research and researchers. The clearest example of the everyday language bias comes from the manuscript peer review, where it is frequently reported that reviewers judge research based on the quality of non-native English speakers' writing instead of the content of the manuscript. In this way, the excellence of research is too often equated with English-language publications, especially those published in journals having an impact factor.
What can we do to improve this situation? Making the link between multilingualism and assessment clear has been one of the main goals of the Helsinki Initiative on Multilingualism in Scholarly Communication. The initiative promotes language diversity in research assessment, evaluation, and funding systems. Ideally, language is a nonissue in assessment and researchers should be recognised and rewarded according to their results and the impact of their research. Language biases are produced in evaluations based on research metrics, as well as in evaluations based on expert assessment. Thus, all actors involved in research assessment should make sure that in the process of expert-based evaluation high-quality research is valued regardless of the publishing language or publication channel. Moreover, when metrics-based systems are utilised, journal and book publications in all languages should be adequately taken into account. The selection of evaluators and their language skill should be a relevant concern in the assessment.

Many decades ago, we gave away, as researchers, the publishing of journals and books to small commercial publishers. This was a well-functioning relationship for a long time, but today 75% of European spending on scientific journals goes to the big five publishers. On top of this we can see new mega-publishers emerging from the new author payment model for open access. These mega-publishers treat publishing research mostly as a means of increasing profits. By doing this, they have been demolishing the idea of open science, making it just another way to increase their own profits. In such a business model, multilingualism is just an obstacle because it is easier to publish and sustain infrastructure only in English. The losers are both scientists and citizens. Therefore, reclaiming communication channels and owning them by the academic community and learned societies is a crucial step in maintaining and achieving the goals of open science. Only in this way will we be able to fully open up the publications and the metadata used in research assessment.

To achieve a fully open science, we must, among others, make sure that national journals and book publishers are safeguarded in the transition to open access. Policymakers and the academic community must make sure that not-for-profit journals and book publishers have both sufficient resources and the support needed to maintain high standards of quality control and research integrity. Taking into account all the presented arguments, I want to highlight that multilingualism is integral to accessibility and should be part of the European research assessment reform.
## To go a step further

"It is crucial to describe, discuss and talk in various languages if we want to reach different audiences, to show them that they are analysed and understood in their own language. English is not enough."

"Our aim is not to produce the Tower of Babel. The aim is to recognise that researchers already publish in many languages. The aim of the Helsinki Initiative is not to force researchers to learn new languages and publish in more languages – of course it would be great if they did so – but to acknowledge in the current existing systems that researchers actually publish in many languages. For example, when we analysed data from the current Polish national research information system, we saw that the humanists over a six year period published in seven or eight different languages. They are not translations. They publish in so many languages because they analyse various classical literature and ancient history texts. The idea therefore is not to push to publish in many languages and translate evidence in various languages but to use, for example, machine translation to facilitate reading."

## References

1. Federation Of Finnish Learned Societies et al., "Helsinki Initiative on Multilingualism in Scholarly Communication", 2019, https://figshare.com/articles/Helsinki_Initiative_on_Multilingualism_in_Scholarly_Communication/7887059.

2. Emanuel Kulczycki et al., "Publication Patterns in the Social Sciences and Humanities: Evidence from Eight European Countries", *Scientometrics* 116, n° 1 (July 2018): 463-86, https://doi.org/10.1007/s11192-018-2711-0.
Round Table –
Research Assessment

Roundtable participants:

* Jean-Éric Paquet, Director-General of DG Research and Innovation, European Commission

• Maria Leptin, President of the European Research Council - ERC

* Marc Schiltz, President of Science Europe and CEO of the Luxembourg National Research Fund - FNR

• Michael Murphy, President of the European University Association - EUA

* Bert Overlaet, on behalf of the President of the League of European Research Universities - LERU

* Thierry Coulhon, President of the High Council for Evaluation of Research and Higher Education - Hcéres

Animator: Paul de Brem (scientific journalist)

Animator: A few months ago, open access was made a legally mandated requirement for all projects funded by Horizon Europe¹. Three months ago, the European Commission published a report called ‘Towards a reform of the research assessment system²’. One month later, the European Commission released a Call for interest³ in order to create a coalition on reforming research assessment. Jean-Éric Paquet, how many signatories did you get?

**Jean-Éric Paquet:** It is still ongoing, but we are way beyond 160 in a matter of a few weeks. I expect that to mushroom now and that we will have many more. We hope to rally as many actors as possible in Europe around that call, and across a spectrum of organisations. We want researchers and the organisations that represent them, universities, evaluation actors in the national systems research centres and research funders, to be part of it. We need this participation across the spectrum, and we already have a wide range of actors in the 160 plus signatories so far, so this is really looking good. We have a few non-EU countries already represented, but we now largely have all EU member states. We do not have all of them yet, but I think we are above 22 out of 27. I have no doubt we will get there.
There is amazing momentum behind open science. The assessment of research, the assessment of research organisations and the assessment of researchers themselves is very much at the heart of a systemic change in open science practices, which is why it is so important. Beyond that, we have seen in the pandemic management that open science is an absolute must. The impact that open science has had on scientists around the world, particularly in Europe, in making a difference has been amazing. The Commission, together with the European Molecular Biology Laboratory and other organisations, created a data platform through which thousands of researchers shared their data and viral sequences. This accelerated the knowledge sharing around COVID-19. Also, publications behind paywalls were made available in immediate open access. With open science the world has moved remarkably fast in dealing with the pandemic, and the vaccine is one example. It is therefore a necessity, but it is also a process which is decades old. There was a lot of frustration at the lack of action, as open science is a systemic effort with many different components which largely need to move together to be relevant and impactful, which explains why it was so difficult. The European Commission can have an impact in terms of policy frameworks and funding, but we cannot do it in isolation. Today's momentum really started with coAlition S⁴ about open access to publications, which created some momentum, including for research assessment

It started with ministers agreeing that the European Research Area needs a new momentum. A key component for ministers is indeed open science and, within that, they identified research assessment as a critical component. It is very clear that when you look at research assessment this is not just about looking at and grading individual scientists or labs or institutions. It is about fundamentally looking differently at how we are producing science and how we are organising ourselves to do so. The Commission did a rather deep dive into the issue in the course of 2021 and produced a report⁵ at the end of last year setting out the key elements of a process which could lead us collectively to really change the way research is assessed. That was then the basis for a call we launched on 16 December 2021, which now has real momentum.

The idea of working together is to produce a coalition agreement in order to make proposals for concrete reforms of research assessment, and the Paris Call is a direct contribution to it. It calls for organisations to come together and commit to research assessment reform, and to helping us to produce this coalition agreement. We will use last year's report as a strong starting point, and we are now working with Science Europe and the European University Association to start putting a text together. The Paris Call will be a key input into it. We will
of course make this work available as it progresses, and it will be a work of co-creation with many actors participating. I hope we will have it by the summer of 2022, and we will then have a very strong basis to start, act and deliver.

This coalition should certainly be open to participants beyond Europe, which would be fully aligned with the UNESCO open science recommendation⁶ released a few months ago. The hard work starts with like-minded European players, but we are open to see key players from around the world join. It makes sense because science is global. There is no doubt a need to have players from beyond Europe joining. I think the work will be strong enough to attract big players from outside of Europe, especially given the concrete and practical step-by-step delivery process.

There will no doubt be roadmap elements. This is not just a declaration of principle. We need to move beyond that to delivery, which is challenging because it is very broad as an effort. It is disruptive and it will change practices, habits, and the positions of actors. It is not something which is neutral at all. It is a deep disruption for the better, but it requires us to create a large coalition and genuine ownership with the actors of that coalition, and that we then have a real practical way forward with well-planned steps and strong momentum within a reasonable timeframe. With Horizon Europe we will play our part by changing the way we assess proposals. We will move away from the use of journal impact factors in particular and look much more at the content of research.

**Animator:** Maria Leptin, the essence of your organisation, the ERC⁷, is to select excellent researchers who will get fantastic grants over a period of many years, and they will be able to create their own teams and pursue their own scientific ideas. I guess that every scientist in the world would like to have an ERC grant and I guess that you must be challenged as an organisation as to how to assess these candidates and the work they have done?

**Maria Leptin:** It is indeed a challenge. First of all, a logistics one, because there are thousands of applications coming in each year and only between 10% and 15% of them can be funded, so the selection is very tough. The ERC relies on top experts in all fields of academic thought and research, so we need a large number of experts. The challenge is of course how to identify the best. We do rely on experts who look at the science. They spend a lot of time. They read the grants in detail. We get external assessments as well, expert views on details. They meet to discuss them, and they meet again to interview the short listed candidates. It is a huge amount of work and a huge amount of intellectual input.
The ERC really looks at the content and not so much at metrics. The ERC has been a signatory of DORA⁸ since last year. We have completely banned the use of impact factors. Their use has been discouraged for some time. The committees are told not to look at other metrics, such as whether researchers work in glamorous institutions or whether they have a high h-index. They are really encouraged to look at the essence of the work. There are a number of things that matter that have not actually come up in some of these discussions. For example, not only the impact factor of where you publish but the overall output can vary a lot, and it can vary a lot depending on the location where you are. If you are in a very supportive huge centre of research, it is easier for you to be productive and to train good people, because the good people flock to these places, than if you are in a smaller place. We urge our panels to pay attention to that and they do.

The Scientific Council, which has 22 researchers from throughout Europe and which represents all fields, can monitor the way panels assess these researchers by sitting in on the panels and listening. They are not allowed to say anything. They are not allowed to influence. They sit in, monitor, and get feedback. They thereby collect evidence from the panels and then report it back for discussion and review, so we constantly self-criticise and get feedback. Some people may fall into traps, but reminders are given to try to prevent that. We make use of narrative CVs. Researchers are allowed to choose individual papers they want to highlight and to tell the panels and the referees what to use. But there is still room for reform. As a researcher myself I would never say what I do is perfect and ready. It never is. It has to evolve.

There is still a lot to be done in an organisation such as ours for research assessment. When we talked about this before, we were talking about open access. We are now talking about research assessment. They are very different. What is happening right now, and I really welcome this, is that we are getting feedback about what we are not getting right and where we can do things better. We are about to set up a taskforce within the ERC Scientific Council to also look into this. When it comes to open access, which I fully subscribe to, the ERC demands publication in open access. In other words, any work funded by the ERC has to be published in open access venues. However, the entire community still needs to look for additional solutions.

I welcomed yesterday the highlighting of Diamond publication. Diamond Open Access is defined as a method where the researcher does not pay to publish and the reader does not pay to read, which means that somebody else has to pay. There are very good examples of that, but they are restricted to researchers who are funded through particular venues. What about all the other papers out there? If we want global open access, we need to find a way of doing that. These are big issues that remain to
be solved, but it is good to see an impetus going in that direction. I do not know
how things will pan out with what we and the community come up with in terms
of recommendations for research assessment, but I am sure there will be similar
logistical issues and possibly also financial issues.

If the Scientific Council decides so, it would be possible for the ERC to, for instance, response positively to the Paris Call and join the coalition, as we already signed up to DORA. We have created a taskforce, that is not fully formed yet, but we met in preparation of this meeting. We will look into what the Scientific Council thinks, with feedback from the panels, about what is needed for and works for the ERC. Once we have formed a view of the best way forward, we hope we will lead in implementing the reforms that will come from the consultation with the community.

**Animator:** What about open science and partnership between the academic and private sectors? In other words, how to reconcile open science with the incentive to file patents by certain research institutes?

**Maria Leptin:** I would simply say that question is not for the ERC to answer. Researchers at the ERC are free to publish whenever they like and whatever they like. They are also encouraged to consider patenting and bringing their research to the next level of application. I think human curiosity is the driving force of discovery and invention, so it does not have to go anywhere. However, researchers are encouraged to think about that and are helped to do that. There are specific grants that help people to get patents. Then there is the great transition to the EIC⁹. Proof of concept is within the ERC, but then there is a different agency, the European Innovation Council, that does the next step.

**Animator:** Marc Schiltz, Science Europe¹⁰ is an association that brings together many funding agencies. You have 38 member organisations from 29 countries. You told me that when it comes to reforming the assessment system, we know what we do not want anymore, but more difficult you say is what we want to replace it with, because the question is so complex and because we are trying to assess people engaged in so many various activities. How are you going to compare all these people?

**Marc Schiltz:** I think that we should not, and we will not replace a narrow, simplistic and monolithic evaluation model with another one. The question is not so much what the new evaluation system will be. I do not think we should go for a unique new evaluation system. I think that is the wrong question. Here I can make a reference to open access. With cOAlition S, one of the reasons we started on that was to liberate
the scientific community from a monolithic publication model, but we were not saying
which one should be the new one. However, we needed to be liberated from those
chains and, as we heard yesterday, there is now innovation in the publication industry.
With research assessment, it is the same. We need to be clear on what we do not
want and what we wish to abandon.

Where we can agree on a future assessment system, and this is what is written in the Paris Call, is that we should recognise that researchers' activities, for both individuals and institutions, are much broader than what is measured through some current narrow metrics. That is the first principle. For example, in the COVID-19 crisis many researchers have advised governments and many scientists have been on TV and on radio. This public outreach and scientific communication have suddenly become extremely important. That is just one example of taking a broader view of what researchers do and what should be valued.

What we also need is what is called research on research. Because I think if we are
going to introduce new ways of assessing, we should also give ourselves the means
to assess these new methods and these new approaches because we do not want to
introduce new biases. We have to be very careful. We know that our current system is
heavily biased, but we need to be careful that evaluation systems or methods involve
critical self-reflection. My funding agency (the Luxembourg National Research Fund)
has introduced narrative CVs, as have others, but we are now collecting feedback
on that. We need to assess the impact and effect of narrative CVs to make sure that
we are not making things worse. We must experiment, and I do not think there will
be a one-size-fits-all evaluation method. Different institutions will adapt it to their
needs. We can be creative here and it is the community itself which should mainly
define these things. But the question is what we value in science, and it is more than
just publication. We have to take care of the science system itself by being open, by
sharing results and by caring for each other, including actions like training, mentoring
and reviewing. The individual institutions know more or less what they want, but we
have to get rid of these chains which we have put ourselves into. The ERC is a very
good model of not looking at simplistic metrics and its participation in the coalition
would be welcomed.

**Jean-Éric Paquet:** One element to note is the diversity of research activity, which Marc touched on. What we really need to value more and reward more is the work which scientists do on data, which also includes software, and then making it available to the community. This is never looked into. It is very, very time consuming. It requires a lot of resources and if we do not reward it and assess it this will never bring us to where open science wants it to be.
**Animator:** In your opinion, can you lock in the increased openness in scholarly communications that we have seen during the pandemic? Is there the political will in your opinion?

**Marc Schiltz:** There certainly is from the funders' side. The planned policy is to make everything immediately open access. Publishers – even legacy publishers – have agreed that this is the case for everything related to COVID-19. Then we can arguably ask why just COVID-19? Is cancer research not as important? Why should cancer research be locked up behind a paywall for 12 months? I think publishers have in a sense admitted their defeat because what holds for one topic necessarily holds for the other. We cannot have an artificial hierarchy on what deserves to be immediately open access and what not. So we will see what happens, but certainly the funders in coAlition S have always supported immediate open access without embargo for everything – not just COVID-19. So we will stick to that.

**Jean-Éric Paquet:** The funders do not allow anymore publications not to be immediately accessible. Under Horizon Europe, any research funded will need to be in open access at publication.

**Animator:** Michael Murphy, do you have the feeling that universities are ready to think and create this research assessment system that we are looking for, a flexible one of course, not a monolithic one? Do you have the feeling that they are eager to have such a system reformed?

**Michael Murphy:** I do, because we have lots of evidence to justify reform. The EUA and specialist groupings of universities, most of which are in the EUA as well, have been working on the Open Science agenda for well over a decade and we have been monitoring the evolution of the application of open science within the sector. Visiting here for the past day or so one can see the consensus among the breadth of society represented here and the enthusiasm to move on this agenda. Open Science is a moon-shot, something we do not because it is easy, but because it is hard, and it is something that our species needs to do.

Society as a whole is embracing on sustainability. The UN Sustainable Development Goals¹¹ effectively encapsulate our societal and sectoral objectives and open science is a critical tool for achieving the SDGs.

Adoption of open science principles and practices should be one of the sector's contributions to the UN SDGs by 2030. I think one of our failures to date is that we are afraid to set targets. The targets have to be realistic and that is why I am looking
at 2030. Yes, the universities are committed, and we have strategies and in most
institutions we have policies, but the implementation gap is what we now really have
to focus on.

The question now for us is how do we close the implementation gap between our
aspirations and what we need to do? We are working as a sector, and with all the
societal partners who have influence on our sector, to adopt collective approaches
to reform academic assessment as an enabler of Open Science. The Paris Call is a
wonderful example of collaboration. We have a tough job ahead of us because of the
complexity of the matter. It is critical that in our work with the Commission, a key
partner, we arrive at agreed implementation principles among all the actors and that
we are realistic in the timeframes that we adopt.

In addition to accommodating complexity, we need to accommodate diversity.
The needs of people in various disciplines are not the same regarding research
assessment. Whatever frameworks we adopt, we have to be sensitive to that diversity.
If we are to be fair in rewarding people, we have to make sure that our methodologies
exhibit equity, which will be very difficult because the metrics in STEM can be very
different from those in the humanities. But many of the measures that we will use are
common to both. We also have to recognise that countries differ in terms of available
resources, regulatory frameworks and opportunities to exhibit leadership.

EUA released a refreshed open science agenda a few days ago covering 2022 to 2025¹². It sets out our open science strategic priorities and addresses assessment reform as well. It begins with an endorsement of UNESCO's open science definition. This is quite important as the open science agenda spans the globe in its application. The challenges that we are trying to face are global and require that we engage talent everywhere in the world to combat them. Having a body such as UNESCO aligned with what we are trying to achieve in Europe is really, really important. Some countries and some institutions have been braver than others in the steps that they have taken. We as global bodies, or at least pan-continental bodies, have a responsibility to try to make sure that the gap between the leaders and followers does not become a disincentive to the latter to keep on pushing. It is critical that we are all in this together.

** MAKER:** Bert Overlaet, you were the lead author of a paper published recently¹³.
What does it say?

Bert Overlaet: LERU has been dealing with research assessment for some time
and trying to move forward. We exchanged all these experiments and novel ideas
we had about that and turned it into a sort of new framework that is described in
the paper that was launched last week. It is a way of trying to create a picture of
the future of the assessment of researchers. We are not complaining about what
we did before. If we want to go forward then we need to have a multifaceted and
much broader perspective in assessment, which means that we must go away from
the single-minded and very narrow measurements often used today. That is one
element of the paper, and it describes the complexity and the variety of the duties of
a researcher today in a university.

The ERC is here today. Another great European institution is the European Institute of Innovation and Technology¹⁴, where you have these networks of sometimes 400 partners trying to move research forward in the European context. We need researchers who are able to work inside these kinds of coalitions. Networking is becoming so important. In the COVID-19 era it has become clear that leadership, collaboration, and innovation are topics that are coming much more to the foreground. It is not sufficient anymore to be a good researcher. You are also expected to be a good teacher in universities because we do not only produce patents and papers. Most importantly we produce people. We produce young people with research training, who are well trained to play a role in society. This diversity is therefore described in the paper along with the experiments we are doing in LERU.

Some universities have used narrative CVs for as long as 15 years, but we are still working on them, making changes and trying to improve the format because every format has its advantages and disadvantages, and it is difficult to strike the right balance. The aim is to be precise, efficient, and also nonexclusive. If narrative CVs are too long, then there are complaints that they take too much time to fill in. If they are short and concise then the risk is that those with a typical profile will be more easily remarked on by the panels than those with an atypical profile. At the heart of our paper and our endeavour is that we want to recognise and reward all the different contributions people now make in the research area, which is much more than just publishing. We want people who stimulate progress in their field of work, and they can contribute to that in so many different ways.

** MAKER: How do we concretely get around university rankings driving us back to a monoculture?**

**Bert Overlaet:** Ghent University was mentioned earlier. Ghent is the closest competitor and colleague of the University of Leuven. The difference between the two is that at some point in time Ghent decided that the quantitative criteria for funding used by the Flemish government should translate into the assessment of the researchers themselves and they got a bonus for each publication they made because
they got bonuses from the government. This is the trickle-down principle that was
used there. This is something that Leuven has never done. There is a Chinese wall
between the funding of the university and the assessment of the researchers. We do
not want the same criteria to apply. At some point, Ghent reformed and returned to
the old situation because they came to see the trickle-down effect as destructive for
the quality of science, the culture and the organisation. I am sure that if you have a
good assessment process, you will achieve a collective form of excellence that will
lead to good funding, but I think it is a devastating strategy to couple them directly,
which has been shown in some places.

**Michael Murphy:** Rankings are like death and incomes taxes. They are inevitable. It is regrettable that those with the most influence are copper-fastening the importance of metrics that we are now trying to abandon in this reform agenda. Universities as a sector must encourage the evolution of a ranking system or ranking systems that deliver support to open science and our broader agenda. One that exists, that goes back to the last French Presidency, funded by the EU, the U-Multirank<sup>15</sup>, is I think the closest to that objective, though it too needs to evolve. It behaves us to work to support and evolve a methodology that helps us attain our open science agenda.

**Marc Schiltz:** To the rankings, we also have to be careful that we are not ending up again in a situation, as with open science, that we give our very precious data away to these entities that establish these rankings. That's what is currently happening: we give all this data over when it would be much better utilised for our own research on research purposes.

**Bert Overlaet:** There is one dimension I would like to mention as well, which is elaborated on in the paper and was already mentioned by Maria here. It is contextualisation. Today rankings, individual assessments of researchers and metrics are happening in a vacuum. If you are in a very good environment, it is easier to get publications than when you are in another institution. We have avoided contextualisation so far and one of the elements that we would like to focus on more in our assessment of research in the future is how to deal with this contextualisation without getting in a situation where you cannot decide anymore. We still have to make decisions about hiring and promotions, but how can we incorporate this context more than we do today?

**Animator:** Thierry Coulhon, you are President of the High Council for Evaluation of Research and Higher Education (Hcéres<sup>16</sup>). It is an independent public body in France which is in charge of assessing institutions, universities, national
research bodies and so forth, but you do not evaluate people. How do the open science principles apply to an organisation such as yours?

**Thierry Coulhon:** The world is changing as far as research is concerned. There is a movement at the European level. There is a French movement as well and we are fully on board. How are we concerned? As you said, we do not assess only research. We also assess institutions and curricula. This means that we can ask the institutions whether they have an open access policy, whether they have an editorial policy, whether they have repositories and so forth. Also, as you said, we do not evaluate individuals and we do not evaluate research projects. Nevertheless, I think that we have to be coherent with the evolutions of the funding agencies and the institutions that hire and promote.

How do we work? We evaluate *ex-post*. That is the first thing. We evaluate every entity on a five-year basis, with the country divided into five geographical waves. We set up guidelines for each category of entity, namely the institutions, universities and national research bodies. There are guidelines for curricula and for research teams. On the basis of these guidelines, we ask questions about the open science strategy of the entity. Then the entities write a self-evaluation report. Then we set up a committee and the committee writes an independent report.

How do we comply with the open science movement? First, we hunt down impact factors in order to replace them with an assessment of the quality of journals and productions. This is not the most difficult part. Also, we tend to cover all the aspects of research activity. Of course, we do ask for research outputs, but all the kinds and categories of research outputs. We also ask for information regarding connection with society, outreach and so forth. We are also keen to take care of the diversity of disciplines. For each set of disciplines, we have a guide of products and activities of research outputs. These guides are elaborated by the communities, and they heavily depend on the discipline. Finally, we tend to move from quantitative data to qualitative analysis. Of course, we ask for quantitative data, on which we build the peer review. We recently started this for the current wave. We ask research teams for a portfolio of research outputs they want to set forward and the experts read them and assess them.

Let me insist, finally, on two things. We also have the mission of taking care, through a specific office, of the issue of science integrity. It is fair to say that France has been late on this issue and that we are trying to catch up. We host the French Office for Science Integrity¹⁷ and it has a scientific council. They try to disseminate good practices and to animate the community of scientific integrity throughout the various
institutions. This is one thing. On the other hand, I wanted to emphasise the fact that we not forego quantitative analysis at the proper level and the proper level is the integrated level. In the High Council we have a Science and Technology Observatory¹⁸ and they recently published a paper on the French scientific position¹⁹. At this level you can certainly count publications and study scientific profiles and so forth.

Things have to become collective in order to make the reform of the research assessment system happen. All French actors have been coordinated by the French Ministry, which has done a terrific job on this issue. We elaborated the Paris Call together. That is one thing. Second, at the European level there is a strong push by the Commission, and we discussed things with the people from the Commission. All this is happening at the same time. What I meant is that we have to be coherent. The High Council does not have the most difficult part. The most difficult part is when you hire and promote and there you should not take the easy solution of just counting and looking at easy metrics. However, of course when we do evaluate the ones who hire and promote we should be coherent with that. That is what I meant by the idea of moving all together, albeit maybe not at the same pace. I am fairly optimistic.

We are at the beginning of a landslide. I must say it is a matter of age also. I spent most of my career in another world when one used to count the h-index and the number of publications and so forth without asking questions, and I must say that a few years back I was still sceptical about some things. The model of diffusion of scientific knowledge would change, but one did not know when and how. Also, yes, the assessment system could change, and it did change. One can observe this at the French level and at the European level. It is now inevitable. It is being done.

**Animator:** Your job is to evaluate research infrastructures. They are key players in implementing open science concretely. These research infrastructures are key players. Could you say a few words about your ambition concerning the evaluation of research infrastructures?

**Thierry Coulhon:** Let me emphasise the fact that it is relatively new for us. We were not in charge of evaluating research infrastructures, and the recent law on research gives us this mission, so we are working on it. We already evaluated research infrastructures at the European level inside a consortium called ERIC²⁰. I therefore will not be able to give many details since we are currently working on it. However, it is of course extremely important because research infrastructures collect a lot of data and the question of openness of data is crucial at this point. It is very much connected to what we are talking about today, but more is to be seen.
** Animator:** A question from Tunisia for Mr Paquet: 'We are from a data-engineering and semantics research unit at the University of Sfax in Tunisia. We have an idea of innovating research assessment through the use of open knowledge graphs. Can we contribute ideas to the coalition now'?

**Jean-Éric Paquet:** Tunisia is a research and innovation country and system which is very closely interacting with us. They are absolutely welcome, and any instrument is certainly worth looking into. Yes, please look up the call for expression of interest, which is on the website of the European Commission, and then connect with it to the effort.

**Animator:** People in the comments provide support to the Open Research Europe²¹ initiative. Could you explain this initiative?

**Jean-Éric Paquet:** This is essentially a publishing platform which has been put in place by the European Commission to help researchers under Horizon 2020 and Horizon Europe comply with the obligation to make their publications immediately open access. We do not prescribe those researchers where they should publish - and some will still want to publish with legacy publishers, which have great journals but not open access yet. And so, these researchers could still publish there but they also need to make the article available in open access. So Open Research Europe, as an open access publishing platform, is a facilitation. Moreover, I think there is no doubt also that the ambition to use our publishing platform is for all its other aspects of open science. As we move forward, it might also become a useful place to help with changes in research assessment.

**Animator:** We have seen yesterday that publication bias, that is non-publication of negative results, is a longstanding issue in science. How could researcher assessment tackle it?

**Bert Overlaet:** One idea that we already apply in some of our universities is that we ask candidates for promotion or for hiring to say something about one of their failures, because maybe you tried something, and it failed. What did you learn from that and how does it affect your future project in science? How does it affect you as a scientist? This is the innovation that I mentioned earlier that is missing. We are all looking at the past and the outputs that have been produced in the past, and we can make that broader and anything you want, but we also have to look at what the potential of the person is for the future. That is what we base our decisions upon.
**Maria Leptin:** I think it is a wonderful initiative and all these kinds of different assessments are great, but negative results are not failures. They are work done with a clear outcome, and if it is important I think that can be rewarded. It should be published and there are repositories. Of course, it seldom goes into a high-visibility journal, but it should be put into the repositories and cited. That is perfectly clear.

**Michael Murphy:** In this regard there is a responsibility on university leaders to make sure that, within their institutions, there has been pre-registration of all studies and prior publication of the protocols. Ethics committees within universities too must follow up on all studies that have been examined for ethical compliance to ensure that publication has taken place. Publication of all results is an ethical requirement in the conduct of research.

**Animator:** Mr Schiltz, you have shared your interest in research on research. Is not now the time to set up an initiative on this topic? The Research on Research Institute²² has done a great job assessing the impact of COVID-19 on open science.

**Marc Schiltz:** We are joining the Research on Research Institute, as have a number of other funders and members of Science Europe. For instance, the Swiss National Science Foundation²³ and the Austrian Science Fund²⁴ have already been members since phase one. There will be new members now. We at the Luxembourg National Research Fund²⁵ are part of it in phase two. Europe-wide, this is probably the major initiative to move forward because here again it is also about sharing our experiences and sharing data. Take again the narrative CV project, which funders have to some extent set up and tested independently from each other, but most of it is based on the narrative CV model that was issued by the Royal Society²⁶. I think most of us have taken that up and then have tailored it to our own needs. However, now that we have experience with this model for one or two years, we should all sit together and discuss what lessons we take from that.

It is, however, more than that. The Research on Research Initiative is really also about conducting research, so having projects where we trial new methods for evaluation, but we do it in a scientific way so that we put this whole process and these methods on more of an evidence-based approach. It is interesting that we are scientists, but when it comes to our own peer review and assessment methods, we have not really based them on any scientific evidence. There is virtually no evidence that things such as journal impact factors really are valuable and correct assessment methods and yet we continue using them. It is more a religious belief in a sense, so we need to move away from that as well.
**Animator: What exactly is a narrative CV? What do you expect from it?**

**Marc Schiltz:** The narrative CV is where we do ask the researcher to describe much more broadly the contributions, not just to the scientific field, but also to the scientific community because that is something which is valuable, like mentoring people, like sharing data, participating in open science initiatives, reviewing, which are services to the scientific community. Then the third pillar I think is service to society. How do you engage with society? Of course, it is more important in some fields than in others. If you are in the field of medical research, you should engage with patients and patients' associations. If you are involved in more theoretical physics maybe that is less important, but still there are other aspects. It is not just engaging society. It is also communicating. It is public engagement, communicating about science, teaching the wider public. It is largely left open to the research institute to decide what to put into a narrative CV. To some extent we leave it to them. It is therefore not just a long list of publications. We have probably all had a 60-page CV with a long list of publications in front of us, which most likely no one reads.

**Bert Overlaet:** The advantage of this narrative is that it is open-ended. It is not predetermined. It does not tell you to do this or that. We started by asking people what the five most important publications are. That was 30 years ago. Now we ask people what their five most important achievements are. Achievements can be totally different things, and people can immediately focus on their strengths. It also means that it is not necessary to have 60 criteria and that the committee has to grade all of them for every candidate. However, the major advantage is that the candidate can propose himself or herself in the best possible way.

**Marc Schiltz:** COVID-19 again is an example. Our government at some point decided to set up a scientific advisory mechanism and, even beyond that, really a taskforce to provide them with modelisations and simulations. Some of the scientists spent lots of their time, and at some point they asked questions: 'I have now put aside my normal research, and what I am doing now will probably not end up in a high-level publication, so will it hurt me in my career?' These are the kind of questions which we no longer want to see. If you can contribute to solving such an important topic as the pandemic, as a scientist you should not have to ask yourself, 'Is it going to hurt my career because I have set aside my other research'?

**Michael Murphy:** When you examine a CV or an application for a grant, you are looking for evidence of excellence. But you must also look for evidence of what a person is doing or what a group are doing to sustain excellence, and the evidence that they adduce to show sustainment of excellence.
** Animator:** One recommendation of the Initiative for Science in Europe²⁷ (ISE) is to engage researchers in all decisions regarding changes to research assessment. How can we do that? Does Europe want this as well? In the Paris Call they say at the end that all researcher organisations should be involved of course.

**Maria Leptin:** I fully subscribe to that. We have had our careers and it indeed does not matter where my next papers are published. It really does not matter for us, but it does matter for the young ones. We are sitting here pontificating. We have to engage. I will therefore encourage the taskforce at the ERC to talk to these communities. There is Eurodoc²⁸, there is ISE and there is the Young Academy of Europe²⁹. All these groups should think for themselves and let us know. I mean they should obviously contribute on our webpage, but we will specifically listen to them.

**Jean-Éric Paquet:** They are part of the coalition now. The very construct of the coalition entirely covers associations of researchers. I mean without them this cannot happen properly. You need ownership and you need input. That is the way this is structured and therefore again, individual scientists and associations at national level and at discipline level should join up for the coalition.

**Marc Schiltz:** Yesterday we had a presentation by Cherifa Boukacem that I found very inspiring, about the study that she has conducted on early-career researchers. It is very clear that they have a feeling that they have to play the present game of journal impact factors and so forth, but they are profoundly opposed to it - or at least it - does not align with their own values. That makes me very optimistic in a sense because I think we can build, and we must build on them.

**Michael Murphy:** Just a thought reflecting on the conversation: it is all about the positive steps that we must take to achieve open science. One subject that we have not touched on is avoiding the creation of new barriers to open science. It is very important that national governments, regulatory agencies, and the Commission should “open science-proof” any new legislation or regulations. An example I am thinking of right now is the Digital Services Act³⁰. The Act has the potential to create new barriers to what we are trying to achieve, and it is really important that the Act is “open science-proofed” in the remaining weeks in legislation development.

**Bert Overlaet:** If you are looking at the Paris Call in terms of assessment or researchers, it is important that we avoid mechanisms that lead to all kinds of box-ticking bureaucracy. One of the things that always strikes me is that if you look at the last 15 to 20 years we have made a lot of progress, and recently we have included general equality plans in the Horizon Europe application forms. These are
all steps forward. However, if you then finally get a financial audit on your project, and if you have invested in educating your young researchers it is rejected because it is not research. We have to make sure that the institutions get sufficient support in different forms for these efforts that they are making in experimenting with new ways to assess researchers. LERU wants to call upon the Commission to have a more active role as a sort of facilitator than was initially projected. If we want new tools to be developed, I think we need some European support in order to avoid them being given out again to commercial players. There is still a lot of work to do there, and we would ask for stronger support from the European Commission in that respect.

**Jean-Éric Paquet:** The Commission is indeed positioning itself now as the coalition is progressively emerging as a facilitator and that is the way we should position ourselves now. Obviously as the coalition comes together, as we draw out the coalition agreement and move to implementation there should be no doubt that we will move into executive mode as well. We will be available for that.

**Animator:** Mr Coulhon, France has published a decree about research integrity³¹. The High Council, over which you preside, hosts the French Office for Research Integrity too. Is this office also connected to open science?

**Thierry Coulhon:** As I said earlier, it is very helpful to have this French Office for Research Integrity inside the High Council because it does have influence on our guidelines. This institution could be and probably will be independent someday, but we host, develop and nurture it in the meantime. It influences us in a positive direction.

**Animator:** Mr Paquet, do you have the feeling that Europe is taking the lead at the moment on open science?

**Jean-Éric Paquet:** Yes, we are. I think we are leading on science generally and I think we are also leading on open science, starting with coAlition S and now moving on research assessment, of course benefiting from global frameworks, which are a very strong foundation, but moving I think in very practical implementation. We are leading, and we will obviously only be able to lead if we really come together in this coalition, so, for the third time now, please join it. We all agree and now that we need to engage and commit time, resources, and attention. There is always a step between the two. You asked Thierry earlier about whether the landslide is happening. I think the landslide is indeed probably happening. We do not see it yet, but it is happening, and therefore the Commission will put very significant
resources in terms into supporting the building of the coalition as a facilitator and
then supporting what we will construct together.

** Animator: Are you all confident that the open science principles will be broadly put into reality? **

** Michael Murphy:** Everybody in the system needs to have a good understanding of the purpose of Open Science, how it is going to happen, how they can contribute and how it is going to impact them. There is a lot of communication work to be done and a lot of leadership development and informing to be done. We all need to have a common sense of what a realistic timeframe is and commit to it.

**Marc Schiltz:** It is to some extent a culture change that we are implementing. Culture changes cannot be dictated. I am very optimistic, but culture changes take time. We must discuss and discuss and discuss again with all those involved. Culture changes are successful if there are leaders, but they must be co-created by these leaders and that is the way we should embark on this.

** Animator: Co-created is what we are looking for, and we can see here on this panel and more broadly that stakeholders are sticking together, rising up and working together in order to propose a concrete implantation process. It is now happening, part of the fate of science is at stake here and I understand that failure may not be an option.**
References

1. "Horizon Europe", accessed 16 March 2022, https://ec.europa.eu/info/research-and-innovation/funding/funding-opportunities/funding-programmes-and-open-calls/horizon-europe_en.

2. Commission Européenne, "Towards a Reform of the Research Assessment System: Scoping Report" (LU: Publications Office of the European Union, 2021), https://data.europa.eu/doi/10.2777/707440.

3. European Commission, "Call for Interest - Towards an Agreement on Reforming Research Assessment", 16 December 2021, https://ec.europa.eu/info/news/call-interest-towards-agreement-reforming-research-assessment-2021-dec-16_en.

4. "cOAlition S", accessed 16 March 2022, https://www.coalition-s.org.

5. Commission Européenne, "Towards a Reform of the Research Assessment System."

6. UNESCO, "UNESCO Recommendation on Open Science", 2021, https://unesdoc.unesco.org/ark:/48223/pf0000379949.

7. "European Research Council", accessed 16 March 2022, https://erc.europa.eu/.

8. American Society for Cell Biology, "Declaration on Research Assessment (DORA)", DORA, 2012, https://sfdora.org/.

9. "European Innovation Council", accessed 16 March 2022, https://eic.ec.europa.eu/index_en.

10. "Science Europe", accessed 16 March 2022, https://www.scienceeurope.org/.

11. UNESCO, "The 17 Goals - Sustainable Development", accessed 16 March 2022, https://sdgs.un.org/goals.

12. European University Association, "The EUA Open Science Agenda 2025", February 2022, https://www.eua.eu/downloads/publications/eua%20os%20agenda.pdf.

13. Bert Overlaet, "A Pathway towards Multidimensional Academic Careers - A LERU Framework for the Assessment of Researchers", January 2022, https://www.leru.org/publications/a-pathway-towards-multidimensional-academic-careers-a-leru-framework-for-the-assessment-of-researchers.

14. "European Institute of Innovation and Technology", accessed 16 March 2022, https://eit.europa.eu/.

15. "U-Multirank", accessed 16 March 2022, https://www.umultirank.org/.

16. "High Council for the Evaluation of Research and Higher Education", accessed 16 March 2022, https://www.hceres.fr/en.

17. "French Office for Research Integrity (OFIS)", Hcéres, accessed 24 March 2022, https://www.hceres.fr/en.

18. "Science and Technology Observatory (OST)", Hcéres, accessed 24 March 2022, https://www.hceres.fr/en.

19. Observatoire des sciences et des techniques, « La position scientifique de la France dans le monde et en Europe, 2005-2018 », February 2021, https://www.hceres.fr/fr.

20. "European Research Infrastructure Consortium (ERIC)", Text, European Commission - European Commission, accessed 24 March 2022, https://ec.europa.eu/info/research-
and-innovation/strategy/strategy-2020-2024/our-digital-future/european-research-infrastructures/eric_en.

21. "Open Research Europe | Open Access Publishing Platform", Open Research Europe | Open Access Publishing Platform, accessed 24 March 2022, https://open-research-europe.ec.europa.eu/.

22. "Research on Research Institute", accessed 24 March 2022, https://researchonresearch.org.

23. "Swiss National Science Foundation (SNSF)", Swiss National Science Foundation (SNSF), accessed 24 March 2022, https://www.snf.ch/en/FKhU9kAtfXx7w9Al/page/undefined/en/FKhU9kAtfXx7w9Al/page/.

24. "Austrian Science Fund (FWF)", accessed 24 March 2022, https://www.fwf.ac.at/en/.

25. "FNR - Luxembourg National Research Fund", FNR - Luxembourg National Research Fund, accessed 24 March 2022, https://www.fnr.lu/.

26. "Résumé for Researchers", The Royal Society, accessed 16 March 2022, https://royalsociety.org/topics-policy/projects/research-culture/tools-for-support/resume-for-researchers/.

27. "Initiative for Science in Europe", accessed 16 March 2022, https://initiative-se.eu/.

28. "Eurodoc", accessed 16 March 2022, https://eurodoc-net.com/.

29. "Young Academy of Europe", accessed 16 March 2022, https://yacadouro.org/.

30. Proposal for a regulation of the European Parliament and of the Council on a Single Market For Digital Services (Digital Services Act) and amending Directive 2000/31/EC. 15 Decembre 2022. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:52020PC0825.

31. « Décret n° 2021-1572 du 3 décembre 2021 relatif au respect des exigences de l'intégrité scientifique par les établissements publics contribuant au service public de la recherche et les fondations reconnues d'utilité publique ayant pour activité principale la recherche publique - Légifrance », Pub. L. No. Décret n° 2021-1572 du 3 décembre 2021 (2021), https://www.legifrance.gouv.fr/jorf/id/JORFTEXT000044411360.
The Software Pillar of Open Science
<figure><img src="image_43.png" /></figure>
# Building the Software Pillar of Open Science

Roberto Di Cosmo
Software Heritage/Inria/University of Paris

Thanks a lot for having me here today. It is a real pleasure to open this session about the relevance of software in open science. Let me start by providing a little bit of context. If we look around us, we see software everywhere. It powers our industry, fuels innovations and is essential to academic research. It is the fabric that binds our digital and professional lives together. It is thanks to software that we are able today to participate in this conference despite the fact that we are in the middle of a pandemic. However, when we talk about software, sometimes we forget that it does not come out of the blue. It is not just a piece of data that comes out of an instrument. Software is written by human beings in the form of source code, which is a precious form of knowledge. It is actually a very unique form of knowledge, because it is built to be understood by humans and executed by machines. As Professor Abelson from MIT wrote in a beautiful book in 1985, 'Programmes must be written for people to read, and only incidentally for machines to execute'¹.

What did he mean by saying this? Maybe it is easier to understand if we think of some particular examples of software source code. A piece of source code which is old is a fragment of the source code used on the lunar-landing module of the Apollo 11 mission that allowed us to put a man on the moon. Some of its text is very complicated to understand because it was assembly language of the kind used on the early machines of the 1960s. However, alongside the assembly code we find comments in human language that describe what the software is meant to be doing. This is a message from humans to humans. It is not just a message for a machine. More recently, if you look at programmes that are written with a higher level of programming language, like C, you can find beautiful pieces of software where the language has evolved over time. You have a name for the variable, a name for the function, but again you need comments to understand what is going on, although sometimes even with the comments it is not so easy.

Len Shustek, the founding chairman emeritus of the Computer History Museum, beautifully stated in his seminal 2006 paper on preserving software, 'The access to source code provides us with a view into the mind of the designer²'. This is very important to know because again it is human ingenuity that produces all this. It is not
just a tool. It is much more than that. The history of software is quite short, unlike
many other disciplines. In the 1960s we had the chance to put a man on the moon,
and this was done, by the way, thanks to a woman, Margaret Hamilton. She led the
team of engineers who developed 60 000 lines of code used in the mission. Those
60 000 lines of code were enough to send a man to the moon and bring him back.
Today, some 50 years later, a Linux kernel with over 20 million lines of code is one of
the many components fount in the phones that we have in our pockets that allows us
to send a smiley to a friend, or to send a message to somebody.

The reason why we had this lightning-fast growth is of course because software is
changing the world we live in, but also because there is the open source software
movement that started over 30 years ago. This led to an incredible collaborative effort
by tens of millions of developers worldwide to work together and build the incredible
software infrastructure we all use today. There is an old saying that we should build
on the shoulders of giants, and we are doing so by reusing over and over again many
components of previous work done by others in the very same spirit of open science,
albeit this movement started much earlier than the term ‘open science’ was actually
getting noticed.

I am particularly stressing this because sometimes you still find people who think
that software is just a piece of data, a sequence of zeros and ones. It is not. It is
much more. It is very special, very different. Software projects evolve over time.
Some software projects may last for decades. The development history of how we
changed it, who changed it, what, when and how is key to its understanding. The
software we use today exhibits incredible complexity in different forms. It may be
complex because it is a big piece of software with millions of lines of code. It can be
complex because even a tiny programme may rely to perform its function on a broad
spectrum of other subcomponents and subroutines, and each of these dependencies
may be developed by a large number of other people. We should bear in mind that the
software we use in research is just a thin layer on top of the general incredible set of
software components developed by many developer communities around the world.
To finish up on this point, again software is the fruit of human ingenuity. You cannot
compare software source code to just a bunch of numbers that you got from one of
your instruments. It comes from people working together! And even from the legal
point of view, it falls under copyright law, unlike what happens with data.

Now that we have established the general picture, let's focus on the fact that
software is fundamental in science too. People have started to notice how software
is now essential in all disciplines. It is not just a matter of computer science: most of
the research software we see is not written by computer scientists. It is written by
colleagues in many other disciplines. Today I think it is important to bring forward the message that when we are talking about open science, we really need to recognize that there are at least three essential pillars: of course open access to the articles published by our colleagues, and unfettered access to the data that is used in our experiments, but this would not be complete without a third essential pillar, which is the source code of the software which is used to manipulate, create and keep this data. It is the software pillar of open science.

Software in research is a multi-faceted object. It can be a tool used by somebody to create and analyse data. It can be the outcome of a research effort as proof of a result or because it embodies new algorithms or revolutionary data structures, or it can even be the object of research to see how software is built correctly. No matter which facet we look at, we need to have access to the *source code* of the software and so open source, which you could call something like open access to the source code, but it is much older than that, is really necessary. It is necessary to avoid reinventing the wheel and to accelerate scientific discovery, and for that we need to keep the history of all the source code built to enable reproducibility of research results, And this is essential to make it easier to accept the result of research because you can access the tools used to get these results.

If we look at the academic world, what kind of needs can we identify around software and source code? Depending on who you are – you may be a researcher, you may be a team leader or responsible for a laboratory, you might be running a big research organisation – you will need to have places where you archive and reference the software you are using in an article to make sure somebody else can find the same result you did. You want to get credit of course for what you did if somebody is using your software. You might want to reproduce a result from a colleague or build on top of it. These are all the kinds of things you need if you are a researcher. If you are head of a laboratory or of a team you usually need to produce a report, know what software is developed, maintain a webpage and track the software contributions. If you are a research organisation you need to know what software you are using, and what software you are contributing to, because it is important to have technology transfer in order to get an idea of what your impact on society is, because software built in research, as we will see later, is not just for research. Sometimes it has a direct impact on society. This is also needed to establish a funding strategy and to use for career evaluation.

In order to address all those needs there are many, many things that need to be done. I would like to start with what we could call the easy part. Of course, we need an archive, that is a place where you can actually store software and be sure that
you will be able to retrieve it later on. This is not what you can do by using the typical code-hosting platforms that everybody uses to develop software: projects stored there come and go, and even the platforms themselves come and go, they are not archives.

We need a way of referencing exactly, precisely the software artefact that we are interested in using to make sure that when we run it again, we can reproduce a result. Then we need to provide a proper description and proper metadata to make it easy to discover and reuse these artefacts. Finally, and this is touchier because it is connected to evaluation, we need to find the proper way to actually credit the contributions of those contributing to research software, which is not the same as just referencing a piece of software.

If we want to do this, a starting point is to have a look at what kind of infrastructures we need to support these kinds of cases in research software. Software is all over the place, so there are many ecosystems involved in software. There is a scholarly ecosystem, the one we are interested in today, and there are many others involving, for example, industry, public administration, cultural heritage and so forth. As analysed in a report published in 2020³, after six months of work by a very broad European working group, you can identify scholarly repositories, like open access repositories, as well as publishers and aggregators that are all collecting and exchanging data. Their mission used to be focused on publications, and is now extending to data, but what about software? If you want to properly address software you need to base your work on a universal software archive that connects the scholarly ecosystem with all the other ecosystems and ensures that we get archival and reference, not just for the thin layer of source code that is developed in the research ecosystem, but for all the other components that are necessary to make it work.

The Software Heritage Initiative we launched some seven years ago comes into play here by archiving everything that is available around the world. It provides the common layer on top of which the scholarly ecosystem provides this added value that comes from curation, description, citation, credit and so forth. Here in France, we have been working for many years to implement this vision. There is a real workflow that can be used by researchers today in France to automatically archive their software and the software they need, not necessarily just theirs, into the Software Heritage archive⁴. They can also deposit proper metadata, which is curated, into the HAL national access portal⁵. This allows you to get a beautiful presentation of a software project with proper attribution, with reference to the institute that funded it, with a precise description of how to cite it and with a pointer that brings you to the universal archive, which is
Software Heritage, that provides you a full view of this software as another piece of research results and not just as a bunch of zeros and ones.

<figure><img src="image_44.png" /><figcaption>Fig. 15 : Research Software Infrastructures: Overall Architecture</figcaption></figure>

Source: EOSC Executive Board Working Group (WG) Architecture Task Force (TF) SIRS, "Scholarly Infrastructures for Research Software", Website (Publications Office of the European Union, 7 December 2020), http://op.europa.eu/en/publication-detail/-/publication/145fd0f3-3907-11eb-b27b-01aa75ed71a1/language-en.

This example shows that these interconnections can be successfully established, and is also showing how to do it properly. If I may, I would like to make a call here to other people, other organisations, and other countries to join forces in the same kind of initiative. Let us try to avoid the major risks and major mistakes we have been taking and making in other areas, for example, falling into the trap of balkanisation. We should not build a multiplication of different and incompatible infrastructures and silos all over the place. Unfortunately, it is a big temptation for everybody to "just build his own archive", but then the result is that you will end up with duplicated objects in different archives with different identifiers and then you will need to spend a lot of money, time, and effort to try to build a federation after the fact instead of before the fact. We should avoid using closed or for-profit platforms that we cannot control, and we also should avoid using project money to fund operations, which are a completely different issue, in the research sector.
That was a look at things that can be done at the infrastructure level, but let me take a step back and focus now on the broader policy issues we also need to address. If we really want to have software playing its main role in open science, it is important to have policies for the dissemination and reuse of software developed for doing research. We really need to set the default to open source for research software. Open source creates value: look at the industry sector, where it is creating billions in value, and you will see that it is not incompatible with technology transfer. We just need to adapt our traditional way of doing technology transfer to open source.

We also need a framework for evaluation and recognition of researchers because unfortunately many countries still spend time developing beautiful high-quality software that is needed for research, yet it does not count in a research or engineer's career, and this needs to change. However, when we do this for evaluation, we need to avoid the mistakes that we have already made in the publication systems: in particular, we must avoid relying only on quantitative indicators, which are even more damaging in software than in other places. We also need to address the issue of the sustainability of open source on the technical, organisational, and financial levels.

There is, however, good news here because awareness is rising. In 2018, some 40 experts from all over the planet came to Paris to work on the Paris Call on Software Source Code as Heritage for Sustainable Development⁶. If you look at this Call, which was published on the UNESCO website back in 2019, one of the points it puts forward is the need to promote software development as a valuable research activity and recognise it in the careers of academics if they produce high-quality software. More recently, the report on Scholarly Infrastructures for Research Software from a working group created under impulsion of the European Commission called to make research software available as open source unless there are strong reasons not to do⁷. Moreover, in the recently published Open Science Recommendation from UNESCO⁸ there is a call to use only non-profit, long-term infrastructure for open science and to operate on a community-based scale.

Implementation of these high-level recommendations has started already in France. If you look at the French 2nd National Plan for Open Science⁹ there is now a chapter fully dedicated to software, which is on par with publication and data. Among the many recommendations is for creating a charter for research software policy at the national level and for recognising software development. This second point is already implemented, as you will see in the software awards just after this session, and there are many other significant provisions that I do not have time to delve into right now.
There would have been so many other things to say, but I needed to pick some that I think are very important. If you look at the road ahead, of course we need to spend more energy, money and time on building proper infrastructure for research software and recognising software as a key enabler of research, not just as a tool. This has many implications and I think Professor Lucke's talk will address some of these in the next presentation. Then of course we need to connect with the scholarly ecosystem, by linking software with publications and data. Here the role of the publishers is fundamental, but I would ask the publishers to take the time to consider that software is a noble research output, not just a piece of data, so you need to use specific infrastructures and identifiers.

Last but not least, at the institutional level we need representation and support, the same way this has been done for the other aspects of open science. We need an office in charge of the strategy around research software and open source, and not just in terms of technology transfer. We need to help our colleagues to do proper funding, governance and so forth. Finally, as for incentives and recognition in evaluation, valuing quality research software is possible, but again, beware of quantitative indicators. We do not want to have an s-index, as a software index, like the h-index in publication. There are other ways of doing this. For example, one is the software awards ceremony you will attend today. We really need to build the software pillar of open science together. The time has finally come. As always in academia, change takes time, but I firmly believe that together, if we work coherently together, we really can make it.

## To go a step further

"The next step would be to see European or international collaboration in which academic institutions and research organisations would actually put together their effort and build on the common ground on archiving software, on referencing software, on providing proper metadata, on credit and citation, and on finding the right way to credit researchers for what they do in software development."

"The national awards given on the OSEC occasion represent the first time that at the highest institutional level we are putting in the limelight the people who have spent an incredible number of hours building the software that is essential for research today."

"Outside the fields of academia there are too many areas where software was seen as just a tool, and you do not have much respect for something that is just a tool. When you realise software is more than just a tool then you start noticing that you should
pay attention. We therefore now need that impulse. We have vice presidents for open science in universities, however, we do not have a vice president for open source in academia."

"There are many awards for free software in general, but this is the first time we are setting up an award for free software in research at the level of a ministry of research. The objective is to put the spotlight on the importance of software for research. It is a way of giving recognition to researchers who have done an incredible job over a very long time without proper recognition in academia. Software is very broad. It is used all over the place. Of course, we have prizes for software in other places, but we need them in academia because we need to build the software pillar of open science."

## References

1. Harold Abelson, Gerald Jay Sussman, Julie Sussman, *Structure and Interpretation of Computer Programs*, The MIT Press, Cambridge, 1985.

2. Len Shustek, "What Should We Collect to Preserve the History of Software?", *IEEE Annals of the History of Computing* 28, n° 4 (October 2006): 112-111, https://doi.org/10.1109/MAHC.2006.78.

3. EOSC Executive Board Working Group (WG) Architecture Task Force (TF) SIRS, "Software Heritage", accessed 24 March 2022, https://www.softwareheritage.org/.

4. "Software Heritage", accessed 24 March 2022, https://www.softwareheritage.org/.

5. HAL Science Ouverte", accessed 16 March 2022, https://hal.archives-ouvertes.fr/.

6. Software Heritage, "Paris Call: Software Source Code as Heritage for Sustainable Development", 2019, https://unesdoc.unesco.org/ark:/48223/pf0000366715 locale=fr https://unesdoc.unesco.org/ark:/48223/pf0000366715 locale=fr https://unesdoc.unesco.org/ark:/48223/pf0000366715.

7. EOSC Executive Board Working Group (WG) Architecture Task Force (TF) SIRS, "Scholarly Infrastructures for Research Software".

8. UNESCO, "UNESCO Recommendation on Open Science", 2021, https://unesdoc.unesco.org/ark:/48223/pf0000379949.

9. Ministère de l'Enseignement supérieur, de la Recherche et de l'Innovation, "Second French Plan for Open Science", *Ouvrir la Science* (blog), 29 June 2021, https://www.ouvrirlascience.fr/second-national-plan-for-open-science/.
# The role of Infrastructure for Software in Open Science

Ulrike Lucke
University of Potsdam

Let us first dive a bit deeper into the issues related to research software before we find out how a proper infrastructure might help us here. Software is a complex thing, and it can be different. It can be an artefact we create in our research, a tool we use in our research, or the subject of research itself.

The last perspective of those three is computer science, but let us have a closer look at the other two, which are relevant for others disciplines, too. Unfortunately, there is no common pattern that research software follows. Research software is as diverse as the disciplines are. Based on an overview of software used for research¹ as analysed by the French Ministry of Research, there are only a few, rather generic tools broadly used (dominated by proprietary office software) and then a broad spectrum of other different software. Some are rather tools, meaning software that is used for research as it is. For others, researchers are also involved in the production of that software as a part of their research process. Both have their own challenges for sustainable maintenance of that software.

These challenges keep us from being effective at the moment. Long-term availability of research software is crucial for the reproducibility of research results, a key factor for quality assurance in research. However, many studies have revealed that reproducibility is often hindered by missing availability of the used software.

The results of a study carried out by Daniel Nüst and others in 2018 in the field of geosciences² showed that even if data is available, there is little or no reproducibility of the overall results. Throughout the pipeline of research from input data, pre-processing and analysis of that data, over code to the computational environment and the final results, there is a leakage with the code or, latest, with its execution. Christian Collberg and his colleagues had already identified reasons for this leakage in 2014 for the field of computer systems³. More or less the same picture exists today. If available at all, and this is still the main problem – software is not available, neither in an archive nor by email – the code of that software simply fails in terms of being built to an executable program or this programme is not executable anymore on a new platform.
Therefore, a finding could be that more emphasis should be put on the quality of
developing, publishing and maintaining that software. The methodological basis
of software engineering could help here. Software engineering is not the same
as mere coding. It is more like a structured design process, and it could help to
overcome such reproducibility issues. Software engineering is a professional activity
requiring significant effort, training, and skills. It is not enough to just write 100 000
lines of code to deserve recognition, but it is the usability of that software by others
that makes this a result to be recognised.

One could argue that we do not have such problems if we use existing tools; not to
develop our own software, but to use existing ones and just rely on that basis. For
instance, the presentation that I gave on this topic at the conference was designed
with MS Office 2013 and I rely on that, aware that this might result in conflicts
with other tools. Many developments of virtual reality applications in my group
are developed in Unity, version 2019 at the moment, and they will hardly run with
any other tool. Another example: one of the very top-ranked tools in the French
Ministry's list of tools referred to earlier is MS Excel, again another proprietary
tool that might prevent us from sharing our research. Unluckily, many groups,
associations, work packages or task forces in open science still use proprietary
tools instead of open tools, because it is simply so easy. We need to put much more
effort into highly functional, comfortable and interoperable open source tools and
then the situation might change.

You might say store it as comma-separated values and, yes, indeed, that helps,
but it is not enough. Depending on the version of whatever software you are using
to process your data, to visualise your data, things may change. For instance,
depending on the number of bits used to store the data in your computer, decimal
positions might be cut off or rounded up or down using different algorithms. That's
why data is always closely tied to the very special version of that software you used
to generate it, to process it, to visualise it. There is no way to completely separate
data from software.

To make things even worse, there is not a single software or tool that we use for
a certain research project. We are using more likely a complex IT environment
as a matter of course. Most of the components are hidden beneath the surface.
There are, for instance, operating systems. I assume you know if you are working
on Windows, Linux or Mac, but do you know which drivers, libraries and codecs
you have installed? Do you know which execution environment your software is
using? Typically not, and to be honest I do not want to know that. Should this not be
hidden? However, if you would really draw an image of all the software components
you are using in your research environment, it may look like a wobbly tower of
bricks. And I bet that somewhere there is this one tiny little component on which
everything else is built – and that makes everything crash if it fails.

Do you remember in 2014 the bug in that openSSL encryption, that we all use for
communication on the Internet, that caused a security breach called ‘Heartbleed⁴³ and our local data was put at risk? Do you remember in 2016, there was a bug in the
fMRI software that was broadly used to analyse neuroimaging⁵? There was no security
reason, but brain areas were declared as active, but in fact they were not. This is a
research issue. Do you remember in 2021 the bug in log4j⁶, just a few weeks ago? You
could say writing a logbook, is that critical? Yes, it is if that piece of software opens a
backdoor to let others enter our computers. Such issues are typical for software that
is not well maintained, and this puts our research at risk.

That, however, is enough of the bad news. Let us come to the good news. First,
there is a technical approach common in data centres to help tackle such problems.
Software, along with the data processed by it, along with the whole computing
environment around the software, can be packed into containers, technically boxes.
Those containers can be easily moved to other systems, can be easily updated if new
versions of any components are released and can be easily recovered after a crash.
That is huge progress. You might be reminded of those Jupyter Notebooks⁷. You just
put together the data, the code and the documentation and press ‘execute’, and
this is possible because of the standardised infrastructure below that software that
enables you to easily run and share your research results, including your software.

Using such a container helps not only to maintain but also to publish and reuse the
software. Publication comes along with curation, assuring a certain level of quality
and, moreover, running software in such a sandbox prevents your system from being
damaged.

At the moment, within the German National Research Data Infrastructure my
colleague Michael Goedicke and I are preparing such a solution for computer science
in Germany⁸. You might compare this approach with the infrastructure we use for
electricity, water, the Internet and so on. There is an invisible network of pipelines,
cables and switches beneath the surface of our streets, and we just use the outlets
in our walls. Plug the device in and everything works, as long as we have the right
power adapter in place when we are at a conference oversees. We do not see all the
pipelines, cables, and switches. We do not see the many people taking care of it.
We just pay our monthly fee and go ahead. This situation can be compared to the IT
infrastructure used for research software. Having powerful servers with virtualisation
technology in place and having this container technology in place, your software
works. If there is an additional layer to handle your recipes – yes, this is like cooking –
you can easily set up different combinations and configurations of software in those
containers⁹. This can be used for all the tools we already have in our research portfolio
and for all the research software that we will continue to develop.

<figure><img src="image_45.png" /><figcaption>Fig. 16: Digital Research Infrastructure</figcaption></figure>

Source: Lucke, "The role of Infrastructure for Software in Open Science". Media used during the presentation (available on the conference's website).

This sounds difficult, does it not? Do not try to copy this at home. Let the data centre
do their jobs. They are professionals for that infrastructure. We are professionals in
research. We only have to know how to do our research, not how to maintain such a
containerised infrastructure. However, yes, we should finally know how to use such
an infrastructure, how to put our software and data in that box and deploy it, or how
to maybe adapt what somebody else deployed and use it for our own research. This
means additional education and training will be necessary for researchers to achieve
these digital skills.

Again, there is good news. There are successful examples, like a series of hackathons
carried out by a colleague of mine¹⁰. Peer Trilcke is one of the leading researchers in
digital humanities. He makes use of software to answer questions from philologies
through mainly quantitative research. That is why he is also teaching his students how
to do that, how to install a computing environment, how to design an algorithm and
how to answer their research questions with that. He does it in a very practical way,
using textual data, like Drama Corpora, or visual data, like urban graffiti, or audio data
from Spotify – cultural data in a broader sense. He creates a kind of data culture from
this, and students love that and learn a lot.
Since students are able to learn how to work this way, whatever their research field is, researchers should be able to learn that too. We are currently preparing a systematic evaluation of the competencies that the students gained over the past three years from these hackathons. At the moment, there is such an introductory course established across different fields of study in that discipline, and a broader one is in preparation. I hope it will be available online free for other fields, too, maybe in forms of micro-credentials to easily invoke it. That way future researchers can be trained in using such kind of infrastructure for their later work.

If you were the good fairy and you asked me what I wished for, I would mention the following three things:

* First, recognise the production of research software as a result of research. This affects the time that we need for preparing high-quality code on a project, which requires both funding and certain skills.

* Second, recognise the availability of research software as a valuable outcome. We have to extend our assessment beyond publication count and metrics towards open data and software and we have to consider these, for instance, in our appointment procedures when we select new professors.

* Third, recognise the quality of research software as an important issue. This means that we have to evolve practices for coding towards professional software engineering and to establish effective IT infrastructures and support structures for research software.

This is all about software as a key enabler of research and I think there is no way around it. However, as I told you, software does massively benefit from a proper infrastructure. Is the fairy still in the room? Maybe three more wishes.

* Number four: increase the maturity and integration of current infrastructures and tools. For this we have to identify and to further promote the promising approaches that already exist.

* Number five: adapt the funding instruments to digital infrastructure so we have to add development and maintenance costs to the funding where currently human costs are predominant.

* Finally, number six: establish an international collaboration in infrastructure for research software.

Let us bring together all the relevant stakeholders to support their work. This would bring research significantly forward. And some final good news: we do not have to start from scratch. As we have heard before, there are so many initiatives on the
international level, the European level and the national level involving many smart and
dedicated people with many great ideas. Let us please therefore just come together
and cooperate, as we are doing here today.

References

1. Le Béchec, Mariannig et al., State of open science practices in France (SOSP-FR), January 2022, https://www.ouvrirlascience.fr/state-of-open-science-practices-in-france-sosp-fr/. Accessed 15 May 2022.

2. Daniel Nüst et al., “Reproducible Research and GIScience: An Evaluation Using AGILE Conference Papers”, *PeerJ* 6 (13 July 2018): e5072, https://doi.org/10.7717/peerj.5072.

3. Christian Collberg and Todd A. Proebsting, “Repeatability in Computer Systems Research”, *Communications of the ACM* 59, n° 3 (25 February 2016): 62-69, https://doi.org/10.1145/2812803.

4. “Heartbleed Bug”, accessed 24 March 2022, https://heartbleed.com/.

5. Bec Crew, “A Bug in FMRI Software Could Invalidate 15 Years of Brain Research”, *ScienceAlert*, 6 July 2016, https://www.sciencealert.com/a-bug-in-fmri-software-could-invalidate-decades-of-brain-research-scientists-discover.

6. Bree Fowler, “Log4j Software Bug: What You Need to Know”, CNET, 21 December 2021, https://www.cnet.com/news/privacy/log4j-software-bug-cisa-issues-emergency-directive-to-federal-agencies/.

7. “Project Jupyter”, accessed 24 March 2022, https://jupyter.org.

8. National Research Data Infrastructure for and with Computer Science (NFDIxCS), accessed 13 May 2022, https://nfdixcs.org/.

9. Sven Strickroth, Dirk Bußler, Ulrike Lucke, "Container-based Dynamic Infrastructure for Education On-Demand". In: Kienle, A., Harrer, A., Haake, J. M. & Lingnau, A. (Hrsg.), DELFI 2021, Bonn: Gesellschaft für Informatik e.V., S. 205-216, 2021. https://dl.gi.de/handle/20.500.12116/37012.

10. Dennis Mischke Sluyter-Gäthje Peer Trilcke and Henny, “Hackathons Als Kollektiv-Kreative Bildungsereignisse. Ein Konzept Zur Gestaltung Offener Lehrveranstaltungen in Den Digital Humanities”, accessed 24 March 2022, https://proposals.dhd2022.de/264.html.
# Open Science, Publications and Code

Melissa Harrison

EMBL-EBI

I am a publisher and have been a publisher for most of my career, including working at eLife. I now work at EMBL-EBI managing an open access repository. We have over 7.5 million full-text articles and preprints, the COVID-19 preprint subset and some others. The perspectives I have gained from those two different roles are interesting. From the perspective of an open access publisher, one of the problems that we encountered at eLife when trying to encourage responsible behaviours such as the use of software and its citing, is that when you are trying to do all these things, you are asking your authors to do yet another thing. There was the response from authors: ‘How do I cite code’? Quite often we found that one of the issues is that when we were trying to dig out what the code is that people were talking about, they would just give us the citation to an original research article publication instead, and that is not really the point. We need to get access to and promote the actual source code underneath and it is not just about research publications. Software does not fit the usual citation styles and publishers do not know how to tag them or how to deal with them. It is a problem that I have been looking at.

Also, as mentioned, software is not static, so it is not like a research publication where you get your DOI, you publish it and then it is done. It is also not the same as data. These are all problematic issues for us as publishers trying to deal with citing code. One solution that has been very popular within the publishing community is the link between GitHub and Zenodo. Authors of code can quite easily archive their code on to Zenodo and they get a DOI. DOI is very publisher-centric and they understand what this is. We have been using DOIs via Crossref for journal articles for a long, long time and it is easy thing for publishers to recognise, and the code is archived somewhere, so it feels like a winner to everyone in that situation. Then Zenodo can also link back to the GitHub site, so if there are changes people can see what is there.

Using some work from a data scientist at Europe PMC, we mined 7 000 random XML documents we had, and we found 300 000 citations. Of that large number there were 10 citations to Zenodo that we found, using a Zenodo DOI. As Roberto was talking about earlier, having a dedicated archive for software, Zenodo is not that. Of those 10 citations, three were datasets, one was a journal article, and the rest were software.
However, looking at what the publishers called that content when they tagged the citations within their XML format, there was only one that was identified as software and the others were all a mishmash of different things, which is problematic if we are looking for proper credit, attribution and reuse of software.

One of the issues that I have often thought about is that “Zenodo” does not equal “code”, so it is not possible for publishers to say, ‘Okay, fine, there is a Zenodo DOI there, tag it as code’, because it is not that simple. Zenodo holds lots of different things.

Publishers are very familiar with Crossref. It is a publisher organisation. They have DOIs and they craft and use the tools there to validate reference, check them and pull in extra metadata if it is missing, or use it to add DOIs to their publications if authors have not provided them. DataCite is the DOI provider for Zenodo and a lot of publishers are not using the tools of DataCite to crosscheck their references, so if there are Zenodo-based DOIs they are quite often not getting the extra validation steps that would allow the publisher to add in the fact that it is code, data or something else.

Another solution, other than the Zenodo solution, is the Software Heritage solution, as you have heard Roberto talk about. This was the way that eLife went. There was a strong feeling within the team that DOIs were publisher constructs, and they were not necessarily the way forward to reference and archive all the different varieties of software. Software Heritage is archiving all this content from a vast number of different sources. Software Heritage links to specific parts of the code and it is reingesting, creating updates, and it indexes more sources. The other thing key thing for eLife was that we did not have to ask the authors to do the work to create the Zenodo link and archive their code themselves. We could do it for them, and that was a key thing for us because, as you know, my first point was, ‘Oh, yet another thing they are asking me to do’. We felt that we did not want to insist that they did the extra work. We were doing it for them.
<figure><img src="image_46.png" /><figcaption>Fig. 17: Software citations in eLife overtime</figcaption></figure>

Source: Melissa Harrison, «Open Science, Publications and Code». Media used during the presentation (available on the conference's website).

In terms of the eLife experience, I am really proud of how many changes we made. Regarding the VoR (version of record) publications that we published over time, eLife is a modest-sized journal compared to all of the content now that I now deal with at Europe PMC, but by the end of 2021 the number of software citations matched the number of VoR publications. That is not to say that every article we published had a software citation – some might have had two or three and some might have had none – but it is a far cry from where we were at the beginning, so this is really great to see. However, this takes time and effort and eLife, because of our mission-driven status, had the resources and time to look into this. We introduced quality control, instructions, and documentation. There was training for our staff and vendors. We introduced automated checks. We wrote a lot of code to make sure all of these things were being discovered, from finding a GitHub link within an article and identifying that, asking the author for the reference to also validating and checking all software references as well. The editorial team, as well as the production team, were putting a lot of effort in at the beginning as well, which was great.

Just going back out to the wider concept again, using the skills at Europe PMC and the archive that we have there, using that same 7 000 XML documents, the random documents we found at the beginning, 15 of them had a software publication type, and these only came from eLife and PeerJ, representing a tiny fraction of the citations we are finding. Having found that, my colleague then just looked at eLife and PeerJ
publications, and we found 6 205 software references that were identified. Out of the top sources identified, 46% were not available and then it was GitHub and then a variation below that. There was then a long, long tail of unique sources.

One of the things I am very involved with as a publisher is JATS for Reuse¹, which I still chair. Most publishers that are creating full-text context use XML to produce that and JATS, the Journal Article Tagging Suite, is a standard that most people use. Because it is a standard for so many publishers there is a lot of variety, and it has to be effectively loose so that anyone can use it, so JATS4R is a group that has been developed for reuse. The original group that developed this was comprised of open access publishers, thinking about the interoperability and the mining of all this information from that content.

A Software Citation Working Group worked from 2017 to 2020, and we published our recommendation in November 2021². This recommendation is still relatively new, but I remember that when we first set up the working group, a lot of publishers were saying they did not know how to cite software, that there was not a code or term for them to use. eLife was using one, because we invented it and we just went with it, but we realised that that was a blocker for a lot of publishers. One of the key things about the recommendation therefore is giving the publishers the ability to recognise a reference as software and indicate that in the XML so that that is then clear to anyone who is reusing that or who needs to mine for software references.

The other thing about this recommendation was that it had to be as minimal as possible, because the more complexity you add to these things the harder it is for people to take them on. eLife put a lot of effort into very full-structured references, but it is hard. The next thing is that we wanted a pub-id. That is a well-known ID and an identifier for something that is properly archived. GitHub unfortunately does not fit into that remit, but Zenodo and Software Heritage do. In that subset of articles that we looked at from eLife and PeerJ we found that 202 had a pub-id and they were all to DOIs. Failing at that, the next best thing is to find an external link. We all know that external links are not great. It is much better to have a system identifier within the reference, but failing that, an external link works, and so we found in our subset 2 524 external links. However, the sad thing is that 3 834 had no pub-id or external link, which makes them so much harder for people to find, to reuse and just to investigate on the references.

This leads to my conclusion that there is a long road ahead of us and we recognise that, but a lot of progress has been made. Software citation is gaining traction. The fact that it is one of the pillars here as we speak today shows how important it is and
how important people think it should be. Within the publishing world, the fact that I had
such a large group within the JATS4R working group who were interested in creating
this recommendation and driving it forward was notable. One of the issues I see for
people is which path to take. For instance, at eLife we took the Software Heritage
route. There is also Zenodo. I am sure there will be more coming up and different
ways of dealing with things, and that does cause problems in itself sometimes. You
have to reach that tipping point where people all go down one way, but at the moment
we are not quite sure yet.

There are many contributors to all of this. It is not just publishers, it is not just
authors, it is not just the researchers. There are so many moving parts here, and a big
issue is educating people, from publishers to researchers, and everybody needs the
education. We also found that when we were talking to our vendors, we had to explain
the production and editorial issues, and the feedback loops are extensive, as well as
updating on tooling. So, from a publisher perspective, using DataCite and automating
checks between, would help with this cause a lot.

To go a step further

"The hardest thing to overcome to improve software citation probably was the authors' resistance, when trying to explain to them how important it is to get to the source code and cite that rather than a paper associated with it."

"Software is very different to data, there are different infrastructures involved and it is a different environment, but I definitely would say that I did learn a lot from the implementation of data citations and applying them to software, even though it is a different type of content."

References

1. "JATS4R", JATS4R, accessed 24 March 2022, https://jats4r.org.

2. JATS4R software citation subgroup, "Software Citations", JATS4R, November 2021, https://jats4r.org/software-citations/.
<figure><img src="image_47.png" /></figure>
# Policies to support Open Source and Open science

Sayeed Choudhury
Johns Hopkins University

Software is a pillar of open science. The universities produce basically three of these pillars: articles, data and software. Those are connected and in many ways those three pillars of open science and the connections between them relate to what Roberto described as the scholarly ecosystem. In large part, those are the projects that universities do through their research. They are typically funded through grants and other kinds of external funding, and are very much the things that happen within the walls of the university. At some point, if you wish to move the impact of research and education outside the walls of the university then you have to translate them. It is useful to publish papers and to search data. It is also extremely useful to share the software.

Over the last two years we have all been experiencing COVID-19 and there has been a lot of discussion about how open science led to the vaccines that have made it possible for us to deal with it. I would submit that the most important piece of that was actually not the sharing of articles or even the data being shared, but rather the software, the code that people were using to analyse the data associated with the vaccines that we eventually developed.

The open source programmes office, or OSPO, is a university organisation that can support this translation function. It is the unit that we have created within Johns Hopkins¹ to help our researchers translate their research and their education activities outside the walls of the university. One of the ways that we have advanced the work of the OSPO is what you heard from my previous panellists, this statement or affirmation that software is a primary research object. If you ask people within a university, either researchers or administrators or even students, 'Do you care about open source software'? you will get different responses, but if you ask them if they care about research, every one of them will say, 'Yes, of course I do'. Therefore, being able to make this argument that open source software supports research, is a tool, but is also a research object as we have heard, has been incredibly important in terms of advancing the work of this OSPO within Johns Hopkins University.
We have also made the case, as you heard earlier today, that software is a better way to understand what data are being produced rather than the articles that cite those data. The citation or the connection between an article and a dataset is one way, is often a direct mechanical kind of link, but the connection between data and software is much richer. If you look at the software, you can actually understand a great deal more about what data are being used and how those data are being analysed. I will talk about a specific example that describes this.

We created at Johns Hopkins this open source programmes office, or OSPO, and this is something that is used in the private sector by companies, technology companies, but also companies like American Airlines and Walmart, which are not technology companies, to manage their open source software. While there are some common elements between the OSPO in a private sector company and in a university, the OSPO at the university is also doing unique things that are important in the context of research and education and translation. We are helping with a lot of the best practices that you heard in earlier talks, which have to do with things like how to engineer your software, what licences to choose, what risks you may be introducing with those choices and how you will in fact work with publishers and even directly with society at large. The OSPO is therefore building on what has happened in the private sector, but it is expanding and extending this in the context of the university.

I am also pleased to note that while Johns Hopkins created the first OSPO in a university in the United States, there are now at least three others at the University of California, Santa Cruz, at the Rochester Institute of Technology and at the University of Vermont. It is my understanding that there will be additional universities creating these OSPOs, so we can see ourselves coming together as a network of these organisations.

Johns Hopkins is involved in a project called the Observational Health Data Sciences and Informatics, or OHDSI project². Paul Nagy is a researcher in our School of Medicine who is involved in this project. He approached me after hearing about our OSPO saying that they needed help. What he explained was that when it comes to health or medical data it is very difficult to share outside of your research team. You typically have to create a very complex legal agreement. It has to be approved by every party that signs that agreement, and this can be a time-consuming process. What the OHDSI group did instead was create open source software and share that software with each other, and by sharing the software they actually learned more about what types of data each of them were using and what types of analysis they were using on those data. This goes back to the earlier point I made where software