# Proceedings of the Paris Open Science European Conference

Justine Fabre

<figure><img src="image_1.png" /></figure>

## To cite this version:

Justine Fabre. Proceedings of the Paris Open Science European Conference. Académie des sciences; OpenEdition Press; EDP Sciences, 2022, 10.4000/books.oep.15829. hal-04211377

HAL Id: hal-04211377

https://hal.science/hal-04211377v1

Submitted on 19 Sep 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
PROCEEDINGS OF THE
PARIS OPEN
SCIENCE EUROPEAN
CONFERENCE

OSEC 2022

OSEC 2022

Paris Open Science European Conference
# Proceedings of the Paris Open Science European Conference

## OSEC 2022

## Open Science European Conference

DOI: 10.4000/books.oep.15829

Publisher: OpenEdition Press, Académie des sciences, EDP Sciences

Place of publication: Marseille

Year of publication: 2022

Published on OpenEdition Books: 13 October 2022

Series: Laboratoire d'idées

Electronic EAN: 9791036545627

<figure><img src="image_3.png" /></figure>

https://books.openedition.org

### Printed version

EAN (Print version): 9791036578403

Number of pages: 200

### Electronic reference

OPEN SCIENCE EUROPEAN CONFERENCE. *Proceedings of the Paris Open Science European Conference: OSEC 2022*. New edition [online]. Paris, Marseille: OpenEdition Press, 2022 (generated 16 mai 2023). Available on the Internet: <http://books.openedition.org/oep/15829>. ISBN: 9791036545627. DOI: https://doi.org/10.4000/books.oep.15829.
ABSTRACTS

For more than twenty years, the international research community has affirmed its support for open and collaborative practices that improve the quality, transparency, reproducibility and inclusiveness of science. In France, this orientation has been reflected in the adoption of two National Plans for Open Science, in 2018 and 2021.

In this context and on the occasion of the French Presidency of the Council of the European Union, France organised the Open Science European Conference (OSEC) on 4 and 5 February 2022. This conference on the transformation of the research and innovation ecosystem in Europe was an opportunity to address in particular transparency in health research, the future of scientific publishing and the opening of codes and software produced in a scientific context, but also the necessary transformations of research assessment, summarised in the Paris Call presented during the event and calling for the creation of a coalition of actors committed to reforming the current system.

This international event was organised by the French Académie des sciences, the Ministry of Higher Education and Research, the French National Center for Scientific Research (CNRS), the National Institute of Health and Medical Research (Inserm), the High Council for Evaluation of Research and Higher Education (Hcéres), the National Research Agency (ANR), the University of Lorraine and the University of Nantes.

EDITOR'S NOTE

All the material associated with the conference (video recordings, media used by the speakers and from which the illustrations in this book are taken, additional data, etc.) is available on the OSEC website: https://osec2022.eu/en/accueil/
PROCEEDINGS OF THE
PARIS OPEN
SCIENCE EUROPEAN
CONFERENCE

OSEC 2022

OSEC 2022

Paris Open Science European Conference
# Proceedings of the Paris Open Science European Conference

## OSEC 2022

### Open Science European Conference

Publisher: OpenEdition Press, Académie des sciences, EDP Sciences

Place of publication: Marseille

Year of publication: 2022

Series: Laboratoire d'idées

<figure><img src="image_4.png" /></figure>

OpenEdition
Books

https://books.openedition.org

Printed version

Number of pages: 200

Electronic reference
ABSTRACTS

For more than twenty years, the international research community has affirmed its support for open and collaborative practices that improve the quality, transparency, reproducibility and inclusiveness of science. In France, this orientation has been reflected in the adoption of two National Plans for Open Science, in 2018 and 2021.

In this context and on the occasion of the French Presidency of the Council of the European Union, France organised the Open Science European Conference (OSEC) on 4 and 5 February 2022. This conference on the transformation of the research and innovation ecosystem in Europe was an opportunity to address in particular transparency in health research, the future of scientific publishing and the opening of codes and software produced in a scientific context, but also the necessary transformations of research assessment, summarised in the Paris Call presented during the event and calling for the creation of a coalition of actors committed to reforming the current system.

This international event was organised by the French Académie des sciences, the Ministry of Higher Education and Research, the French National Center for Scientific Research (CNRS), the National Institute of Health and Medical Research (Inserm), the High Council for Evaluation of Research and Higher Education (Hcéres), the National Research Agency (ANR), the University of Lorraine and the University of Nantes.

EDITOR'S NOTE

All the material associated with the conference (video recordings, media used by the speakers and from which the illustrations in this book are taken, additional data, etc.) is available on the OSEC website: https://osec2022.eu/en/accueil/
PARIS OPEN
SCIENCE EUROPEAN
CONFERENCE
This book has received the support of the French Académie des sciences, EDP Sciences and OpenEdition.

The international event it reports on was organised by the French Académie des sciences, the French National Center for Scientific Research (CNRS), the National Institute of Health and Medical Research (Inserm), the High Council for Evaluation of Research and Higher Education (Hcéres), the National Research Agency (ANR), the University of Lorraine, the University of Nantes and the Ministry of Higher Education and Research.

<figure><img src="image_5.png" /></figure>

INSTITUT DE FRANCE
Académie des sciences

<figure><img src="image_6.png" /></figure>

edp

sciences

<figure><img src="image_7.png" /></figure>

OpenEdition

anr

agence nationale
de la recherche

<figure><img src="image_8.png" /></figure>

OUVRIR
LA SCIENCE !

<figure><img src="image_9.png" /></figure>

MINISTÈRE
DE L'ENSEIGNEMENT
SUPÉRIEUR
ET DE LA RECHERCHE

Édité
Égalité
Fraternité

<figure><img src="image_10.png" /></figure>

Inserm

<figure><img src="image_11.png" /></figure>

Nantes
Université

<figure><img src="image_12.png" /></figure>

UNIVERSITÉ
DE LORRAINE
PARIS OPEN
SCIENCE EUROPEAN
CONFERENCE

OSEC 2022

<figure><img src="image_13.png" /></figure>

OpenEdition
PRESS

<figure><img src="image_14.png" /></figure>

INSTITUT DE FRANCE
Académie des sciences

ecd sciences
On the cover : *Toile de jute grossière*

© unclesam/stock.adobe.com

Graphic design : Veronica Holguín

Editorial coordination : Justine Fabre (Académie des sciences) and OpenEdition
Layout with Métopes. Méthodes et outils pour l'édition structurée.

Collection « Laboratoire d'idées », 2022

This book is online and open access:

https://books.openedition.org/oep/15829

https://www.edp-open.org/books/edp-open-books/442-proceedings-of-the-paris-open-science-european-conference

Text : Creative Commons Attribution License - 4.0 International
https://creativecommons.org/licenses/by/4.0/deed.fr/

<figure><img src="image_15.png" /></figure>

ISBN papier : 979-10-365-784-03

ISBN électronique : 979-10-365-935-36
All the material associated with the conference (video recordings, media used by the speakers and from which the illustrations in this book are taken, additional data, etc.) is available on the OSEC website: https://osec2022.eu/en/accueil/
<figure><img src="image_16.png" /></figure>
Opening of the Conference
<figure><img src="image_17.png" /></figure>
# First Inaugural Speech

Frédérique Vidal

French Minister of Higher Education, Research and Innovation

What is open science? Quite simply, it is the unfettered dissemination of the results, methods and products of scientific research, particularly through the development of digital technologies and products. Why is it important? Because it fosters collective progress, by allowing science to incorporate practices of reproducibility and collaboration. Also, because it leads to a democratisation of access to knowledge, by taking scientific publications and methods out of their traditional field, and by demanding more transparency. You know how important it is to me that scientific culture, which is too often impoverished in our society, should be universally shared and should be able to enlighten the decisions and consciences of citizens. Finally, because it promotes the multilingualism of scientific knowledge, which is no longer an obstacle thanks to the growing power of deep learning translation software and the potential plurality of scientific editions. And this diversity, which guarantees a plurality of ways of thinking and allows for a fairer evaluation system, is the only way to preserve scientific integrity and to produce a science that is up to the challenges of today.

This is not the first time that we have wanted to take strong political action in favour of open science in Europe, and this conference is part of a line of major milestones. Firstly, there are milestones at a European level since the European Commission recommended the generalisation of open access to scientific publications as early as 2012¹. In 2016, the Amsterdam Call to Action for Open Science was launched, calling for action on open science in all European countries². The European Open Science Days that I have the honour of inaugurating today, organised under the French Presidency of the European Union, are part of this legacy.

But we have also initiated major advances at national level. Together we have launched a first then a second National Plan for Open Science in 2018³ and 2021⁴, and progress is already being made. Our invaluable French Open Science Monitor⁵, the latest edition of which was launched last week, now shows almost 62% open access, which is significant progress. During this period, we have managed to create a Committee for Open Science⁶, and a National Fund for Open Science⁷. We have integrated into the Law on Research Programming⁸ the objective of achieving 100% open access to scientific publications. We have also included these concerns in the
decree of 3 December 2021 on scientific integrity⁹. Finally, before the summer, we will launch the national platform for research data¹⁰, which I announced last July.

At international level, UNESCO recently issued a recommendation on open science¹¹, which in many respects shares the concerns I mentioned earlier. It showed that open science is not a matter of convenience, not an option or even a fashion, but a necessity that can save lives, and I would like to pay tribute to UNESCO and to all the member countries that discussed the details of this fine text, with great rigour and always in the collective interest. Having a global recommendation on open science confirms in many respects its universal importance, not only French or European.

Obviously, there is still a long way to go, and that is why initiatives like these are important. We still need to discuss, debate and decide on a number of issues such as the transparency of health research, the evolution of research evaluation, or the essential contribution of new technologies to open science. The subject of health in particular, during a global pandemic, was bound to come up. It is necessary, at a time when lives may depend on the effectiveness of the solutions proposed, to identify ways of improving practices for communicating results, and the French Open Science Barometer has been a decisive tool for assessing the transparency of health research. Even today, although the rate of openness of health publications is higher than the national average, the results of clinical trials and observational studies, particularly those that are negative, are still not made public enough. This problem is not specifically French, it can also be seen on a European and international scale. We need to improve collectively on this point.

Another possible point of improvement, and in this I agree with the UNESCO recommendation, is scientific publishing. I strongly support the principle of bibliodiversity¹², and I am convinced, like many of you, that publishing diversity is an element that contributes both to academic freedom and to the dissemination of knowledge in society and among scientists. The publishing landscape sometimes suffers from its great concentration and could be more diverse and more balanced, and we must work in this direction so that everyone can find their place.

I would also like to talk to you about the reform of research evaluation, an evaluation that focuses more on quality and concrete impact than on the quantity or format of publications. With the Paris Call¹³ launched today, we are committing the scientific community, within the framework of the French Presidency of the European Union, to reflect on a fairer and more relevant method of evaluating research, one that better promotes the quality of research work. The European Commission's recommendation
for a reform of the research evaluation system, published last November¹⁴, is also
along these lines.

Finally, this diversity of research activities, which we hold dear, is only made possible
by the fundamental role of digital technologies in sharing knowledge for all. For this
reason, we wanted to focus in this conference on research software that are first-
class contributions to research, discovery and knowledge, and we were keen to
reward their actors. Thus, the Director General of Research and Innovation, Ms Claire
Giry, will present the first six open science prizes for free research software tomorrow
on my behalf. Without, of course, revealing the names of the winners in advance, I
would like to pay tribute to them and say how privileged we are to have such talented
people in our universities and research organisations who have chosen to develop
their software under a free licence so that it can be used, modified and improved by
everyone. This was an essential contribution to the digital heritage of humanity.

I therefore wish you collectively for these games to be able to raise awareness, to rethink, to find innovative solutions for a more open and more universally shared science. This is the future of European and global research, the only way to ensure that this research is equal to the challenges that await us.

References

1. European Commission, "Recommendations on access to and preservation of scientific information", 17 July 2012, https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=C ELEX:32012H0417&rid=1.

2. Government of the Netherlands, "Amsterdam Call for Action on Open Science", 2016, https://www.openaccess.nl/sites/www.openaccess.nl/files/documenten/amsterdam-call-for-action-on-open-science.pdf.

3. Ministère de l'Enseignement supérieur, de la Recherche et de l'Innovation, "National Plan for Open Science", *Ouvrir La Science* (blog), 4 July 2018, https://www.ouvirlascience.fr/national-plan-for-open-science-4th-july-2018.

4. Ministère de l'Enseignement supérieur, de la Recherche et de l'Innovation, "Second French Plan for Open Science", *Ouvrir la Science* (blog), 29 June 2021, https://www.ouvirlascience.fr/second-national-plan-for-open-science/.

5. Ministère de l'Enseignement supérieur, de la Recherche et de l'Innovation, « Baromètre de la science ouverte », enseignementsup-recherche.gouv.fr, 28 January 2022, https://www.enseignementsup-recherche.gouv.fr/fr/barometre-de-la-science-ouverte-83405.

6. Committee for Open Science, “The Committee for Open Science”, *Ouvrir La Science* (blog), accessed 16 March 2022, https://www.ouvirlascience.fr/the-committee-for-open-science/.
7. Committee for Open Science, “National Fund for Open Science”, *Ouvrir La Science* (blog), accessed 16 March 2022, https://www.ouvrirlascience.fr/national-fund-for-open-science/.

8. « Loi n° 2020-1674 du 24 décembre 2020 de programmation de la recherche pour les années 2021 à 2030 et portant diverses dispositions relatives à la recherche et à l'enseignement supérieur », Pub. L. No. 2020-1674 (2020), https://www.legifrance.gouv.fr/jorf/id/JORFTEXT000042738027/.

9. « Décret n° 2021-1572 du 3 décembre 2021 relatif au respect des exigences de l'intégrité scientifique par les établissements publics contribuant au service public de la recherche et les fondations reconnues d'utilité publique ayant pour activité principale la recherche publique - Légifrance », Pub. L. No. Décret n° 2021-1572 du 3 December 2021 (2021), https://www.legifrance.gouv.fr/jorf/id/JORFTEXT000044411360.

10. Committee for Open Science, “Recherche Data Gouv, the Federated National Research Data Platform”, *Ouvrir La Science* (blog), 15 September 2021, https://www.ouvrirlascience.fr/recherche-data-gouv-the-federated-national-research-data-platform.

11. UNESCO, “UNESCO Recommendation on Open Science”, 2021, https://unesdoc.unesco.org/ark:/48223/pf0000379949.

12. See also “Jussieu Call for Open science and bibliodiversity”, 10 October 2017, https://jussieucall.org/jussieu-call/.

13. French Open Science Committee, “Paris Call on Research Assessment”, *OSEC 2022* (blog), February 2022, https://osec2022.eu/paris-call/.

14. European Commission, “Towards a Reform of the Research Assessment System: Scoping Report” (LU: Publications Office of the European Union, 2021), https://data.europa.eu/doi/10.2777/707440.
# Second Inaugural Speech

Mariya Gabriel

European Commissioner for Innovation, Research, Culture, Education and Youth

This conference on open science is covering many aspects of strategic relevance to build the future of Europe, a future based on knowledge leading to more opportunities for younger generations to express their talent in attractive research careers, be it in universities, research organisations or industry. Some sessions will be dedicated to discussing the assessment of research and researchers' performance, which affects their behaviours. This debate is timely and justified.

Most researchers are used to measuring in their labs when collecting and processing data. Scientists have to measure with accuracy to produce reliable and verifiable results. The Nobel laureate Richard Feynman started one of his famous lectures challenging his students about the complexity introduced over the centuries by using different measurement units. Measuring research and researchers' performance is of a different nature and more complex in many ways. It has direct impact on people and a far-reaching policy dimension for the European Research Area. We therefore agree on the need to ensure objectiveness and fairness as guiding principles and we can use assessment methodologies to support broader policy objectives, considering that we live in times of change, including in the scientific methods and the public perception of the role of science.

Research processes are in constant evolution, accelerated by the current pandemic and the transition to a resilient, green and digital economy and an inclusive society. The variety of activities making increased use of advanced technologies puts us in a situation where research deliverables are richer and not limited to delivering and counting publications. Modern science delivers other outputs, mostly digital, like datasets, software, algorithms or protocols, and it delivers highly skilled people through specialised education and mentoring. Multidisciplinary collaborations are required to address new scientific questions arising from increased complexity and that is another aspect of this evolution. More and more researchers from different domains, from biology and physics to economics, anthropology or humanities, work together in the boundaries of areas of knowledge.
These types of change invite us to question the system of rewards and incentives and it would be interesting to have feedback from all stakeholders on issues such as whether researchers are too dependent on publications, not reflecting the wider and deeper *raison d'être* of the scientific mission, which includes the dissemination of knowledge and mentoring of new generations. Do we have unhealthy and disproportionate competition between individual researchers? Are we heading to a vicious circle in which more publications translate to more funding, penalising the critical review of the scientific substance and limiting the access of newcomers? Are we risking a lack of rigour, preventing fully enabled peer review in favour of expediency and are we undervaluing research results other than publications and weakening the capacity and willingness of researchers to collaborate?

These issues have existed for a while, but we have an opportunity now following the signature of Member States of the Pact on the new European Research Area and the recent Strategy for Universities. Our opportunity here is to deepen the debate and start acting. Beyond cultural and methodological aspects, opening science and innovation to citizens requires adequate infrastructures, like the European Open Science Cloud, new sets of skills for researchers and for academic and research organisations. We can make progress in designing a conducive assessment system for researchers, research projects and research institutions. An assessment system embedding values such as ethics and integrity, which encompasses excellence and opens new opportunities for the research community through collaborations.

To attract more people to science, particularly women and younger people, we need to increase trust in science among our citizens, and to find a good balance between curiosity-driven and more downstream impact-driven research. 2022 is the European Year of Youth. We should avoid putting young generations in a vulnerable position. Any reform of the assessment systems should have a positive impact on the quality of research and improve the attractiveness of researchers careers and make such careers more accessible.

One of the actions included in the first Policy Agenda for the European Research Area, for the next three years 2022-2024, is to advance towards the reform of the assessment system for research, researchers and institutions to improve their quality, performance and impact. We consulted a variety of stakeholders in 2021, including organisations representing researchers, public funders, private funders, universities, research centres, academies, national evaluation agencies and policymakers, and we received strong support for the objective of ensuring a combination of qualitative and quantitative indicators to evaluate research and researchers without compromising on objectivity and peer review.
The European Commission has started the process to engage over 150 European organisations interested in this work. We aim to advance this year on the commonly agreed principles for assessment and commitments for implementation within concrete timelines. After this initial phase, the work will continue to pilot the updated evaluation criteria and associated processes. Let's expect that the European Universities alliances, for example, will contribute and test the proposed reforms on the ground.

These transformations will require systemic and cultural changes and we cannot dissociate this work from what happens in the rest of the world given the strong collaboration interconnections researchers establish internationally. I ask for all stakeholders to be closely associated to the design and implementation of the proposed changes and to take ownership of them. Good coordination of efforts will be crucial. I expect concrete results and I look forward to working with you because we share the common objective to make science an example of transparency, openness, integrity and opportunities. We all aim to have an inclusive scientific culture that permeates our society through education and disseminates knowledge and our values.
<figure><img src="image_18.png" /></figure>
# Third Inaugural Speech

Shamila Nair-Bedouelle

Assistant Director-General for Natural Sciences, UNESCO

The Paris Open Science European Conference comes at a critical juncture for science. Never before has the need for universal access to science seemed as evident as during this current pandemic. Timely access to science has proved to be key to designing an effective response to COVID-19.

More broadly, we know that timely access to science is a vital factor in tackling the myriad of environmental and medical issues that the world faces today. In fact, access to scientific knowledge and its benefits is a human right enshrined in Article 27 of the Universal Declaration of Human Rights. It is also a question of equality, not just equality between countries and within countries, but equality between men and women. Yet there remain inexcusable divides. We are confronted with a digital divide, a technology divide, a knowledge divide, a knowledge divide between and within countries. As countries call with an ever-greater insistence for these gaps to be closed, transitioning to open science has never been more urgent. Indeed, open science is one of the transformative tools at our disposal to reduce inequalities in science, technology and innovation and fulfil the human right to science and accelerate progress towards the sustainable development goals of the United Nations. Open science not only improves the quality and efficiency of the scientific process, but it makes the process more transparent, more inclusive and more democratic.

It was in recognition of this transformative power of open science that UNESCO's 193 Member States adopted the first international standard-setting instrument on open science in November 2021¹. This Open Science Recommendation of UNESCO came together after two years of discussions between scientists, librarians, data experts, young researchers, coders, publishers, engineers, innovators, decision-makers and indigenous people around the world, who worked hand in hand in a true spirit of compromise to develop a ground-breaking international framework for open science. This Recommendation considers the specific challenges that scientists and other actors face in developing countries, in particular in embracing open science, and this recommendation provides the first internationally agreed definition of open science. It presents key objectives for enablers of open science, but it accommodates the fact that open science cannot be fairly and equitably implemented around the world with a one-size-fits-all strategy.
Building on the four pillars, namely open scientific knowledge, open science infrastructure, open science engagement of societal actors and open dialogue with other knowledge systems, the UNESCO Recommendation on Open Science, the first ever, defines open science as the various movements and practices set out to make multilingual scientific knowledge openly available. It is also set out to boost scientific collaboration and sharing of information for the benefit of science and society, to ensure that no one is left behind. Finally, it extends the processes of scientific knowledge, creation, evaluation and communication to societal actors which operate beyond the traditional scientific community.

For the first time in history, we have a text which presents a set of shared values and principles for the operationalisation of open science. The Recommendation proposes innovative approaches at different stages of the scientific cycle and therefore calls for the barriers to open science to be removed, particularly those barriers relating to research and career evaluation systems.

We are confident at UNESCO that this Recommendation will ensure that open science does not replicate the failures of traditional closed science systems and makes a difference for society. We are conscious of the challenges that our Member States will face in implementing this Recommendation, such as the need to fully understand the challenges and benefits of open science for the various actors and to understand how countries can develop appropriate open science policies, strategies, infrastructure, financing and monitoring frameworks and how they can also finance and provide incentives for implementation.

The transition to the global open science model has only just begun. Countries and scientists across the world will need support, your support, and guidance every step of the way. Science is mankind's greatest enterprise, and we need to promote a scientific humanism. The role of UNESCO and its intergovernmental international scientific programmes have promoted access to the benefits of science and technology for the last 75 years to ensure that no one is left behind for a more peaceful world. Louis Pasteur, one of our greatest scientists of all time, said, 'Science has no motherland because knowledge is the motherland of humanity'. These conferences represent a critically important initiative in that sense.
References

1. UNESCO, "UNESCO Recommendation on Open Science", 2021, https://unesdoc.unesco.org/ark:/48223/pf0000379949.
<figure><img src="image_19.png" /></figure>
# Fourth Inaugural Speech

Antoine Petit

CEO of the French National Center for Scientific Research – CNRS

This conference is the first in a series of events to be organized by France on scientific research and innovation, coinciding with the EU French presidency. The French National Centre for Scientific Research (CNRS) is the leading player in research in Europe, and will be mobilised during these six months by organising and contributing to a number of events that will discuss, and challenge, this Europe of scientific research and innovation. This first Open Science Day has already attracted more than 2000 participants across Europe, which demonstrates, if it was necessary to do so, how important open science is. It is a revolution, a very much needed revolution, for research, a revolution in which CNRS is particularly voluntarily involved. Back in November 2018, CNRS published its own roadmap for open science¹, as well as a plan for research data a year later².

The topics that will be developed today are of critical importance, and I would like to thank the staff of CNRS who helped organise this conference. Our speakers for these next two days come from all over the world, because the issue of open science can only be properly understood in an international logic.

Let's mention some key actions of the CNRS in the service of open science:

100% of publications from CNRS units must be open access. Thanks to the involvement of the scientific communities, this action is gradually being achieved. Just under 50% of these 45,000 yearly publications were open access in 2017. Today, this is the case for almost 80% of them, with a particular progression of 10% in 2021, as a result of a strong policy in favour of the French national open archive HAL³. Since 2019, CNRS researchers have to deposit their publications in this platform. This measure has also had an exemplary effect, judging by the latest figures from the French Open Science Barometer⁴, deployed by the Ministry of Higher Education, Research and Innovation (MESRI), which shows a 10-point increase, with a rate of 62% of French publications in open access. This shows that opening science to all is possible.

At the same time, the CNRS supports numerous publishing models, such as the Diamond Open Access model, which will be at the heart of the conference discussions,
with high-quality platforms such as the Centre Mersenne⁵, developed in Grenoble, OpenEdition⁶, essentially in the field of social sciences and humanities, as well as epi-journals⁷, which are backed up by open archives: from manuscripts deposited in HAL and examined by peers, articles are selected for an epi-review. The rise of these publishing models is absolutely essential for open science.

However, we must be careful as these models must have a reasonable cost and be centrally financed by the academic community. This is what will be discussed in the session on scientific publishing, which will demonstrate that there is an alternative to the so-called Gold Open Access model of publication, which involves the payment of publication fees directly by authors to publishers. This model is not sustainable, neither financially nor ethically. Open access publishing is not synonymous with payment of publication fees, and the CNRS intends to go further to counter the drift of these additional fees imposed by publishers.

In order to profoundly and sustainably change the landscape of scientific publishing, an indispensable step is to modify our individual evaluation practices. CNRS is working on this in a long-term effort, which involves convincing the communities to no longer systematically rely on quantitative bibliometric criteria such as journals' impact factors or authors' h-index. It is necessary to change our practices collectively, and our institutions, universities, research organisations, funding agencies or means agencies must be proactive. It is only at the cost of this collective effort that we will convince scientists to stop trying to publish as much as possible, and only in high-impact journals. To this end, beyond the DORA⁸, signed in 2018, the CNRS has proposed new, more qualitative criteria, which aim to stop relying on journal rankings and to emphasise the fact that we need scientists with diversified profiles. Those who are champions of innovation, of interaction with society, and of open science also deserve all the attention of CNRS, and an evaluation that considers all the professions of researchers.

For individual evaluation practices to evolve, a consensus is needed, and I welcome the current effervescence around the issues of evaluation, whether at national or European level. A text entitled Paris Call for Action on Scientific Research Assessment⁹ has been prepared by the Open Science Committee and published on the Open Science European Conference (OSEC) website. This follows a text proposed by the European Commission in December 2021¹⁰.

The CNRS wants to be part of the reflection undertaken by the European community. This subject of evaluation is another key topic of the OSEC days, in a dedicated conference with many speakers from very different backgrounds.
As this conference today shows, the challenges of open science go beyond the
national and even the European perimeter. But it is particularly up to the European
communities to set an example and to exchange more, to collectively assert their
interest to decision-makers and to research and innovation stakeholders.

It was in this spirit that the CNRS formed the G6 a few years ago, an informal European club that brings together, in addition to the CNRS, the Italian CNR¹¹, the Spanish CSIC¹² and the Max-Planck Society¹³, the Helmholtz Association¹⁴ and the Leibniz Association¹⁵ in Germany. This network brings together some 135,000 multidisciplinary research staff and aims to represent the interests of research communities to European decision-makers in Brussels. Open science and research evaluation is an important issue for the G6, which recently issued a declaration in favour of open science to reiterate its common position and to emphasise six priorities that will help accelerate the movement towards open science, and hence research excellence:

1. Open access to scientific publications as the default mode of academic publication;

2. Research data to be "as open as possible, as closed as necessary", in line with the FAIR principles;

3. Extension of open science principles to research software;

4. New evaluation procedures and criteria, compatible with the principles of open science;

5. The development of new skills with appropriate training;

6. Appropriate and available infrastructures and services within Europe and internationally.

Some of these priorities will be addressed in the sessions in the course of today and
tomorrow. I wish you a great conference.

References

1. "CNRS Roadmap Open Science" (Centre national pour la recherche scientifique, 18 November 2019), https://www.science-ouverte.cnrs.fr/wp-content/uploads/2019/11/CNRS_Roadmap_Open_Science_18nov2019.pdf.

2. "CNRS Research Data Plan" (Centre national pour la recherche scientifique, November 2020), https://www.science-ouverte.cnrs.fr/wp-content/uploads/2021/04/Cnrs Research-Data-Plan_mars21.pdf.

3. "HAL Science Ouverte", accessed 16 March 2022, https://hal.archives-ouvertes.fr/.

4. Ministère de l'Enseignement supérieur, de la Recherche et de l'Innovation, "Baromètre
de la science ouverte", enseignementsup-recherche.gouv.fr, 28 January 2022, https://www.enseignementsup-recherche.gouv.fr/fr/barometre-de-la-science-ouverte-83405.

5. "Centre Mersenne pour l'édition scientifique ouverte", accessed 16 March 2022, https://www.centre-mersenne.org/.

6. "OpenEdition", accessed 16 March 2022, https://www.openedition.org/.

7. "Episciences", accessed 16 March 2022, https://www.episciences.org/.

8. American Society for Cell Biology, "Declaration on Research Assessment (DORA)", DORA, 2012, https://sfdora.org/.

9. French Open Science Committee, "Paris Call on Research Assessment", OSEC 2022 (blog), February 2022, https://osec2022.eu/paris-call/.

10. European Commission, "Towards a Reform of the Research Assessment System: Scoping Report" (LU: Publications Office of the European Union, 2021), https://data.europa.eu/doi/10.2777/707440.

11. "Consiglio Nazionale delle Ricerche", accessed 16 March 2022, https://www.cnr.it/.

12. "Consejo Superior de Investigaciones Científicas", accessed 16 March 2022, https://www.csic.es.

13. "Max-Planck-Gesellschaft", accessed 16 March 2022, https://www.mpg.de.

14. "Helmholtz Gemeinschaft", accessed 16 March 2022, https://www.helmholtz.de.

15. "Leibniz Gemeinschaft", accessed 16 March 2022, https://www.leibniz-gemeinschaft.de/.
# Fifth Inaugural Speech

Patrick Flandrin
President of the French Académie des sciences

When the project to organise the Open Science European Conference 2022 was launched, the *Académie des sciences* enthusiastically offered to host it in the Palais de l'Institut de France, in its beautiful André and Liliane Bettencourt auditorium. We had long hoped that this would be possible, with its promise of direct exchanges and informal meetings. Unfortunately, the prolongation of the Covid-19 pandemic decided otherwise, and we had to move from presence to distance.

It is obviously with regret that we had to give up meeting 'in real life' but, to look on the bright side, we can say that what we lose in conviviality, we gain in terms of greater participation than our on-site capacity allows... The registration figures - over 2000 - speak for themselves in this respect, underlining, if it were still necessary, the great interest that the scientific community has in opening up science. Presence or distance, the *Académie des sciences* is happy and proud to be co-organiser of the event that brings us together during these two days and, on its behalf, I am particularly pleased to welcome you all to this moment of sharing!

When we talk about open science, there is in fact a long association between many of the principles underlying it and the *Académie des sciences*. Since 2009, the *Académie* has been committed to thinking in a very general way about the issues surrounding scientific publications. Under the impetus of Jean-François Bach and Denis Jérôme in particular, it has issued several opinions and reports in this regard, both on the conditions of access to scientific production¹ and on publication and research evaluation practices².

With regard to access first of all, open science aims to make the products and results of research, most of which come from public funds, «as open as possible and as closed as necessary», in order to democratise knowledge and share scientific advances with as many people as possible. This desire, made possible by digital advances, was clearly stated by the *Académie des sciences* in 2018³, in a statement affirming its support for free and universal access to scientific publications. It was then renewed on the occasion of an international colloquium on Open Science Foresight that it organised in the spring of 2019⁴, during which the
diversity of disciplinary customs and the need to take them into account for any implementation of the principles of openness were appreciated.

It is one thing to be able to access as many research products as possible as freely as possible, but scientific production is constantly increasing. We can be pleased about the increase in knowledge that this represents, but we can also be concerned about the proliferation of information of all kinds that this generates, which can quickly overwhelm us. Today, no one can hope to read everything that is written in their field!

As in any jungle, in order to find one's way, one needs to have some reference points that help to separate the essential from the superfluous, to extract useful information from the noise. To do this, it is very tempting to rely on purely quantitative indicators: how many publications? in which journals? how many citations? While such criteria may be useful, the *Académie des sciences* has repeatedly expressed its reservations about an overly accounting approach to evaluation, stressing that, rather than being quantitative, evaluation must be qualitative, based on the intrinsic quality of what is at the heart of the work and not on figures derived from impersonal, simplistic metrics: excellence cannot be read in an Excel table and it is up to researchers to reappropriate the evaluation activity!

This approach is demanding for evaluators, but it should prevail generally in the higher education and research ecosystem. Favouring the qualitative over the quantitative is not only virtuous, it is also an essential springboard for encouraging fewer but better publication, for the greater benefit of all. This is a position that the *Académie des sciences* defends relentlessly, confident that positive developments in this direction are possible, and it is in this sense that it has signed DORA, the San Francisco Declaration on research evaluation⁵.

However, various obstacles are holding back this virtuous movement, which can be explained in part by the very high costs of disseminating scientific knowledge and the preference given to certain closed journals considered to be very prestigious for the promotion of researchers' careers and work. To overcome this obstacle, the *Académie* insists, in a very recent opinion that it has just published⁶, on the need to change publication practices and research evaluation methods. In particular, it recommends improving the system for publishing research products and encouraging bibliodiversity. The *Académie* recognises that the visibility of validated research results on institutional servers must be taken into account fairly by evaluation committees. As such, it is natural that it should align its own practices with its recommendations. For example, the *Comptes Rendus de l'Académie des sciences*, a historical publication founded by Arago in 1835, has been available online for more than two years under the Diamond Open
Access formula, which makes all articles permanently available worldwide, at no cost to either the readers or the authors. This is of course only possible thanks to institutional support, and I would like to thank the CNRS via the centre Mersenne⁷⁸.

Opening science and making it the ordinary scientific practice of tomorrow is a matter for all of us, and for the younger generations. We must make as many researchers as possible aware of the advantages and methods of open science and evaluation, starting with doctoral studies. Training students in evaluation and scientific integrity, and encouraging the introduction of teaching modules on these subjects in all doctoral schools, is essential.

The presentations and round tables of these two days will be as much a testimonial of what exists as an incentive to push further the principles of open science that we are calling for.

## References

1. Jean-François Bach et Denis Jérome, « Les nouveaux enjeux de l'édition scientifique », 24 June 2014, https://www.academie-sciences.fr/pdf/rapport/rads_241014.pdf.

2. Académie des sciences, « L'évaluation individuelle des enseignants-chercheurs et des chercheurs en sciences exactes et expérimentales », 8 July 2009, https://www.academie-sciences.fr/fr/Rapports-ouvrages-avis-et-recommandations-de-l-Académie/evaluation-individuelle-chercheurs-enseignants-chercheurs-en-sciences-exactes-experimentales.html; Académie des sciences, Leopoldina Akademie, et The Royal Society, « Déclaration de Trois Académies Nationales - Académie Des Sciences, Leopoldina et Royal Society - Sur Les Bonnes Pratiques En Matière d'évaluation Des Chercheurs et Des Programmes de Recherche », 27 October 2017, http://www.academie-sciences.fr/pdf/rapport/avis111217.pdf.

3. « L'Académie des sciences soutient l'accès gratuit et universel aux publications scientifiques », 3 October 2018, https://www.academie-sciences.fr/fr/Communiques-de-presse/acces-aux-publications.html.

4. "Foresighting Open Science", 2 April 2019, https://www.academie-sciences.fr/en/Colloquia-Conferences-and-Debates/foresighting-open-science.html.

5. American Society for Cell Biology, "Declaration on Research Assessment (DORA)", DORA, 2012, https://sfdora.org/.

6. Académie des sciences, "The French Academy of Sciences confirms its commitment to open and ethically acceptable science", 27 January 2022, https://www.academie-sciences.fr/en/Press-releases/the-french-academy-of-sciences-confirms-its-commitment-to-open-and-ethically-acceptable-science.html.
7. Six of the seven disciplinary series of the *Comptes Rendus de l'Académie des sciences* are published with the help of the Centre Mersenne. The *Comptes Rendus Palevol* series is published with the help of the Muséum national d'histoire naturelle, also in open access. https://comptes-rendus.academie-sciences.fr/

8. « Centre Mersenne pour l'édition scientifique ouverte », accessed 16 March 2022, https://www.centre-mersenne.org/.
# Health Research Transparency
<figure><img src="image_20.png" /></figure>
# Transparency of Health Research

Isabelle Boutron

Cochrane/Inserm/Université de Paris

I would like to start this presentation by referencing a paper that was published by Alessandro Liberati¹, who was a great epidemiologist, very much involved in the Cochrane collaboration, wherein he reported his experience as a multiple myeloma patient. When his disease relapsed, he had to make a very difficult decision and decide whether he would get a second transplant. As a methodologist he wanted to base this decision on evidence. He searched for evidence; identifying four completed randomised control trials. However, none of them provided fully published results. Consequently, he was forced to make his decision knowing that information was somewhere but not available and he was unable to make an evidence-based decision.

Access to clinical trial information is essential and is actually an ethical requirement. In the Declaration of Helsinki² which states the ethical principles for medical research involving human subjects, it is clearly stated that every research study must be registered, and that authors have a duty to make publicly available the results of their research and are accountable for the completeness and accuracy of their reports. There is also now some legislation in place. In the United States we have the FDA Amendments Act³, which requires that all trialists who have at least one site in the US in their trials must post the results of their trials, no later than one year after the completion of the study. We have similar requirements in Europe with the need to post the summary results of clinical trials in the EU Clinical Trials Register⁴.

Do we, however, have access to clinical trial information? A study was done that evaluated the public availability of results of trials assessing cancer drugs in the United States⁵. All the trials selected were governed by the FDA Amendments Act and were supposed to post their results. However, three years after the study was completed 69% of the trials had no results posted at the ClinicalTrials.gov register and half of them had no results publicly available, meaning no results that were either posted or published.

We also now have access to data for France thanks to the French Open Science Monitor⁶ whose last edition was made public last week. It shows that only 57% of the clinical trials with at least one site in France have their results available, with some difference according to the sponsor: 76% for industry-led trials and 31% for academic-led trials.
A very important initiative and tools have been developed by Ben Goldacre and his team at Oxford University⁷. They developed the TrialsTracker⁸, which identifies all the studies that have to post their results on ClinicalTrials.gov and on the EU Clinical Trial Register and data mines whether they actually fulfil the law. The number of trials overdue, meaning that they should have posted their results but have not, increases over time. Compliance, meaning posting results one year after the study was completed, has increased slightly, but remains low and currently stable at 40%. When you assess the compliance with the requirement to report on the EU Clinical Trial Register the team evaluated 7,274 completed trials and they found that only 49% actually reported results, 68% for industry-funded trials and only 11% for academic trials.

What are the consequences? Of course, first, we do not have access to the results and cannot make evidence-based decisions. However, we also have some evidence that the fact that results are published depends on the actual results and there is some evidence of publication bias. Erick Turner showed this in a paper published in the *New England Journal of Medicine*⁹. He identified 74 clinical trials assessing antidepressant treatments that were submitted by the pharmaceutical industry to the FDA to obtain marketing approval. He systematically searched for publications for these trials and found that in one-third of the trials there was no publication. Then he looked at the results and showed that when the trials were positive, showing that the treatment was beneficial, 97% of the trials were published. When the results were questionable, meaning we do not really know whether the treatment is beneficial, half of them were not published and the other half were published but with a conclusion that was not consistent with the FDA decision. When the results were negative, showing that there was no evidence the treatment had any effect, 68% were not published.

Further, the author tried to do a meta-analysis, summarising all the data, using first only the data that were published and compared it to the data including all trials that were published and not published. This showed that meta-analysis of journal dataset, using only data that were published, showed an increase in effect size of 32%, from 11% to 69%, according to the individual drug evaluated.

What could be done? We are meta-researchers and so we always try to find easy low-cost interventions that could improve research. Here we propose to send an email reminding trialists of the legal requirement to post results on ClinicalTrials.gov. We identified 379 trials that met the FDA requirements but did not post their results on the registry one year after the completion of the trials¹⁰. These trials were randomised. Half of them did not receive any intervention and the other half received an email that was presented like a survey to try to understand why they did not post
their results, and in the survey they were reminded that it was a legal requirement and
that there could be a fine of USD 10 000 per day of delay. Our outcomes were the
percentage of studies with results posted, which showed, at least at six months, an
increase in the percentage of results posted when they received the email. The effect
of this intervention is small, an increase of 10%. However, we could imagine that the
effect would be larger if the FDA instead of a French research team sent the email.

When results are made available through publication one question is whether the reporting is complete. Do we have enough information to be able to critically appraise the trials and use and interpret the results? We have some reporting guidelines, such as the CONSORT statement¹, which consists of a checklist indicating all of the essential information that should systematically be reported in a report of a randomised control trial. These guidelines are widely endorsed by editors who request compliance with the guidelines in their recommendations to authors.

How complete is the reporting of the literature? This was assessed in a very large
study evaluating more than 20,000 trials that were included in Cochrane's Systematic
Reviews¹². When we do a Cochrane systematic review, we always try to evaluate the
methodological quality of the study and we evaluate the risk of bias. In this particular
study the evaluation of the risk of bias was impossible in a large percentage of trials
because the reporting was incomplete. On the positive side, reporting is improving
over time.

We are currently doing a large living systematic review the evaluation of which
includes all randomised control trials involving potential treatments for COVID-19¹³.
This is an open review where we put all the results online. Here we are trying to
evaluate transparency indicators and the completeness of reporting of the first trials
that were included in the review in the first 17 months of the pandemic. The results
show that transparency indicators are not well fulfilled. Only 55% of the trials were
prospectively registered. Protocol and statistical analysis plans were available in only
39% and 29% of the trials respectively. Nevertheless, 68% reported a data-sharing
statement. In terms of completeness of reporting, results were also very disappointing.
Only one-third completely reported the primary outcome. 58% provided essential
information related to the conduct of the study. Half reported the results adequately
and only 15% adequately reported harm in terms of the safety of the treatment.

One question is: what is the role of the peer review process? Is the peer review
process able to improve transparency? This was assessed in a study on 93 published
randomised control trials¹⁴ where it was possible to have access to the first papers
submitted, to the peer review documents and to the published articles and to
compare completeness reporting before and after the peer review process. There is some improvement for some domains related to the methodology, such as sequence generation, allocation concealment and blinding, but other domains did not improve at all after the peer review process and overall the reporting remains very poor with information available for these important items in less than 60% of the trials.

What can be done? We tried to provide some tools. First, we developed an online writing aid tool entitled COBWEB¹⁵. This tool aims to guide authors when they are writing the first draft of their manuscript by providing the CONSORT items, explanations about the CONSORT items and examples of adequate reporting. We showed in randomised control trials that when authors were using the writing aid tool, they produced a more complete manuscript than when they did not have access to the tool. We also developed an online peer review tool that is dedicated to early-career researchers and associated with a training programme. We showed that when early-career researchers were using this tool they were more likely to detect inadequate reporting during the peer review process than the usual peer review process.

Completeness of reporting is important, but the reporting also needs to be accurate. In the protocols in particular, investigators have to specify primary and secondary data that will be assessed and analysed in their trials. We expect that these results are consistently reported in the published reports. However, there is some evidence of selective reporting of outcomes¹⁶. For example, in the studies that systematically compared the outcomes reported in the registry, which are the outcomes that are planned in the protocol, to the outcomes that were reported in the manuscript there was some discrepancy in one-third of the randomised control trials, and these discrepancies favoured statistically significant results.

Finally, there is some evidence of spin in publications¹⁷. Spin is a specific way of reporting that might distort the interpretation of the study results. For example, the authors will present the results to convince the reader that the treatment is more beneficial than is actually shown by the results. There is evidence of spin and quite a prevalence of it in published manuscripts in clinical trials, observational studies and systematic review, whatever the study design. This is problematic because there is some evidence showing that spin can impact readers' interpretation. This was assessed in randomised control trials where we randomised readers to read an abstract with spin or the same abstract reported without spin¹⁸. We found that readers were more likely to believe that a treatment would be beneficial when they read the abstract with spin.
Another important topic is access to individual patient data, mainly to be able to reproduce the results, to reproduce the analysis and to check whether we can trust the results. It is also very important to develop other types of research and to synthesis the trial's research in individual patient data meta-analysis. These types of analysis are very useful, particularly to identify the patients who are more likely to benefit from the treatment.

Some journals, such as *The BMJ* and *PLOS Medicine*, have a very strong policy in that they require data sharing as a condition for publication of clinical trials in their journals. A study led by Florian Naudet published in *The BMJ* in 2018¹⁹ evaluated whether we actually had access to this data. He selected 37 published randomised control trials, requested the individual patient data from all of the authors, but could access the data in only half of the trials.

I would like to finish this presentation with a quote from Professor Doug Altman, who was a Professor of Statistics at the University of Oxford, founder of the Equator Network²⁰ and one of the great leaders in the field of transparency and meta-research. He wrote that the scientific community and the public at large deserve an accurate and complete record of research. We need to make changes to ensure that we will get one.

## To go a step further

"Particularly when we look at compliance with the law, pharmaceutical companies are doing better than academic researchers because I think it is more in their process – there is a law, and they need to follow it – and they probably have more support internally to be able to do it. On the other side, for academic trialists there is definitely a lack of incentive to comply with this requirement of posting results. This is linked to the fact they are not as interested and some of them feel that it is additional administrative requirements. They are going to publish their results, so why should they also post them? There needs to be better understanding in the community and, all the more, more incentives and greater recognition of people who actually fulfil this requirement."

"All of the stakeholders have an important role. Institutions have a huge role in decision-making in terms of the criteria they will use to promote their researchers, and currently transparency and good research are not among the criteria they use. They also need to set up strong support for sharing individual patient data because this is currently not at all well organised. The funders have an essential role. Some
try to implement some actions. For example, the NIH I think requests that access is given to all individual patient data, and in the UK that is also the case, but we need more recognition and support from the funders.”

"An important point to remember is that researchers want to do good research. Then there are some incentives that will make them go in some directions more than others, and perhaps we need to play with these incentives. We definitely need to be very careful in the way we want to change the system and to be very careful in the way we are going to implement these changes. The way things are implemented, and how well understood the reasons are, is going to be very important to achieve a successful outcome."

## References

1. Alessandro Liberati, "So Many Questions, so Few Answers", *Bulletin of the World Health Organization* 88, n° 8 (1 August 2010): 568-69, https://doi.org/10.2471/BLT.10.030810.

2. World Medical Association, "WMA Declaration of Helsinki – Ethical Principles for Medical Research Involving Human Subjects", *World Medical Association* (blog), 9 July 2018, https://www.wma.net/policies-post/wma-declaration-of-helsinki-ethical-principles-for-medical-research-involving-human-subjects/.

3. U.S. National Library of Medicine, "FDAAA 801 and the Final Rule", *ClinicalTrials.gov*, accessed 16 March 2022, https://clinicaltrials.gov/ct2/manage-recs/fdaaa.

4. European Medical Agencies, "Home", EU Clinical Trials Register, accessed 16 March 2022, https://www.clinicaltrialsregister.eu/.

5. Thi-Anh-Hoa Nguyen et al., "Public Availability of Results of Trials Assessing Cancer Drugs in the United States", *Journal of Clinical Oncology: Official Journal of the American Society of Clinical Oncology* 31, n° 24 (20 August 2013): 2998-3003, https://doi.org/10.1200/JCO.2012.46.9577.

6. Ministère de l'Enseignement supérieur, de la Recherche et de l'Innovation, "Baromètre de la science ouverte", enseignementsup-recherche.gouv.fr, 28 January 2022, https://www.enseignementsup-recherche.gouv.fr/fr/barometre-de-la-science-ouverte-83405.

7. Nicholas J. DeVito, Seb Bacon, Ben Goldacre, "Compliance with Legal Requirement to Report Clinical Trial Results on ClinicalTrials.Gov: A Cohort Study", *The Lancet* 395, n° 10221 (1 February 2020): 361-69, https://doi.org/10.1016/S0140-6736(19)33220-9.

8. The Evidence-Based Medicine Data Lab, University of Oxford, "Who's Not Sharing Clinical Trial Results?", EU Trials Tracker, accessed 16 March 2022, http://eu.trialstracker.net/.

9. Erick H. Turner et al., "Selective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy", *New England Journal of Medicine* 358, n° 3 (17 January 2008): 252-60, https://doi.org/10.1056/NEJMsa065779.

10. Annabel Maruani et al., "*Impact of Sending Email Reminders of the Legal Requirement for Posting Results on ClinicalTrials.Gov: Cohort Embedded Pragmatic Randomized Controlled Trial*", BMJ 349 (19 septembre 2014): g5579, https://doi.org/10.1136/bmj.g5579
11. The CONSORT Group, "CONSORT 2010", CONSORT - Transparent Reporting of Trials, 2010, http://www.consort-statement.org/consort-2010.

12. Agnes Dechartres et al., "Evolution of Poor Reporting and Inadequate Methods over Time in 20920 Randomised Controlled Trials Included in Cochrane Reviews: Research on Research Study", *BMJ (Clinical Research Ed.)* 357 (8 June 2017): j2490, https://doi.org/10.1136/bmj.j2490.

13. Philipp Kapp et al., "Transparency and Reporting Characteristics of COVID-19 Randomized Controlled Trials" (medRxiv, 4 February 2022), https://doi.org/10.1101/2022.02.03.22270357.

14. Sally Hopewell et al., "Impact of Peer Review on Reports of Randomised Trials Published in Open Peer Review Journals: Retrospective before and after Study", *BMJ (Clinical Research Ed.)* 349 (1 July 2014): g4145, https://doi.org/10.1136/bmj.g4145.

15. Centre de Recherche en Épidémiologie, French Equator center, "Home", COBWEB Consort-based WEB tool, accessed 16 March 2022, https://cobweb.clinicalepidemio.fr/.

16. Sylvain Mathieu et al., "Comparison of Registered and Published Primary Outcomes in Randomized Controlled Trials", *JAMA* 302, n° 9 (2 September 2009): 977-84, https://doi.org/10.1001/jama.2009.1242.

17. Clément Lazarus et al., "Classification and prevalence of spin in abstracts of non-randomized studies evaluating an intervention", *BMC Medical Research Methodology* 15, n° 1 (13 October 2015): 85, https://doi.org/10.1186/s12874-015-0079-x; Francesco Emilio Vera-Badillo et al., "Bias in Reporting of End Points of Efficacy and Toxicity in Randomized, Clinical Trials for Women with Breast Cancer", *Annals of Oncology: Official Journal of the European Society for Medical Oncology* 24, n° 5 (May 2013): 1238-44, https://doi.org/10.1093/annonc/mds636; Kellia Chiu, Quinn Grundy, Lisa Bero, ""Spin"" in Published Biomedical Literature: A Methodological Systematic Review", *PLoS Biology* 15, n° 9 (September 2017): e2002173, https://doi.org/10.1371/journal.pbio.2002173; Eleanor A. Ochodo et al., "Overinterpretation and Misreporting of Diagnostic Accuracy Studies: Evidence of "Spin"", *Radiology* 267, n° 2 (May 2013): 581-88, https://doi.org/10.1148/radiol.12120527; Emmanuelle Kempf et al., "Overinterpretation and Misreporting of Prognostic Factor Studies in Oncology: A Systematic Review", *British Journal of Cancer* 119, n° 10 (November 2018): 1288-96, https://doi.org/10.1038/s41416-018-0305-5.

18. Isabelle Boutron et al., "Impact of Spin in the Abstracts of Articles Reporting Results of Randomized Controlled Trials in the Field of Cancer: The SPIIN Randomized Controlled Trial", *Journal of Clinical Oncology: Official Journal of the American Society of Clinical Oncology* 32, n° 36 (20 December 2014): 4120-26, https://doi.org/10.1200/JCO.2014.56.7503.

19. Florian Naudet et al., "Data Sharing and Reanalysis of Randomized Controlled Trials in Leading Biomedical Journals with a Full Data Sharing Policy: Survey of Studies Published in The BMJ and PLOS Medicine", *BMJ (Clinical Research Ed.)* 360 (13 February 2018): k400, https://doi.org/10.1136/bmj.k400.

20. "Enhancing the QUALITY and Transparency Of Health Research", The EQUATOR Network, accessed 16 March 2022, https://www.equator-network.org/.
<figure><img src="image_21.png" /></figure>
# How Psychology deals with Open Science. Principles and practices

Ioana Cristea
University of Pavia

I will start with reference to a 2015 study that got wide attention that looked at the reproducibility of psychological science¹. They took 100 experiments in three top psychology journals and tried to replicate them with larger sample sizes while following the procedure as exactly as possible. Comparing the original effect size and the replication size show that the original effect size was almost all higher, much higher in some cases. The mean was that the replication effect size was about half of the original effect size. This led to the question of whether we really know how much of the science in psychology is replicable and whether some of our key findings may in fact just be artefacts or things that were not robust enough from the beginning.

From this, two main ideas started. One is the idea of open science, which very generally means making everything transparent and accessible in terms of both the content of the evidence you have produced as well as the process by which you produced it². Related to this is the concept of reproducibility³, which can apply to different things: are your methods reproducible? Can somebody redo the exact same study and get the same results? Are your results reproducible? If I run a new study and follow your methods exactly, do I get the same or very similar results? Finally, the hardest one, are your conclusions reproducible? Do I draw the same conclusions if I follow the same methodology and get the same results you did?

The paper exploring these ideas is a manifesto. What should we be doing to make science, and particularly psychological science, more open and more reproducible? First, the authors identified what they call threats to reproducible science. Several of these have been known for years. What would explain results such as the open science collaboration paper I referred to at the beginning? Why do we have so much psychological research that does not prove to be reproducible?

One idea that has been studied for a while is the idea of publication bias, the fact that you run more experiments than you publish, and you just keep some in the so-called file drawer. Nobody knows you ran those experiments, and nobody knows what results you got and so they do not inform the literature. Another very well-known
phenomenon that has been studied a lot in psychology is that you look at your data
and you see what patterns you have in your data and then you modify and tamper
with your hypothesis so that your hypothesis is supported by your data. Alternatively,
you apply some sort of filters to your data, using different data analysis tools, to make
the data match your hypothesis. This is called p-hacking and HARKing⁴. There are
other biases, which I will not go into due to time constraints.

There is also the positive side. A set of proposals and initiatives were suggested around five main topics: methods, reporting and dissemination, reproducibility, evaluation and incentives. What could we be doing to make progress in this field? The rest of my presentation will be about some of these initiatives, and I will always present two things: first of all, the initiative and then evaluations that look at whether or not it works. This is a key point in my presentation because as a meta-researcher I want science to get better, but I also want to make sure that the initiatives and the tools we use are effective for sure. Sometimes things seem like a very good idea, but then prove to not be so effective or to be even harmful.

The first one I will talk about is of course registration. Professor Boutron talked about registration of clinical trials, known as prospective registration, mandatory in journals affiliated with the International Committee of Medical Journal Editors since 2005. Basically, this means that you would need to publicly document your trial protocol before you even enrol the first patient. There is a very similar thing in psychology⁵. Preregistration, you declare your study plan, so your hypothesis, methods and analysis, in a public register, something that can be consulted by the public, before starting the study. The definition is very similar. We just use a different name.

What function would this serve? As I said before, there are two ways in which you can
make your results and your hypothesis match. There are so-called evidentiary degrees
of freedom, which means that you can make so many decisions in data analysis and
thereby you can basically make your results fit your hypothesis. For instance, you can
exclude participants so that your results look better, or you can do your statistical
procedure in varying ways and so on. You can also adapt your hypothesis. You look at
your data, see the pattern it shows and you say, 'Well, actually this is exactly what we
were expecting for this and this reason'.

Preregistration declarations would help with both because it would allow decisions
about what you do with your data to be independent before seeing the results.
Therefore, if you, for instance, want to exclude a certain type of participant you have
to say this before you look at your data, and if you have certain hypotheses you
must declare them before you look at your data. This would also calibrate confidence
because people that come after, meta-researchers like me, would have the possibility
to assess what you initially planned and what you decided to do after.

Of course, this all depends on how completely you put all these decisions in your preregistration. Evaluations of preregistrations have also looked at this. I would present two examples. One is one that our group did where we looked at psychological treatments for depression with trials that started enrolment after the mandate was made compulsory⁶. The results showed that only 40% of these trials were prospectively registered, but even out of these 15% did not specify useful information in the registration. They would say things like, ‘We will measure depression’, without saying the scale or the assessment timeframe and so on. Another evaluation looked at the phenomenon in psychology⁷. It took studies that had been vetted to be preregistered. It showed that only two of 27 studies had no deviations, but even more worryingly only one study disclosed all of the deviations. There were a lot of undisclosed deviations in terms of analysis.

Preregistration therefore does not seem to work very well for now because when people do it, they do it in a perfunctory way, a superficial way. A solution that was proposed is to have a particular set of empirical publications. In this you would evaluate and accept study proposals before research is undertaken⁸. You would basically submit the protocol for peer review and if it is accepted then you collect the data and write the report. The report is then accepted regardless of what the data looks like if you follow the protocol. This should help to reduce publication bias because then there would be no more pressure to only publish positive results. We recently proposed a version of this to apply to drug approval, wherein we proposed that any trial that is submitted for drug trial be published afterwards in a registered report and then the entire process follows the rules of the registered report⁹.
<figure><img src="image_22.png" /><figcaption>Fig. 1: Overview of the registered drug approval pathway</figcaption></figure>

Source: Florian Naudet et al., « An Open Science Pathway for Drug Marketing Authorization—Registered Drug Approval », PLOS Medicine 18, n° 8 (9 août 2021): e1003726, https://doi.org/10.1371/journal.pmed.1003726. CC BY 4.0

How do registered reports work? Do they increase transparency? An initial evaluation, conducted in 2018¹⁰, showed that even if their numbers are increasing what seems to be happening is that in a significant percentage, one-third, the initial protocol, so the protocol that was accepted, is not publicly available. This means that essentially, we the readers or researchers, have no way of evaluating whether or not indeed the final publication followed a declared protocol, because we do not have access to what has been declared first. In a way it fixes one problem, that is the protocol becomes more complete because peer reviewers evaluate it, but it leaves a hole in another, that is the protocol is not public.

Another set of studies looked at another important question: are registered reports better in terms of methodological quality than standard reports¹¹? This matters because all this peer review effort goes into evaluating protocols. Does this increase the tension associated with better quality? It seems like it does. The comparison between registered reports and non-registered reports showed that in all aspects regarding methodological rigour and quality of methods and so on registered reports seem to be better. Of course, this is an observational study and it has a lot of limitations, but even before seeing the results, the methods are still described better.

Another study¹² interestingly tried to tap into the question as to whether registered reports reduce publication bias in a way that, since you are no longer pressured to make
your findings positive or pretty or interesting, leads you to publish more studies where
your primary hypothesis result is not confirmed, and it does seem like that is the case.

Finally, in terms of incentives, what can we do to make researchers implement more of these open science practices? One idea came in the very simple form of badges that would acknowledge that in that paper the researcher followed open science practices¹³. Badges that have been proposed include ones for preregistration, having registered a protocol previously, having shared your data, having shared your material and so forth. It is like in kindergarten when kids do something good, and they get a point. It is that same very simple principle.

Do badges work? An initial observational study on the issue showed that with the introduction of badges the percentage of papers that shared data, which was the variable of interest in the study, increased a lot. It therefore seems to point that it works. However, later as the researchers ran a randomised trial at a journal that has a policy asking for data sharing but does not mandate it – it asks to discuss if you are sharing your data or not and why – they had two conditions. In one they just gave the extent of the information and in the other, there was the possibility to also get this nice open data badge with your paper. In the primary outcome they looked at the rate of data sharing, how many papers out of the total really shared their data, and this outcome was only considered met if you really shared the data, so not just saying it is available upon request or ‘contact us for the data’, but saying where it is and checking whether you can find it. The results were very disappointing because out of 160 articles, 80 in each group, only two datasets were shared in each group, so very, very little and no difference¹⁴.

To this point, and this is my last point, the importance of always evaluating the initiatives and proposals – we must make science more reproducible – and there is a final example. This has not yet been studied in a randomised trial. It is a tool called Discrepancy review¹⁵. This is a peer review intervention that would try to reduce undisclosed discrepancies between registrations and publications. They have a set of guidelines and they would have a separate reviewer who would basically check all these aspects. They would do nothing else. They would not review the rest of the paper. They would just be checking whether the paper disclosed deviations from the final publication in the registry. For now, the author has just run a feasibility study, but this is an example of a tool that could be tested in a future randomised trial.
To go a step further

"A lot of work has gone into making ClinicalTrials.gov work, but what seems to be the crucial issue is that even when authors are mandated to register their trial or their study and so forth you also need to have oversight as to whether or not they give all the necessary information. For instance, do they say exactly when they measure what they will measure? Do they say how? Do they say out of 10 tools, which is the primary one? ClinicalTrials.gov is improving in that in the sense that recent registrations are better, but there are still problems there."

"With data sharing, again clinical trial platforms would be a great tool, at least for aggregate data sharing, where, although it is not individual patient data, it is still crucial for meta-analysis and reviews. However, even there again you have to always have in mind that it is not enough to have a registry. You need oversight. There are many examples from ClinicalTrials.gov of researchers or whoever completes the study record putting inadequate information, incomplete information, unusable information and so on. I think the tool could be used, but again what we need to fix is the problem of how we offer support for implementation and from whom. This is the same for journals. Who checks that you did what is required?"

"In experimental clinical psychology, the first step of a PhD is you try to get results that are similar, in the same ballpark, as the seminal studies that inspired you. During my own PhD, I was not. I was getting the opposite results. This was one of the first things that drew my attention. I was starting to look for explanations. I noticed that so many things were not really reported in the original papers I was using, so it was impossible for me to know what I was doing wrong or differently because there was not enough information. Then working on my own data, I noticed that there are a lot of things you can do in terms of data analysis choices, justified choices, so not that you say, 'Well, these participants look bad, I will exclude them', but a certain way of excluding outliers. There are more methods. If you use one and run the study the results of the analysis look more statistically significant. I noticed that is so much in my power to change what the outcome looks like. Of course, I was working on a very small scale, but I started to wonder how many of the papers we are reading are in fact the result of that, and if you start looking for this you start to notice, even with the little information that is given, that there are a lot of these kinds of idiosyncratic decisions or decisions that are clearly *post hoc*. That created a kind of crisis of confidence for me. Later, talking to fellow meta-researchers, many of us have similar stories. We wanted to get the golden findings of experimental research and change the field and then we realised that it might all be just a big house of cards."
"Young researchers and master's students must be trained to open science practices in order to make it by default behaviour, and in a lot of psychology programmes this is happening now. Reproducibility is getting more attention as something that is vital. I see a shift in perspective and in training at all levels, including the undergraduate and graduate levels. The younger generation is very onboard with open science. In psychology replication projects are becoming the norm as student projects. A lot is happening in terms of training in this space, and I am very hopeful that the direction is correct."

## References

1. Open Science Collaboration, "Estimating the reproducibility of psychological science", *Science* 349, n° 6251 (28 August 2015): aac4716, https://doi.org/10.1126/science.aac4716.

2. Marcus R. Munafò et al., "A Manifesto for Reproducible Science", *Nature Human Behaviour* 1, n° 1 (10 January 2017): 1-9, https://doi.org/10.1038/s41562-016-0021.

3. Steven N. Goodman, Daniele Fanelli, John P. A. Ioannidis, "What Does Research Reproducibility Mean?", *Science Translational Medicine* 8, n° 341 (1 June 2016): 341ps12, https://doi.org/10.1126/scitranslmed.aaf5027.

4. "P-hacking" occurs when researchers collect or select data or statistical analyses until nonsignificant results become significant. Megan L. Head, Luke Holman, Rob Lan fear, Andrew T. Kahn, Michael D. Jennions, "The Extent and Consequences of P-Hacking in Science", *PLoS Biology* 13, n° 3, https://doi.org/10.1371/journal.pbio.1002106. HARKing consists in "Hypothesizing After the Results are Known". Norbert L. Kerr, "HARKing: hypothesizing after the results are known", *Personality and social psychology review: an official journal of the Society for Personality and Social Psychology, Inc* vol. 2,3 (1998): 196-217, https://doi.org/10.1207/s15327957pspr0203_4.

5. Tom E. Hardwicke and Eric-Jan Wagenmakers, "Preregistration: A Pragmatic Tool to Increase Transparency, Reduce Bias, and Calibrate Confidence in Scientific Research" (MetaArXiv, 23 April 2021), https://doi.org/10.31222/osf.io/d7bcu.

6. Clara Miguel et al., "Selective outcome reporting and the effectiveness of psychotherapies for depression", *World Psychiatry* 20, n° 3 (October 2021): 444-45, https://doi.org/10.1002/wps.20900.

7. Aline Claesen et al., "Comparing dream to reality: an assessment of adherence of the first generation of preregistered studies", *Royal Society Open Science* 8, n° 10 (27 October 2021): 211037, https://doi.org/10.1098/rsos.211037.

8. Christopher D. Chambers and Loukia Tzavella, "The Past, Present and Future of Registered Reports", *Nature Human Behaviour*, 15 November 2021, 1-14, https://doi.org/10.1038/s41562-021-01193-7.

9. Florian Naudet et al., "An Open Science Pathway for Drug Marketing Authorization—Registered Drug Approval", *PLOS Medicine* 18, n° 8 (9 August 2021): e1003726, https://doi.org/10.1371/journal.pbio.1002102.
doi.org/10.1371/journal.pmed.1003726.

10. Tom E. Hardwicke et al., "An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014–2017)", *Royal Society Open Science* 7, nº 2 (19 February 2020): 190806, https://doi.org/10/ggm5vg.

11. Courtney K. Soderberg et al., “Initial Evidence of Research Quality of Registered Reports Compared with the Standard Publishing Model”, *Nature Human Behaviour* 5, nº 8 (August 2021): 990-97, https://doi.org/10.1038/s41562-021-01142-4.

12. Anne M. Scheel, Mitchell Schijen, Daniel Lakens, “An excess of positive results: Comparing the standard Psychology literature with Registered Reports” (PsyArXiv, 5 February 2020), https://doi.org/10/ggkpqr.

13. Daniel Feiler and Johannes Müller-Trede, “The One That Got Away: Overestimation of Forgone Alternatives as a Hidden Source of Regret”, *Psychological Science* 33, nº 2 (1 February 2022): 314-24, https://doi.org/10.1177/09567976211032657.

14. Mallory C. Kidwell et al., “Badges to Acknowledge Open Practices: A Simple, Low-Cost, Effective Method for Increasing Transparency”, *PLOS Biology* 14, nº 5 (12 May 2016): e1002456, https://doi.org/10.1371/journal.pbio.1002456; Anisa Rowhani-Farid, Adrian Aldcroft, Adrian G. Barnett, “Did awarding badges increase data sharing in BMJ Open? A randomized controlled trial”, *Royal Society Open Science* 7, nº 3 (s. d.): 191818, https://doi.org/10.1098/rsos.191818.

15. TARG Meta-Research Group & Collaborators et al., “Discrepancy Review: A Feasibility Study of a Novel Peer Review Intervention to Reduce Undisclosed Discrepancies between Registrations and Publications” (medRxiv, 21 January 2022), https://doi.org/10.1101/2022.01.18.22269507.
Publication and reporting bias: a
long history towards open science

Tianjing Li
University of Colorado

Both previous speakers have used the terms reporting bias and publication bias many
times. I hope to bring them into the historical context and provide a review of the
history of the research into reporting biases.

The problem of selective reporting has been recognised for hundreds of years. Reporting bias refers to or occurs when the direction and nature of the findings influence how research findings are being disseminated. Early in the 17th Century Francis Bacon noted that, ‘The human intellect is more moved by affirmatives than by negatives’. Robert Boyle, the chemist, lamented the common tendency among scientists not to publish their results until they had a system worked out with the result that ‘many excellent notions or experiments are, by sober and modest men, suppressed’. Many other scientists across many different fields have recognised the problem of selective reporting over the years¹.

More recently, in 1956, the father of medical statistics in Britain, Austin Bradford Hill²,
wrote, ‘A negative result may be dour, but often it is no less important than the
positive and in view of that importance it must surely be established by adequate
publication of the evidence’. American psychiatrist Seymour Kety³ wrote, ‘A positive
result is exciting and interesting and gets published quickly. A negative result or one,
which is inconsistent with current opinion, is either unexciting or attributed to some
error and is not published, so that at first in the case of a new therapy there is a
clustering towards positive results with fewer negative results being published. Then
some brave or naive or non-comformist soul, like the little child who said that the
emperor had no clothes, comes up with a negative result which he dares to publish.
That starts the pendulum swinging in the other direction and now negative results
become popular and important’.

These historical quotes provided you the background of how long we have recognised
this problem. Systematic research into selective reporting began in education and
psychology in the 1950s. Selective reporting of healthcare research was identified in
the early 1980s, but the importance and the consequence of that was not recognised
until a decade later. It has gained a lot of importance as a result of rapid development of research synthesis methods, such as systematic review. Systematic reviews are based on a complete survey of the literature on a given topic. They inform healthcare decisions and policies. When the systematic reviews are based on a selective subset of studies rather than all the studies out there, the findings could be biased. The need to have access to complete information about the methods and results of research is strong motivation for study into selective reporting.

Reporting bias arises when the dissemination of research is influenced by the nature and direction of findings⁴. The nature and direction of findings refers to the statistical significance of the results, the magnitude of the effect and whether the results are in favour of a particular hypothesis. Statistically significant results are results that are unlikely to be due to chance. Instead, some kind of cause is more likely. Selective reporting can manifest in many different ways. To broadly categorise selective reporting, it may concern not reporting studies at all, also known as publication bias; reporting studies in part, also known as selective outcome reporting; reporting in a manner that is difficult for others to access or reporting without transparency, such as duplicate publication or spinning. Spinning means undermining or overemphasising certain results.

Mary Lee Smith⁵ likely first mentioned the term ‘publication bias’ in a manuscript where she compared the meta-analytical results based on published studies with those that were not published. A comparison of the protocols of studies before they actually initiated with the publication – we call those inception cohort studies – provides a reliable estimate of the association between the nature and direction of findings with the publication status. Based on two cohort studies approved by research ethics committees or included in trial registrations, only 49% to 60% of randomised trials were published and the findings were three times more likely to be published when the results were statistically significant⁶. This is a huge problem.

What are the consequences of publication bias? First of all, it is not scientific. It may alter the evidence on the benefit and harms of treatment. One example is a comparison of reboxetine with placebo⁷. If you compare the published results with the unpublished results for both the efficacy and safety outcomes, the published results overemphasised the beneficial effect of reboxetine, which may not exist at all, and underestimated the harmful effects of reboxetine. As an example, with regard to the outcome ‘remission’, only one published trial showed that reboxetine is 2.5 times more likely to reduce remission, whereas six other unpublished trials showed no evidence of effect. In other words, published results are more than two times more likely to show the efficacy of reboxetine, which the efficacy does not exist.
Failure to publish is also unethical. Participants enter research studies with the understanding that their involvement will contribute to generating new knowledge and to science. Failure to publish breaches the trust of the participants. Lastly, failure to publish research studies constitutes research waste. It is not informative because they never made it to the public domain. It was estimated that on average about 12 000 randomised control trials that should have been published each year were not published.

The second form of reporting bias is outcome reporting bias. Outcome reporting bias manifests in different forms when the reporting of the outcome is influenced by the nature and direction of the findings. An-Wen Chan and his colleagues have conducted quite a few studies⁸, which definitely characterised outcome-reporting bias in healthcare research. Outcome reporting bias manifests in different forms, for example, not reporting pre-specified data in the protocols, reporting primary outcomes as secondary outcomes and vice-versa, introducing new outcomes or reporting outcomes such that they are differentially accessible. In other words, outcome reporting bias occurs when the authors cherry pick what they decided to show to the public.

I was involved in a study that examined outcome reporting bias in industry-sponsored trials of gabapentin for off-label use. We had access to the internal company documents as well as the publicly-available documents for 21 randomised trials that compared gabapentin with placebos for off-label use. We compared protocol to publication for the statement of primary outcome⁹. We identified 21 primary outcomes in the protocol. Those are pre-specified primary outcomes. Eleven were reported without changes in the final publication. Ten were not reported or reported as secondary outcomes. In addition, 17 new primary outcomes were introduced. In terms of the p-values for the protocol-defined primary outcomes in those research reports, nearly all of them are non-statistically significant at the 0.05 significance level. However, through selective outcome reporting, by switching outcomes, introducing new outcomes, or omitting outcomes, in the primary publications the p-value became statistically significant. In other words, regarding the efficacy of gabapentin, dependent upon where you look, you may find very different results.

That was one case study. There are many studies like this. When you take all studies together in a systematic review¹⁰, it yields empirical evidence for publication bias and outcome reporting bias. In summary, statistically significant outcomes were more likely to be reported in full for both efficacy and harm outcomes. About 40% to 62% of studies had at least one primary outcome changed, newly introduced or omitted when trial publications were compared with protocols.
The third form of selective reporting is reporting in a manner that is difficult for others to access. One study conducted by a colleague of mine¹¹ looked at full publication of results initially presented in conference abstracts. We all go to conferences. Disseminating your findings at conferences is a great way to reach a broad audience. However, the conference abstracts are usually limited by the word limit, and they do not provide a full depiction of the study methods or the results. This systematic review examined over 307 000 abstracts presented at conferences. The authors found that only about 37% were subsequently published in full, such as in a journal article, and that number was slightly better for randomised trials, again definite evidence for outcome reporting bias and publication bias. Significant results were 1.3 times more likely to be published in full. Positive results, defined as a result favouring the experiment treatment, were 1.2 times more likely to be published in full.

Now you may start wondering whose problem is this? Who contributes to biased reporting of clinical trials¹²? Numerous surveys and other types of studies of researchers have found that investigators' failure to submit trial results for publication was the primary reason for unpublished clinical trials. The frequently quoted reasons are they are not interested in the results and they do not have time to submit results for publication. The reasons for selective outcome reporting mirror those of publication bias. Investigators' misconception that the editors favour positive results was the main reason for not publishing the results. Also, there is an unawareness of the seriousness of underreporting. On the other hand, there is no evidence that peer reviewer suggestions or the editors' decisions are influenced by the nature and direction of the findings. In summary, it was mainly the researchers' problem. We should always submit our results for publication regardless of the direction and the nature of the finding.

To end on a high note, how do we address reporting biases? Both speakers have touched upon trial registration and a result database. I will discuss data sharing and access to study documents. There are also reporting guidelines and core outcome sets that people can follow.

Beginning with the early studies, it became quite clear that to address reporting and publication bias we need to have a full account of all the studies from its beginning. The study of publication reporting biases and their consequences has led to legislation and policy mandating trial registration.
<figure><img src="image_23.png" /><figcaption>Fig. 2: A timeline of trial registration seminal events</figcaption></figure>

Source: Li, "Publication and reporting bias: a long history towards open science".
Media used during the presentation (available on the conference's website).

In 1997, the Food and Drug Administration Modernisation Act required that all the drug trials of severe and life-threatening conditions must be registered. As a result, in response to that legislation, the National Library of Medicine of the National Institutes of Health in the US launched ClinicalTrials.gov, the trial registration database in 2000. However, the registration had not taken off until the International Committee of Medical Journal Editors (ICMJE) made a requirement of prospective registration of trials before the trial results can be published by the ICMJE journals.

In 2006, the World Health Organisation launched an international initiative to establish the International Clinical Trials Registry Platform. The platform provides one-point access to different registries across countries. In 2007, the FDA Amendments Act (FDAAA) expanded the registration requirements – for all drugs, biologics and devices subject to FDA regulation, the FDAAA requires that the registration be done as soon as the trial is initiated. The results should be posted within 12 months of the completion of the trial or within one month of approval of the product. In response to that requirement, in 2008 and 2009 respectively, ClinicalTrials.gov released the results database and also added an adverse events module.

In the European Union (EU), the European Medicines Agency (EMA) took a more open policy compared with the FDA. In 2011, EMA required that all the trials conducted in the EU should be registered and make their results available within 12 months. As
mentioned earlier, the journals also play a very critical role in the transparency of clinical trial research. In 2013, *The British Medical Journal* put in place a requirement for a statement about anonymised participant level data availability at the time of manuscript submission. In 2016, the final rule made by the US Department of Health and Human Services further clarifies the types of trials that need to be registered on ClinicalTrials.gov and the National Institutes of Health (NIH) also issued a policy to promote the sharing of clinical trial data, requiring that any trials that received funding entirely or partly from the NIH should be registered before the start of the trial and have their results posted in a timely fashion.

Access to study documents is also very critical. In a study where we examined adverse events for gabapentin¹³, we compared the number of different types of adverse events that appeared in the public domain with the number of different types of adverse events in the hidden documents, the documents not publicly available, such as clinical study reports. In the first trial¹⁴, there were only seven different types of adverse events mentioned in the publication, whereas 100 different types of adverse events were mentioned in those hidden documents. It does not look better for the serious adverse events. Public access to protocols, including the statistical analysis plan and clinical study reporting, is essential for preserving the societal value of clinical trials.

Publication was traditionally the way of sharing the results from clinical trials. Increasingly the clinical trial increases our knowledge and influences our decision-making through both the summary level and the individual participant level data being shared and reused by the public. Data sharing increases the scientific value. Evidence has shown that it also increases citation. Furthermore, there is a moral obligation to share the data. Participants of trials become more vocal demanding that their data be shared and reused amongst the broader scientific community in order to leverage the data to maximise the knowledge and value that can be gained from their contributions to the trials. Surveys of patients also showed that most people welcome the idea of data sharing if safeguards are in place for their data safety¹⁵.

Increasingly, funders are mandating data sharing. For example, the Gates Foundation and the Wellcome Trust require their awardees to share their clinical trial data and individual participant level data. Journals are also making requirements of making a statement about data availability and data sharing at the time of trial registration, and those statements will be published when the trial reports are accepted for publication.

In summary, the move towards clinical trial data sharing is part of a wider movement towards open science. Reporting of medical research is biased when it is influenced
by the nature and direction of the findings. That is the definition for publication bias and outcome reporting bias. They can take many different forms. Reporting bias typically involves spurious exaggeration of beneficial effects and suppression of harmful effects of interventions. Clinical trial registration, access to protocols and statistical analysis plans, guidelines for transparent and complete reporting, adoption of core outcome sets and data sharing are critical to prevent reporting biases.

## To go a step further

"In several studies, including one of our own, we found that the data collected in ClinicalTrials.gov is a minimal requirement and it does not capture everything you would like to know about the trial, especially with regard to the methodology and with regard to all the results data. It is definitely a first step, and how reliably they can be used for data analysis in systematic review and meta-analysis, I believe we still have a long way to go, but at least we know the existence of trials, so we know if we are missing results or the publication of certain results, we have a place to look. We are getting there."

"The funders have a very important role. In the US, for example, when you are submitting a new grant if your previous publications are not in the public domain, you cannot even submit the grant. It comes down to how different stakeholders play together. The institution has a huge role. The funders have a huge role. However, there are so many studies that are not funded, so that is becoming more problematic, and the public has to get involved and demand transparency in research and demand open science. All the players need to work together to make this happen."

"Negative results are as important, if not more important, than positive results in the sense that if they are hidden away, information in the public domain is cherry picked. It looks rosier than it actually does, which is very problematic. Negative results will help us to use fewer medications."

"Most people would agree that open science is a great idea, but it requires resources to get there. For most research funders, you could build data sharing into the costing of your study; so there are resources available for researchers to share their data with the public."
References

1. The James Lind Initiative, « Reporting Bias », *The James Lind Library*, accessed 16 March 2022, https://www.jameslindlibrary.org/research-topics/biases/reporting-bias/.

2. Austin Bradford Hill, « Discussion of a paper by DJ Finney », *Journal of the Royal Statistical Society* 119 (1956): 19-20.

3. Seymour Kety, « Comment », in *Psychopharmacology. Problems in Evaluation*, by JO Cole and RW Gerard, 583 (Washington DC: National Academy of Sciences, 651).

4. S. Swaroop Vedula, Asbjørn Hrøbjartsson, Matthew J. Page, « Section 8. Introduction to Systematic Reviews », in *Principles and Practice of Clinical Trials*, ed. Steven Piantadosi and Curtis L. Meinert, 1st ed. (Springer, 2022).

5. Mary Lee Smith, « Responses of counselors to case materials of clients differentiated by gender and ethnic group » (Unpublished Doctoral Dissertation, University of Colorado, 1970).

6. Christine Schmucker et al., « Extent of Non-Publication in Cohorts of Studies Approved by Research Ethics Committees or Included in Trial Registries », *PLOS ONE* 9, n° 12 (23 December 2014): e114023, https://doi.org/10.1371/journal.pone.0114023.

7. Dirk Eyding et al., « Reboxetine for Acute Treatment of Major Depression: Systematic Review and Meta-Analysis of Published and Unpublished Placebo and Selective Serotonin Reuptake Inhibitor Controlled Trials », *BMJ* 341 (12 octobre 2010): c4737, https://doi.org/10.1136/bmj.c4737.

8. See for example An-Wen Chan et al., « Empirical Evidence for Selective Reporting of Outcomes in Randomized Trials: Comparison of Protocols to Published Articles », *JAMA* 291, n° 20 (26 mai 2004): 2457-65, https://doi.org/10.1001/jama.291.20.2457; An-Wen Chan and Douglas G. Altman, « Identifying Outcome Reporting Bias in Randomised Trials on PubMed: Review of Publications and Survey of Authors », *BMJ (Clinical Research Ed.)* 330, n° 7494 (2 April 2005): 753, https://doi.org/10.1136/bmj.38356.424606.8F.

9. S. Swaroop Vedula et al., « Outcome Reporting in Industry-Sponsored Trials of Gabapentin for Off-Label Use », *New England Journal of Medicine* 361, n° 20 (12 November 2009): 1963-71, https://doi.org/10.1056/NEJMsa0906126.

10. Kerry Dwan et al., « Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias – An Updated Review », *PLOS ONE* 8, n° 7 (5 July 2013): e66844, https://doi.org/10.1371/journal.pone.0066844.

11. Roberta W. Scherer et al., « Full Publication of Results Initially Presented in Abstracts », *The Cochrane Database of Systematic Reviews* 11 (20 November 2018): MR000005, https://doi.org/10.1002/14651858.MR000005.pub4.

12. Kay Dickersin et al., « Publication Bias and Clinical Trials », *Controlled Clinical Trials* 8, n° 4 (December 1987): 343-53, https://doi.org/10.1016/0197-2456(87)90155-3; Kay Dickersin, Yuan-I. Min, Curtis L. Meinert, « Factors Influencing Publication of Research Results. Follow-up of Applications Submitted to Two Institutional Review Boards », *JAMA* 267, n° 3 (15 January 1992): 374-78; Kay Dickersin and Yuan-I. Min, « NIH Clinical Trials and Publication Bias », *The Online Journal of Current Clinical Trials* Doc No 50 (28 April 1993);
Kay Dickersin and Iain Chalmers, « Recognizing, investigating and dealing with incomplete and biased reporting of clinical research: from Francis Bacon to the WHO », *Journal of the Royal Society of Medicine* 104, n° 12 (December 2011): 532-38, https://doi.org/10.1258/ jrsm.2011.11k042; Marc Mehu and Klaus R. Scherer, « The Appraisal Bias Model of Cognitive Vulnerability to Depression », *Emotion Review* 7, n° 3 (1 July 2015): 272-79, https://doi.org/10.1177/1754073915575406.

13. Evan Mayo-Wilson et al., « Harms Are Assessed Inconsistently and Reported Inadequately Part 2: Nonsystematic Adverse Events », *Journal of Clinical Epidemiology* 113 (September 2019): 11-19, https://doi.org/10.1016/j.jclinepi.2019.04.020.

14. M. Backonja et al., « Gabapentin for the Symptomatic Treatment of Painful Neuropathy in Patients with Diabetes Mellitus: A Randomized Controlled Trial », *JAMA* 280, n° 21 (2 December 1998): 1831-36, https://doi.org/10.1001/jama.280.21.1831.

15. Center for Global Clinical Research Data, « Why Share Your Clinical Research Data », Vivli, accessed 16 March 2022, https://vivli.org/why-share-your-clinical-research-data/; Institute of Medicine of the National Academies, *Sharing Clinical Trial Data: Maximizing Benefits, Minimizing Risk* (Washington, DC: The National Academies Press, 2015), https://doi.org/10.17226/18998.
<figure><img src="image_24.png" /></figure>
# Concluding keynote: Opening and Transforming Biomedical Research

Ulrich Dirnagl

Charité – University of Medicine of Berlin

My talk will focus on the need to transform biomedical research. My key point will be that institutions have a key role in this transformation. I will exemplify this with three practical examples of how such a reform could be successful at an institutional level.

Why do we need to transform biomedical research in the first place? As we have already heard a lot about this from the previous speakers, let me just emphasise this with a few examples. Researchers are actually acutely aware that there is a problem in academic research. According to a *Nature* survey of researchers in many domains¹, 90% of researchers are of the opinion that we are in the midst of a reproducibility crisis. That means we have problems in reproducing the work of others or even our own work. Isabelle Boutron has already mentioned the very influential series of publications that were published in *The Lancet*². One article in it posited that up of 85% of health research might actually be wasted. I am not sure about this is exactly the number, but it is quite clear that there is tremendous waste in this field.

Let me just give you two more examples that I think illustrate the problem. One is from preclinical research. Ioana Cristea told you about a large scale collaborative study that tried to reproduce key research results in psychology. Analogous to this study in psychology, very recently the Centre for Open Science has undertaken a massive project which sought to replicate 50 seminal, highly cited papers in cancer research³. The results were very similar to the psychology study: the effect sizes of the replication studies were much smaller, in fact 85% of the original, and only less than 40% of the studies were able to reproduce these findings on a statistically significant level.

What is also shown by this seminal publication is that the problem goes much deeper. In fact they tried to replicate, as I said, 50 studies and in the end they could only undertake this with 23 because data was not available for sharing, because the reporting of details, e.g. of the statistics, was not appropriate, analytical code was not
available, key reagents were not shared or clarifications of protocols were not given. The results demonstrate the importance of transparency and openness of research.

A second example, and this is very much along the lines of what we have heard just now from Isabelle Boutron, comes from Germany and from our own studies. Daniel Strech and his group of the QUEST Centre have systematically investigated the timely reporting of clinical trial results of German university medical centres. We have heard about what is mandated in this respect. I will bring another stipulation to your attention another one, one from the World Health Organisation: summary results of clinical trials must be uploaded in registries within 12 months of completion of the study and a full publication should be available within two years. The study regarding German university medical centres shows that the numbers are very much congruent with those we have heard from Isabelle. About 60% of studies are not available within two years, and this has not changed over the years. I think that this is unethical.

We all have heard about these problems. We are reading about them. All major journals are running articles about it. I have stopped collecting clips of those articles for my slide because practically every week there are additional papers exposing the problems we are talking about. These issues have also percolated into the general public, they can read about it in newspapers and magazines. In fact, one could speculate that this has actually contributed to some of the mistrust in scientific results and how they are applied to policymaking during the pandemic. It is a serious problem.

How can we improve the situation? My main point here is that while funders and researchers are important stakeholders, the key role must be played by the institutions⁴. They can and should have internal policies and guidelines on all of this, informed, for example, by DORA⁵ and by the Hong Kong Principles⁶. They should be much more transparent about how they are acting to increase the transparency, trustworthiness and openness of their research. We have heard about external tracking by the UK TrialsTracker⁷, for example, but this needs to be done also by institutions on their own. The institutions also need to assist researchers in open research practices because they are demanding and resource-intense. A good example are so-called data stewards who can help scientists to share their data in a FAIR, that is findable, accessible, interoperable, and reusable manner. Maybe even most importantly, institutions need to incentivise and reward open practices in hiring, in tenure, in promotion and also in intramural programmes.

In essence I think we need to change from a situation where researchers are basically mostly evaluated and rewarded on the basis of where they publish – very often it is
just the impact factor as a very abstract number or the h-index – and by the sum of their third-party funding to a situation where we, in addition at least, value open and responsible practices. We have heard about them during the symposium already. We are talking about preregistrations, about replication studies, about publishing open access, all the items that are contained in the umbrella construct “open and responsible science”.

There is a lot of knowledge on this already. In fact there are many manifestos, principles, frameworks and declarations. We all know the San Francisco Declaration on Research Assessment⁸, which was one of the starting points of this movement. I should also mention that the European Union has been very proactive over the last years, coming up with surveys and recommendations on novel metrics and indicators, and demanding that we should be more open in science. What just came out as part of this conference, the Paris Declaration on Research Assessment⁹, should I think be endorsed by everyone. These declarations are very important.

I was involved in one of them, which was produced in a participatory way, so I know a little bit more about it, but they are all very similar, and so I can just mention what is usually recommended. The Hong Kong Principles¹⁰ prototypically call for assessing researchers on responsible practices, valuing accurate and transparent reporting, valuing practices of open science, valuing broad ranges of research and scholarship, which, for example, includes replications and meta-research, and also to value other contributions of responsible research, such as mentoring, outreach and knowledge exchange.

How can we make these recommendations become a reality? What can institutions do about this? I would like to share a few examples with you from an institution where I work, the Charité University Hospital and the Berlin Institute of Health. Remarkably, the Charité recently has come up with a strategy for its next 10 years or so¹¹, and in this strategy we have anchored responsible practices in hiring, and transparency and openness in research, etc. I think that such institutional strategies and policies are a first important step for an institution.

What does this look like? What can we do? One thing I would like to tell you about is a change that has happened in the way we hire professors. Our dean, Axel Pries, is a real champion of this project. If you apply for a professorship at the Charité right now you have to go to a portal where you will be asked many questions¹². You have to upload your CV and all those things you would expect, but things appear now which you might not expect and which I think are very important. It usually starts with an explanation, like: ‘The Charité attaches great importance to transparent replicable
research and supports the objectives of open science'. This includes the registration of studies in registries, the publication of negative and serial results, all the things that we have heard in the previous talks. This is just to explain to our applicants what we are interested in. Then comes the interesting and relevant question: 'How have you been pursuing these goals so far and what are your plans for the future'? Obviously we also use other criteria, but I think these narratives are very important for the hiring and tenure commissions to get a full picture of candidates and make a fair selection.

As second example I would like to tell you about an online dashboard of open and responsible practices that the Charité has developed for its own benchmarking¹³. In a way it is very similar to the Open Science Barometer that the French have now put up, which I think is just great if this is done on a national level. Here it is done on an institutional level. Institutions need to show how they are faring. They need to monitor it. They need to be transparent about it. Our dashboard is constantly updated and curated to keep track of how we are progressing with open access, how we are doing with sharing of open data, open code, preregistrations, timely publications. And all of this is visualised from past years to now to see whether these things are going up or down, so in essence to see whether we are really achieving anything. This allows our own benchmarking as well as benchmarking within a country or between comparable institutions.

My last example is an intervention because, as we have heard from other speakers, we need to make sure that the things that we are changing – and we often have these brilliant ideas of what can be done – are changed for the better, that these ideas really work and that they are not just eating up resources or that they have unintended consequences. Therefore, our activities need to be complemented with implementation research and with meta-research.

As an example I have picked al project¹⁴ by Daniel Strech and his group, in particular Maia Salholz-Hillel and Delwen Franzen, which is funded by the Wellcome Trust. In this project we are approaching our clinicians and researchers performing clinical trials and show them how they are faring in their studies. They will receive a 'transparency card' which has information on a number of items, including timely publication, a link to registries, preregistration etc. to show how they have doing with a particular study. However, it does not stop there. They also get a complementary card, which lays out specific steps on how they can improve on the reporting of that specific study. We are going to check after six months whether this has changed practices and whether this makes any sense, or whether our approach can be improved.
<figure><img src="image_25.png" /><figcaption>Fig. 3: Improving timely reporting and transparency of clinical trial results</figcaption></figure>

Source: Dirnagl, 'Opening and Transforming Biomedical Research'. Media used during the presentation (available on the conference's website). CC BY 4.0.

In conclusion, the general theme of this session is that we desperately need to reduce waste and increase value in biomedical research. I posit that institutions need to play a very important role in all this and that key to this is changing incentives and rewards. We also need to enable our scientists by providing tools, resources and a opportunities to apply to open science practices. We also need more meta-research to find out what really works and what does not work and how our resources are best spent. My last point is: yes, I think it can be done. We have a lot of bad news, but this is good news. It can be done and I think this conference is a very nice example of how things are moving forward in this space.

## To go a step further

> "One of the problems in all this is that if one institution is changing practices like this, if others are not following suit this may be a problem for those who work in the institution. We may judge them with different criteria which would no longer apply in the outside world and would actually potentially disadvantage them. This is why it takes courage. It also takes courage because it takes resources and needs investment. I think it is very important therefore that there is outside pressure from the European Union to do this, which could lead many institutions to come together to do it jointly."
"There are resistances in the scholarly community against what we propose, and I think part of that resistance is understandable because, rightfully, one can point to the fact that the system somehow works. Take COVID-19: we have a vaccine, we are saving lives of critically ill patients. It is therefore a system that works. Some people are afraid that by changing it we may fare worse. The point here is, yes, we have a system that in principle works, but it could work much better and we need to be aware of the waste that we have. This is where implementation research is so important because we need to convince those people, and partly also ourselves, that these changes make sense and make science better. We need to convince those who are critical with real evidence, not just opinions."

"Observational studies can make big contributions and the mandates for preregistration and for timely reporting should be extended to those. An interesting question which may also come up in this context is whether this should also be mandated for preclinical research. It is certainly more challenging, but studies that are more confirmatory in this space, and hopefully there are more confirmatory studies in the future, should actually have all those qualities. This means they should be preregistered and should have timely reporting and so forth. We have good examples for this. I think these mandates should be extended to more classes of dissemination, but probably not to all."

## References

1. Monya Baker, '1,500 Scientists Lift the Lid on Reproducibility', *Nature News* 533, n° 7604 (26 May 2016): 452, https://doi.org/10/gdgzjx.

2. 'Research: Increasing Value, Reducing Waste', *The Lancet*, 8 January 2014, https://www.thelancet.com/series/research.

3. Center for Open Science, 'Reproducibility Project: Cancer Biology', 2020, https://www.cos.io/rpcb; Timothy M Errington et al., 'Investigating the replicability of preclinical cancer biology', ed. Renata Pasqualini and Eduardo Franco, *eLife* 10 (7 December 2021): e71601, https://doi.org/10.7554/eLife.71601; Timothy M Errington et al., "Challenges for Assessing Replicability in Preclinical Cancer Biology", *ELife* 10 (7 December 2021): e67995, https://doi.org/10.7554/eLife.67995.

4. C. Glenn Begley, Alastair M. Buchan, Ulrich Dirnagl, 'Robust Research: Institutions Must Do Their Part for Reproducibility', *Nature* 525, n° 7567 (September 2015): 25-27, https://doi.org/10.1038/525025a.

5. American Society for Cell Biology, 'Declaration on Research Assessment (DORA)', DORA, 2012, https://sfdora.org/.

6. David Moher et al., 'The Hong Kong Principles for Assessing Researchers: Fostering
Research Integrity', *PLOS Biology* 18, n° 7 (16 July 2020): e3000737, https://doi.org/10.1371/journal.pbio.3000737.

7. The Evidence-Based Medicine Data Lab, University of Oxford, 'Who's Not Sharing Clinical Trial Results?', *EU Trials Tracker*, accessed 16 March 2022, http://eu.trialstracker.net/.

8. American Society for Cell Biology, 'Declaration on Research Assessment (DORA)'.

9. French Open Science Committee, 'Paris Call on Research Assessment', *OSEC 2022* (blog), February 2022, https://osec2022.eu/paris-call/.

10. Moher et al., 'The Hong Kong Principles for Assessing Researchers'.

11. Charité-Universitätsmedizin Berlin, 'Rethinking Health - Charité 2030, strategic orientation', November 2021, https://www.charite.de/fileadmin/user_upload/portal_relaunch/die-charite/Strategie-2030/Charite2030_Strategie_11_2021_engl.pdf.

12. Charité-Universitätsmedizin Berlin, 'Charité - Professorship Application Service', Professorship Application Service, accessed 16 March 2022, https://career.charite.de/?site=home&lang=eng.

13. Charité-Universitätsmedizin Berlin, 'Charité Metrics Dashboard', accessed 16 March 2022, https://www.charite.de/en/research/charite_research/charite_METRICS dashboard/.

14. Berliner Institut für Gesundheitsforschung, Charité-Universitätsmedizin Berlin, Max-Delbrück-Centrum, 'TrackValue', accessed 16 March 2022, https://www.bihealth.org/en/translation/innovation-enabler/quest-center/projects/translational-bioethics/trackvalue.
<figure><img src="image_26.png" /></figure>
The future of scientific publishing
<figure><img src="image_27.png" /></figure>
# Academic Publishing and Open Science – Where do we stand?

Pierre Mounier
OpenEdition/OPERAS

20 years after the Budapest¹, Bethesda² and Berlin³ declarations (the B Declarations) open access seems to be no longer a matter of activism, controversial, marginal or even political. Open access reached its tipping point several years ago and entered a new stage of development as a mainstream practice, almost the default way of publishing research results. As a matter of introduction, I propose to take a look at these dynamics after and maybe beyond the tipping point.

In 2013, the European Commission published a press release⁴ presenting the results of three studies by Science Metrix⁵ showing that in Europe at least more than 50% of the articles published two years before were open access. 10 years after having reached that tipping point it could be useful to assess where we stand and what happened to the dynamics that were identified.

A few days ago, the French Ministry of Research published a new version of its Open Science Monitor⁶ based on diverse sources, such as Unpaywall⁷, the Directory of Open Access Journals (DOAJ)⁸ and various repositories. I encourage you to have a look at this interesting instrument developed by Eric Jeangirard and his team with open source technology, open data and a lot of results as food for thought. In particular, the monitor evidences the strong global dynamic of OA adoption across the whole French research community, with 62% of the papers authored by at least one researcher affiliated with a French institution published open access. Of course, OA adoption is different from one discipline to another, and the strength of this adoption can be different, but it is clearly present and high.
<figure><img src="image_28.png" /><figcaption>Fig. 4. 62% and counting in France</figcaption></figure>

Source: Mounier, «Academic Publishing and Open Science: Where do we stand?», media used during the presentation (available on the conference's website). Data from Unpaywall, MESRI.

The same dynamic can be observed in other European countries, such as in Germany thanks to the Open Access Monitor⁹ funded by the German Research Foundation (Deutsche Forschungsgemeinschaft, DFG¹⁰). The data provided by the German OAM are not comparable of course with the French Open Science Monitor because they do not proceed from the same methodology. However, for example, we have data showing a proportion of 61.4% of articles authored by researchers affiliated with a German institution being published open access in 2020.

The platform openaccess.nl¹¹ proposed yet another methodology to monitor the adoption of open access among researchers affiliated to Dutch universities, and here again the dynamics are consistent with the other countries, with a 62% threshold already reached last year. In conclusion, it is interesting of course not to precisely compare the data because they do not proceed from the same methodology. We see that even with three different methodologies we observe the same dynamic in three different European countries.

However, behind the global dynamics it is necessary to dive into the details and try to identify the logic of models that support this movement. Several routes were identified quite soon in the history of open access, starting with the difference between depositing articles in open repositories, the so-called Green route to open access, and articles published directly on the publisher's platform, the so-called Gold route. The landscape became even more complex when the hybrid concept emerged referring to articles published open access in non-open access journals. Then of course there is the Diamond model, referring to Open Access (OA) journals, free for both readers and authors, meaning no charging article processing charges or subscriptions.
What are the respective dynamics of those models of open access? The traditional opposition between the so-called Green and Gold routes theorised by Stevan Harnad 20 years ago is becoming less and less relevant in the current context. Following the multiplication of policies and institutional strategies, there is a growing number of articles made available open access both on the publisher's platform and in an open repository. This evolution, partly but not only due to the role of PubMed Central¹², renders the long-time debate between the two routes to open access somewhat obsolete. It opens the possibility to think of both routes not as in competition but rather as complementary to each other. Each route brings its own way to open access publishing practices. In my opinion this type of evolution is a sign of the maturity of community regarding OA.

However, on the other side, despite multiple disincentives from funders and library consortia that consider hybrid publication as perpetuating the status quo and introducing the risks of double dipping in open access funding, the model is still present in the landscape as evidenced by the hybrid OA dashboard proposed by Najko Jahn¹³. Hybrid publication is indeed a flexible model that accommodate well with the contradictory requirements between the mandates to publish open access and the need of researchers to publish in the existing recognised journals of their discipline. In fact, hybrid is not a model. It is certainly a temporary adaptation to a transition period and that is probably the reason for its permanence until now.

At the same time, the multiplication of so-called transformative agreements (Tas) gives a solid push to hybrid publication¹⁴. The ESAC Registry¹⁵ that records most of the TAs gives evidence of this growth. This is really an important shift in the landscape with the tremendous growth of articles freed by the TAs according to ESAC figures. However, a recent study that surveyed 197 TAs concluded that globally those agreements do not provide any real transformation in the business model of journals, nor did transformative agreements help to flip these journals to open access. That is mainly because this is not in fact the primary intention of the stakeholders signing the contracts.

Which model will survive and wipe the others out from the market? From what precedes it is already obvious that each model brings its own type of success and addresses a specific need in a specific situation, even temporarily. That is not, however, the end of it of course because recent years have brought into focus other open access publishing models that have slipped under the radar so far. The first one is of course the Diamond model, which designates those publications that are free for both readers and authors. In other words, they do not charge
authors for publishing their articles in open access. A recent study brought to light the important contribution of this model to the global development of open access worldwide.

Here again the reality is more complex than what we think, and it would be a mistake to consider Diamond and Diamond journals as an exclusive product of Latin America and social sciences and humanities, where it is indeed extremely important. In fact, what a study reveals¹⁶ is that the Diamond model is prevalent in many disciplines and most of the regions of the world, including Europe. In other words, while most of the open access discussions were focusing on the glamorous model of the publishing landscape, the Gold APC model, the vast majority in fact of open access journals were frankly adopting another model, the impact and importance of which was wrongly being overlooked by most observers.

The Diamond journals represent an overlooked and neglected part of the OA landscape because by nature they are not involved in the most inflammatory debates related to the stronghold of commercial publishers over scholarly communication. Most of the time Diamond publishing venues are owned by scholars through their universities, research organisations and scholarly societies. In fact, they represent the quiet adaptation to the digital age of a long tradition of scholarly publication. They constitute a remarkably persistent model of open access publication with a stable number of articles published throughout the years, and in that sense the comparison with the evolution of articles published OA in exchange for payment of an APC is striking.

<figure><img src="image_29.png" /><figcaption>Fig. 5: Shares of OA Diamond and APC-based open access models in DOAJ-listed journals</figcaption></figure>

Source: Jeroen Bosman et al., « OA Diamond Journals Study. Part 1: Findings » (Zenodo, 9 mars 2021), https://doi.org/10.5281/zenodo.4558704, Data from DOAJ.
The Open Access Diamond publishing model stands outside of the capitalistic economy of for-profit publishing and is therefore insensitive to market incentives. In addition, the OA Diamond publishing model does more than simply address the needs of researchers. It represents an idea of autonomy and self-rule of the scholarship community in the new digital context.

One of the gems uncovered by the Diamond OA study was this continued innovation throughout the whole sector, with a diversity of initiatives across different disciplines, languages and countries, with scholarly communities working constantly on inventing and adapting their communication practices to their needs, taking full advantage of the possibilities the open web offers. In the Diamond study already mentioned a series of open questions were submitted to almost 1 200 open access journal editors about their motivations. Ethical and political values appeared prominently as key drivers for their professional and personal investment in open access. I would like to share with you one of the most compelling answers we received in this study: ‘Our open-living ethos is not just about open access, open software, open review. We are committed to living, discussing, deciding and failing in public. We discuss and develop our project values and policies openly as our public message’.

Beyond the necessary complementarity of the different routes to open access, new types of debates have emerged in recent years around the concept of diversity in the open access movement. Several voices in the community have called out to take a step back and try to reflect more on the why and not only on the how of open access. They highlight that through its inclusion in institutionalised open science policies the open access movement could fail its original purpose at the very time it seems to be succeeding.

It is true that driven by innovation the adoption of open science led to a strong diversification of technologies, formats and workflows in academic publishing, at least in appearance. The number of new services proposing new ways of creating, assessing, disseminating, discovering and reviewing research results is staggering. 20 years ago, Stevan Harnad’s modest proposal¹⁷ to radically transform scholarly communication by sky writing research on the Internet had many offspring: preprints, mega-journals, digital editions, overlay journals, next gen repositories, publishing platforms, data papers, living books, micropublications and of course 50 shades of open peer review. This fascinating world of innovation in scholarly communication has been wonderfully well charted by our colleagues Jeroen Bosman and Bianca Kramer from Utrecht University with their project 101 Innovations¹⁸.
However, at the same time, we have understood that open science and its powerful push for diversification through innovation does not prevent the concentration of power in the academic publishing system. Somehow, like the cyberlibertarian utopia of the open web suddenly reverted to the dreadful control of every means of communication by the GAFAM, many articles have shown that a limited number of commercial companies are in a position to control the entire academic knowledge production workflow through strategic mergers and acquisitions in the open science context. This led to a number of position papers and initiatives, such as the call for action that Kathleen Shearer published last year¹⁹ providing a number of recommendations to foster bibliodiversity in scholarly communication.

This leads us to consider academic publishing from a different perspective. Our colleague Leslie Chan recently gave a striking presentation at the FORCE2021 conference²⁰ to encourage us to challenge structural racism and systematic biases in knowledge production. In his talk he showed how paywalls are just the most visible part of a system of power in knowledge production that prevents the full adoption of open science practices. In other words, he encourages us to look beyond the different routes to open access, beyond the business models that support it, beyond the publications themselves and to start questioning the knowledge infrastructure that lies beneath the surface, including peer review practices, editorial practices and also the way in which the standards, the tools and the technology, nurture a monoculture of thinking in academia and stifle the diversity of epistemes that are essential in a true open science.

As Leslie rightly points out, infrastructures have a crucial role in ensuring diversity or, at the opposite end, providing the backbone for monopolistic strategies leading to epistemic exclusion in scholarly communication. In recent years the role of infrastructures at all levels of open science have become more and more apparent. Because open access publishing is primarily digital publishing, even before being open, the question of sustainability is of utmost importance, but the question of openness applies here as well to ensure that the infrastructure does not recreate enclosures behind the door. It was well put by Geoffrey Bilder, Jennifer Lin and Cameron Neylon, the authors of the Principles of Open Scholarly Infrastructure²¹: ‘Everything we have gained by opening content and data will be under threat if we allow the enclosure of scholarly infrastructures’.

Where do we stand after 30 years of development of open access, 20 years after the B declarations, 10 years after the tipping point? An observer coming back after 30 years of absence from the field would probably be horrified by the level of complexity it has reached since then. The multiplication of economic models, the continuous
creation of new formats, new tools, new platforms as a consequence of permanent
innovation and the intricacies of digital services interacting with each other apparently
transformed academic publishing into a maze where it is easy to get lost.

To better grasp the situation and identify the challenges before us I would like to propose, finally, to take a step back again and consider academic publishing from a different perspective. First, I would like to refer to a book published by one of my colleagues at the French School of Advanced Studies in the Social Sciences (EHESS), Valérie Tesnière. Her book *Au bureau de la revue*²² tells the long history of academic publishing in France during the 19ᵗʰ and 20ᵗʰ Centuries. This long history made of many extraordinary and diverse stories perfectly illustrates the true nature of academic publishing in its diversity, with an incredibly high number of different formats, practices and products constantly renewed throughout the decades. It is an history of constant innovation and creativity in connection with the changing needs of the scholarly community and society itself. This history reminds us that giving access to knowledge through publications is a longstanding effort that started long before the open access movement itself. The relatively short history of open access should be placed in the continuity of the longer history of scholarly communication.

Second, I would like to refer to another book²³ published by another of my colleagues at EHESS, Christian Jacob. In his research Christian studies what he calls places of knowledge, the places of knowledge of these all these objects, tools, frameworks, instruments that support the creation and the circulation of knowledge in various cultures and ages. Christian considers a place of knowledge as intimately linked to its social context and to other places of knowledge. In his view, knowledge cannot be considered independently from the culture and the society it comes. Knowledge should always be understood as part of the social fabric where it finds its meaning. My proposal to you is to consider academic publishing as a network of places of knowledge, a public garden of knowledge with many bifurcating paths, crossroads and meeting places that should be understood as such.

The peculiar situation we are currently collectively living in led many of us to question
the meaning of our activities. What are we doing exactly? Why are we doing it? What
is important? As a social activity, knowledge production is not exempt from these
kinds of questions. Our colleagues running the journals surveyed in the context of
the Diamond study answered, in my opinion, very simply to that question: 'Free
access to scientific information is an essential element for economic and social
development. We have in mind our firm commitment to the democratisation of
knowledge. We should not put a price tag on ideas and public knowledge'. Therefore,
the challenge ahead is quite easy to understand. It is not to guess what the future
of publishing will be. It is not to bet on the future winner of the race between OA models and new technologies. It is not to declare obsolete traditional forms of scholarly communication that are essential to valued scholarly communities.

The challenge ahead in my opinion is rather to efficiently support all those places of knowledge and continue to open them all for the benefit of society. We do need smart policies and strategies that support STEM and SSH, the journals and the books and the proceedings as well and the different languages in which researchers communicate with their peers and with society. We need to have the traditional and native publishing venues supported on their journey towards open access. We need to support those who want to experiment beyond the traditional formats as well as those who continue working on longstanding editorial collections in a cumulative way. We need to support diversity and to preserve it from predation as well. In other words, we need to increase and improve coordination across disciplines, communities and countries. We need to consider ourselves as partners rather than competitors, support each other, learn from one another and consider ourselves as serving the same goal each of us in our own way.

Finally, I think we need a coordinated open science ecosystem where the different ethical and sustainable academic publishing practices are supported equally. They are the building blocks of the knowledge infrastructure, and we are not there yet.

## To go a step further

"The economic model of Diamond publishing is supported on three legs:

* The volunteering coming from researchers themselves, who invest a lot of their time and energy in running their journals, because that is so important for the scientific community.

* The strong institutional support provided by personnel seconded to these editorial venues by the institutions and the research-performing organisations.

* Funding, coming from grants particularly.

There is therefore in fact an economic model. It has been mostly invisible so far, so one of the outputs of this study was to bring light to this economic model, and now what we need is to have better and more efficient institutional strategies
and policies to support more efficiently and in a more sustainable way this kind
of economic model.”

"The time spent by researchers on managing a scientific journal is necessarily time lost for research in the field. Therefore, it is fundamental that this editorial work be recognized as research work."

"Researchers are undoubtedly at the heart of Diamond Open Access, as pilots, but they are not alone. They are accompanied and supported in their editorial activities by robust and diversified infrastructures, which are sometimes complicated to talk about because of their technical nature. It is necessary to better understand and valorise these infrastructures and the indispensable role they play in the dissemination of knowledge, to support them financially and to pay attention to their governance, so that they best meet the needs of researchers."

References

1. "Budapest Open Access Initiative", 14 February 2002, https://www.budapestopenaccessinitiative.org/read/.

2. Peter Suber, "Bethesda Statement on Open Access Publishing", 20 June 2003, http://legacy.earlham.edu/~peters/fos/bethesda.htm.

3. "Berlin Declaration", 22 October 2003, https://openaccess.mpg.de/Berlin-Declaration.

4. "Open Access to Research Publications Reaching "Tipping Point"", European Commission, 21 August 2013, https://ec.europa.eu/commission/presscorner/detail/en/IP_13_786.

5. "Science Metrix", accessed 16 March 2022, https://www.science-metrix.com/.

6. Ministère de l'Enseignement supérieur, de la Recherche et de l'Innovation, « Baromètre de la science ouverte », enseignementsup-recherche.gouv.fr, 28 January 2022, https://www.enseignementsup-recherche.gouv.fr/fr/barometre-de-la-science-ouverte-83405.

7. "Unpaywall", accessed 16 March 2022, https://unpaywall.org.

8. "Directory of Open Access Journals", accessed 16 March 2022, https://doaj.org.

9. Forschungszentrum Jülich, "Open Access Monitor", 2022, https://open-access-monitor.de/.

10. "Deutsche Forschungsgemeinschaft", accessed 16 March 2022, https://www.dfg.de/en.

11. "Open Access Netherlands", accessed 16 March 2022, https://www.openaccess.nl/en.

12. "PubMed Central", accessed 16 March 2022, https://www.ncbi.nlm.nih.gov/pmc.

13. Najko Jahn, Maximilian Held, Henrieke Walter, "Hybrid Open Access Dashboard", 2020, https://subugoe.github.io/hoad/; Heather Piwowar, Jason Priem, Richard Orr, "The Future of OA: A Large-Scale Analysis Projecting Open Access Publication and Readership", preprint (Scientific Communication and Education, 9 October 2019),
https://doi.org/10.1101/795310.

14. Quentin Dufour, David Pontille, Didier Torny, « Contractor à l'heure de la publication en accès ouvert. Une analyse systématique des accords transformants », 17 December 2020, https://www.ouvrirlascience.fr/contracter-a-lheure-de-la-publication-en-acces-ouvert-une-analyse-systematique-des-accords-transformants.

15. Max Planck Digital Library, "ESAC Transformative Agreement Registry", accessed 16 March 2022, https://esac-initiative.org/about/transformative-agreements/agreement-registry/.

16. Jeroen Bosman et al., “OA Diamond Journals Study. Part 1: Findings” (Zenodo, 9 March 2021), https://doi.org/10.5281/zenodo.4558704.

17. Stevan Harnad, "A Subversive Proposal", in *Scholarly Journals at the Crossroads: A Subversive Proposal for Electronic Publishing*, ed. A. Okerson and J. O'Donnell (Scholarly Journals at the Crossroads; A Subversive Proposal for Electronic Publishing (01/01/95), Association of Research Libraries, 1995), https://eprints.soton.ac.uk/253351/.

18. Bianca Kramer and Jeroen Bosman, “Innovations in Scholarly Communication - Changing Research Workflows”, Innovations in Scholarly Communication, accessed 16 March 2022, https://101innovations.wordpress.com/.

19. Kathleen Shearer et al., “Fostering Bibliodiversity in Scholarly Communications: A Call for Action” (Zenodo, 15 April 2020), https://doi.org/10.5281/zenodo.3752923.

20. Leslie Chan, “Beyond Diversity and Inclusion: Challenging Structural Racism and Systemic Biases in Academic Knowledge Production”, 7 December 2021, https://doi.org/10.5281/ZENODO.5763937.

21. Geoffrey Bilder, Jennifer Lin, Cameron Neylon, “Principles for Open Scholarly Infrastructures” (Figshare, 2015), https://doi.org/10.6084/m9.figshare.1314859.

22. Valérie Tesnière, *Au bureau de la revue : une histoire de la publication scientifique (xIX-xxᵉ siècle)*, En temps & lieux, Vol. 103 (Paris : EHESS. École des hautes études en sciences sociales, 2021).

23. Christian Jacob, « Qu'est-ce qu'un lieu de savoir ? Qu'est-ce qu'un lieu de savoir ? », Encyclopédie numérique (Marseille: OpenEdition Press, 2014), http://books.openedition.org/oep/423.
# New forms of publication. Why? How? How far?

Jessica Polka
ASAPbio

I am going to talk about new forms of publications from a perspective within the life sciences, but I think the quote that I am about to share applies to all disciplines of scholarship: “Science is not article-shaped” (Louise Page, former Publisher of PLoS). I think that we have been pushing science for centuries into a format that can be copied by a printing press, mailed out to a library to then sit on a shelf. I acknowledge, as Pierre has pointed out, that there are certainly situations in which this is a robust and powerful model, but scholarship can also be quite nonlinear, evolving and it can encapsulate code, data and many other types of media.

All these traits are supported by the potential associated with digital publishing. The web makes it possible to copy, exchange and disseminate information at very low cost. It enables more people to participate in the creation of knowledge and the ownership of the structures that provide that capability, and it enables new versions to be added, for these versions to be tracked over time and for the community to exchange in open dialogue synchronously or asynchronously. Of course, texts can be integrated with other types of objects, like data or code, and it enables free and rapid dissemination, which is a point that I would really like to highlight today.

Even though we have all these amazing capabilities for rapid dissemination, it is actually taking longer than ever to disseminate some forms of science. Looking at data from a life science graduate programme in the US, it is now taking students a year longer to come out with their first first-author paper¹. This is not only holding back the careers of the young scientists, but it also stunts discovery overall. Because science builds on knowledge created by others these delays in publication affect not only the individual paper but every subsequent paper that references it. Why is this happening? One possible reason is that over the same period the number of panels – not figures, but the panels – in many journals has increased dramatically, even when we ignore the new phenomenon of supplemental data. This is widespread.

We are collecting more data, but publishing slower. Why is this happening? I believe the answer lies in the fact that traditional publishing bundles together three different
functions: the dissemination of science, peer review and curation. A journal does not disseminate work until it has peer reviewed it and decided that it is worthy in a sense to be disseminated under its name and to achieve the sort of prestige that is associated with it. As a result of this authors are incentivised not to share until they build up a big story that will get published in an impactful journal.

The need to bundle these three functions, especially this prestige that will determine funding and jobs, is understandable in a world where it is necessary to print on paper and ship out copies, but we can change this now. I want to talk today about four different approaches to doing that, the first being micropublications. They are a form of publication which are explicitly short. The best-known example is probably microPublication.org², which has over 400 outputs, but other experiments like PLOS Currents³ and Science Matters⁴ also fall into this category. In this model researchers are explicitly encouraged to submit short works. They are peer reviewed and published in a standard way. There is no need for an overarching story to tie together many findings and the outputs are recognised in databases like PubMed and Europe PMC.

A second innovation is preregistration and registered reports. Preregistration is reporting the design of a planned study in an open registry, sharing the methods ahead of time. Registered reports⁵ go a step further. Over 300 journals have agreed to review and publish papers based on the design of the study and its methodology, not the findings. This removes selections for only positive results and bases the publication of the paper, and therefore its prestige or perceived prestige, on the design, not the findings. However, it may not be suitable for all types of outputs, especially those where there is a rapid iteration between hypothesis and experiment.

A third innovation takes the article beyond the PDF as researchers are sharing data, code and visualisations that cannot be printed in a traditional article through the web. Many journals have been hosting supplemental files for many years, but some are going a step further than that. eLife has launched a feature called executable articles⁶. This allows readers to play with the code and regenerate the figures during the paper itself as they are reading it, which is built with Jupyter Notebooks and R Markdown. Many journals do not directly support these objects, so researchers must publish them separately and possibly link them from the publication. There are therefore researchers creating custom websites to host and visualise complicated data and sometimes this is released at the time of journal publication, but it can also be done before.

That brings me to the fourth innovation I want to talk about today, which is the publish, then review model. It separates the functions of publishing. Scholars share
information prior to peer review and the functions of peer review and curation can then be performed later. This dissemination can be in the form of code on GitHub, data in a repository, open lab notebooks or preprints, which are articles posted to a public server prior to peer review, and these have gained significant momentum in recent years. A preprint server, rather than performing peer review, allows the papers to undergo a lightweight screening process to ensure the manuscript is in fact scholarly work. While it does not have to be, this manuscript is often the same one that is submitted for peer review at a journal. While that paper is undergoing possibly multiple rounds of review over months, or even years, the paper on the preprint server can be updated with new versions and it is open for the entire community to discuss and provide feedback on, and it is freely accessible to anyone around the world.

Most people are posting preprints around the time of journal submission, but the property of versioning enables preprints to be used in a new way, which is really to share work well ahead of journal submission. All of these are good reasons for why in the last eight years or so nearly 400 000 life sciences preprints have been posted. Life scientists are late to the game. Preprints have been around for over 30 years in physics and math, and they have also been used extensively by social scientists and economists. This adoption in the life sciences has been helped along by both private and public funders which have adopted policies to formally recognise preprints. This started with the Simons Foundation⁷ in 2016, followed by many other agencies. Recently preprints related to COVID-19 have started appearing on PubMed, along with the fantastic coverage representing preprints on Europe PMC. Last year the journal eLife announced it would move to a preprint-based form of publishing in which it will review only preprints.

I think it is also important to note that, unlike in physics and math, life science preprints are distributed across a range of servers which offer different features, and they are owned by different types of entities, including institutions, societies and commercial publishers. Integration with these publishers is likely introducing new people to pre-printing because they are invited to post when they submit to a journal, but it also raises questions about publisher lock-in, the governance that affects server policies and also the free access to metadata.

Preprints are not adopted equally in all disciplines around the world and in all countries around the world. According to data from prior to the pandemic⁸, there were major disparities in preprints as a fraction of the total scientific outputs from various countries, with some being quite overrepresented and others being underrepresented. Of course, given their ability to rapidly communicate new science, it is no surprise that preprints were extensively used early in the COVID-19 pandemic⁹,
but the preprints as a fraction of the total scientific outputs declined over time. Why is this? One clue might come from a small survey that we conducted in 2020¹⁰ in which we asked researchers and others about their concerns with pre-printing. I do not think this would have come up as a top concern had we done this a year later, but the biggest worry that we found was about premature media coverage of preprints. Researchers are probably sensitive to the fact that there is now a lot more public and media attention to preprints than ever before, and this information can sometimes be taken out of context or amplified in a way that is unhelpful.

One case study to look at here is an example of a preprint that was posted early in the pandemic reporting a connection between COVID-19 and HIV¹¹. This demonstrates that a powerful way to address this potential challenge is to surface expert feedback, comments and peer reviews publicly on the preprints. Within hours of posting this preprint, dozens of researchers posted comments on bioRxiv¹² and they weighed in on Twitter, both of which are prominently displayed on the bioRxiv page. Influenced by this feedback, the authors withdrew the preprint within 48 hours and, as pointed out by STAT news¹³, this is much faster than could have been achieved in a traditional workflow. That is an example of how public feedback can lead to quick self-correction. While there is a lot more public feedback happening on preprints than journal articles, a lot of this feedback is happening privately in a way that is not visible and useful to readers. Right now, just 6% of bioRxiv preprints have public comments on the server.

How can we encourage this important activity of reviewing preprints? There needs to be cultural change that embraces this public dialogue. I think there are three ingredients for this. There need to be communities, these communities need to be visible and they need to be recognised. Fortunately, there are already many peer review communities in the life sciences which together have created over 3 000 reviews on bioRxiv preprints. I should mention that the life sciences journal eLife is also posting those reviews directly on preprints, along with other journals, and have done this thousands of times. These reviews are also starting to be more visible. On bioRxiv, certain reviews are visible alongside the preprint and there are two initiatives, eLife's Society¹⁴ and EMBO's Early Evidence Base¹⁵, that act as aggregators for this type of information. Importantly, because there is so much innovation and experimentation happening in all these different models, it is important to have terminology and to understand the difference between them.

These models fit into the publish, then review model in different ways. For example, Review Commons, which we launched together with EMBO, does a single round of review and then it transfers this refereed preprint to affiliate journals, who agree not to restart the peer review process from scratch. These journals then finish the peer
review process, typically using the same reviewers, and then decide to publish the
article, effectively performing curation according to this more traditional model. On
the other hand, Peer Community In¹⁶, which is supported by the French National Open
Science Fund, operates in communities in many different disciplines and performs
the entire peer review process. It is also compatible with the traditional system as
journals can choose to publish the article or not, and the journals then perform a
curation role. However, PCI issues a recommendation for the article itself, so that
is its own form of curation, and excitingly this is starting to be formally recognised.
There are 18 doctoral schools in France that have recognised PCI outputs equivalent
to journal articles¹⁷.

Moving forward, what can we do to help this ecosystem develop? I think we need to explicitly invite listing non-peer reviewed outputs on CVs as evidence of productivity. There is already a lot of momentum for preprints, but code, data and open notebooks also need recognition. Furthermore, we need to recognise review of these preprints and other outputs, and we also need to support new models of feedback and curation, which includes moving towards some kind of agreement on what and how to formally recognise.

## To go a step further

> "Frequently, various information concerning the protocol, the data or the results of a study are published in open access on various platforms and in various forms. A crucial objective to enhance the value of these different works is to manage to link them, and preprint servers play a fundamental role in this direction. The OpenScience Framework, for example, provides authors with a very intuitive interface when submitting a preprint, allowing them to include links to all the works associated with their manuscript. This kind of reciprocal linking would greatly benefit from being generalized to all bibliographic databases."

> "Micropublications are most useful for scientific communities in which the deposit of research data in disciplinary repositories is already a common practice. The micropublications associated with these repositories allow them to be valorised by giving them the form of scientific articles, which allows them to be better known."
References

1 Ronald D. Vale, "Accelerating Scientific Publication in Biology", Proceedings of the National Academy of Sciences 112, n° 44 (3 November 2015): 13439-46, https://doi.org/10.1073/pnas.1511912112; see also Radames J. B. Cordero et al., "Life Science's Average Publishable Unit (APU) Has Increased over the Past Two Decades", PLOS ONE 11, n° 6 (16 June 2016): e0156983, https://doi.org/10.1371/journal.pone.0156983.

2. "microPublication", accessed 16 March 2022, https://www.micropublication.org/.

3. "PLOS Currents", accessed 16 March 2022, http://currents.plos.org/.

4. "Science Matters", accessed 16 March 2022, https://www.science-matters.eu/.

5. Center for Open Science, "Registered Reports", accessed 16 March 2022, https://www.cos.io/initiatives/registered-reports.

6. eLife, "Executable Research Articles", eLife (eLife Sciences Publications Limited, 29 June 2021), https://elifesciences.org/collections/d72819a9/executable-research-articles.

7. "Simons Foundation", accessed 16 March 2022, https://www.simonsfoundation.org/.

8. John P. A. Ioannidis, "Meta-research to transform science", https://drive.google.com/file/d/1mRxiAepW3nJfbKsggWprq0yo0r85vBCK/view.

9. Ludo Waltman et al., "Scholarly Communication in Times of Crisis: The Response of the Scholarly Communication System to the COVID-19 Pandemic", report (Research on Research Institute, 6 December 2021), https://doi.org/10.6084/m9 figshare.17125394.v1.

10. ASAPbio, "Preprint Authors Optimistic about Benefits: Preliminary Results from the #bioPreprints2020 Survey", ASAPbio (blog), 27 July 2020, https://asapbio.org/biopreprints2020-survey-initial-results.

11. Prashant Pradhan et al., "Uncanny Similarity of Unique Inserts in the 2019-NCoV Spike Protein to HIV-1 Gp120 and Gag" (bioRxiv, 31 January 2020), https://doi.org/10.1101/2020.01.30.927871. This article has been retracted.

12. "bioRxiv", accessed 16 March 2022, https://www.biorxiv.org.

13. Ivan Oransky and Adam Marcus, "Quick Retraction of a Faulty Coronavirus Paper Was a Good Moment for Science", STAT (blog), 3 February 2020, https://www.statnews.com/2020/02/03/retraction-faulty-coronavirus-paper-good-moment-for-science/.

14. eLife, "Home", Sciety, accessed 16 March 2022, https://sciety.org/groups/elife/lists.

15. EMBO, "Early Evidence Base", accessed 16 March 2022, https://eeb.embo.org/refereed-preprints/review-commons.

16. "Peer Community In", accessed 16 March 2022, https://peercommunityin.org.

17. "PCI Network", accessed 16 March 2022, https://peercommunityin.org/pci-network.
Academic Publishing and
Latin America

Arianna Becerril-García

Universidad Autónoma del Estado de México/Redalyc

It is well known that South America enjoys a stronger and more coherent ecosystem around non-commercial scientific publishing. I want to mention a few important points about this ecosystem, its networks, the challenges we face, and the future we see for all of this in our work.

First, I would like to talk about what scientific publishing means in Latin America, beyond the numbers, the business models and the sustainable models. I would like all of us to keep in mind that in Latin America, scientific publishing is being researched, reflected upon, and there is a dynamic community. These objectives, which are eagerly sought at the international level, are already being achieved in Latin America. We organize annual congresses and conventions about publishing, we are innovative, we are promoting developments. There are challenges and opportunities, but also threats to this system. However, it is a very special system that has been around for some time and has a durability that other models of scientific publishing do not have.

It is in fact an ecosystem set up by the scientific community where journals are developed by researchers, professors and university and academic staff, but with the very important peculiarity that it is not only a voluntary work, but a prestigious work, widely recognized. The most prestigious journals are directed by the most eminent researchers. This publishing work is part of a research ecosystem, which generates more knowledge. Of course, young researchers during their studies also participate, as well as all those who wish to train themselves by participating in this ecosystem, so that knowledge can be transferred through this exchange between generations. These links are very intricate and complex, but essential to the maintenance of this ecosystem.

The interesting thing about Latin America is that we have not moved from academic publishing to commercial publishing. This has not happened yet, although it is happening with some projects. The Latindex platform¹ currently hosts over 30,000 journals, of which over 11,000 are active online. About 2,000 of these are journals that have been recognized for their high quality by a variety of criteria on different
platforms. Around scientific publishing, there is a lot of work, as Pierre Mounier mentioned about the three pillars that support open access, but in Latin America it is much more complex than that. Scientific publishing does not only mobilize researchers, academic staff and national agencies, although their support is essential to the system. It also mobilizes different organizations, platforms and infrastructures that complement the work of universities. We have platforms such as Latindex, Redalyc² and SciELO³ that examine the quality of scientific journals and suggest ways to improve their editorial quality to meet international standards, without this being experienced as an extra workload for the journals. There are also organizations such as CLACSO⁴ that have created communities around open access and have worked to integrate it into national policies, particularly in the context of scientific evaluations. This is a very complex ecosystem in which the roles are well distributed in a way that generates benefits for all. This means benefits for Latin American researchers, but also for the world. Latin American journals are publishing more and more articles from many foreign countries, all for free.

This makes it even more important for this system to be recognized for its true value, as it is run completely free of charge, with immediate open access and on a not-for-profit basis, by actors who belong overwhelmingly to the academic world. Among them are about 100 universities and dozens of platforms, which implies several challenges in terms of collaboration, especially in terms of the agreements that need to be made to support this ecosystem. It is therefore particularly important that this work and involvement be recognized.

Unlike in other regions, open access publishing in Latin America is the subject of a very coherent ecosystem, where scientific publication is not only a means of communicating science, but also a research object. Many individuals, platforms, and institutions dedicate their attention and energy to it.

Thus, science communication can be considered a true common good, and not only on the scale of Latin America taken in isolation. The ecosystem we have put in place can also act in favour of multilingualism, bibliodiversity, immediate open access, immediate involvement of the academic community. Everyone can have the opportunity to participate in debates, to become informed, in a way that works towards greater epistemic justice, inclusion, greater participation, and greater equity, because there are no barriers to reading, no barriers to publishing.

Technology is an important central axis to bring scientific publishing to a higher level of maturity. In this, the Diamond Open Access (OA) invites us to reflect on the best way to avoid that science becomes a mere commercial commodity. The first
Latin American transforming agreement has just been concluded in Colombia: it is important to keep in mind that these agreements do not only transform scientific publication, but science itself, turning it from a common good into a mercantile object. They also transform the way we spend scarce resources, replacing capital expenditures with one-time expenditures in the case of Article Processing Charges (APCs), for example. They also transform what we consider valuable and important, what we want to control. Who is going to control the scientific communication? Who is going to own the data published in scientific journals? All these aspects are really transformed by transforming agreements.

This transformation also affects what we can consider to be open. In Latin America, we consider everything that is shared in a public way as open, in the framework of common investments, of a global cooperation for the same cause. Transformative agreements lead us to believe that open access must be paid for. Yet Diamond OA allows the scientific community to own, control and drive scientific research. It does not force the elimination of the copyright retention strategy, because there is no upfront payment for copyright. Diamond OA does not impose any of the restrictions sometimes hidden or retroactively imposed by commercial open access. These restraints include embargoes, constraints on the choice of the place of deposit (in particular, for the deposit of preprints in institutional repositories) or on the search of texts and data, etc. Indeed, it often happens that resources available in open access with APCs cannot in fact be freely searched, whereas these methods are essential for the development of new metrics and new points of view on the evaluation of research.

Academic community-driven Diamond OA also allows for improved interoperability and executability of resources, to provide new tools for information discovery and exploration, for optimizing editorial workflows, for research evaluation, for linking open data to publications, etc. All these benefits are not found in other scientific communication models.

It is important to emphasize that this model is not limited to Latin America but is being extended to other countries. The collaboration of these countries with various Latin American institutions and infrastructures allows them to develop an open access that remains firmly rooted in the academic community, without drifting into the commercial side of this activity.

One example is a cooperation initiated between Mexico and Angola around Diamond OA, which forces the question of how a state with limited resources can best invest in its universities to create appropriate mechanisms for scientific communication and encourage the creation of national platforms and repositories, as well as, more
importantly, how to sustain in the long term and strengthen a collective work around
this ecosystem of non-commercial open access.

It is important to recognize that across regions there are different levels of maturity of Diamond OA. The Latin American system is robust, coherent, and dynamic, thanks to the participation of the entire academic system: researchers, journal publishers, governments, institutions, open infrastructures, repositories, libraries, national and regional networks (which often mandate open access repositories), etc. In Latin America, this entire system was created before the concept of "open access" even appeared. Thus, the obligations for open access did not exert any coercion since open access was already perfectly obvious.

The evolution of scientific communication towards commercial open access in other countries of the world, however, has forced us to explicitly impose on the research community to publish in open access, when in fact it was our natural way of doing things. In the era of APCs or commercial open access, such obligations serve primarily to ensure that researchers opt for non-commercial open access.

It is important to understand that it is possible to do things differently, and to start
from the basis of scientific publication and its needs. In other parts of the world,
Diamond OA is less robust, more dispersed, where creating a scientific journal is
primarily a matter of individual will and less of institutional support, unlike in Latin
America.

I would also like to emphasize that it is essential to support this open access economically. It is very healthy to invest in non-commercial scientific communication, to engage as an economic market player, to fight against the constitution of oligopolies to keep this activity under the control of the academic world. In this way, unlike in some other parts of the world, we can avoid dependence on commercial markets, commercial open access journals with APC or restricted access journals with subscription. Open access managed by the academic community acts as a natural regulator of commercial markets, so it is important to strengthen it and to identify the issues at stake, so that we do not lose our control over scientific communication.
<figure><img src="image_30.png" /><figcaption>Fig. 6: Dimensions for strengthening Diamond OA</figcaption></figure>

Source: Becerril García, « Academic Publishing and Latin America ». Media used during the presentation (available on the conference's website).

We have many challenges to meet in order to strengthen academic open access, in connection with a few important axes: identify the different levels of maturity of this Diamond OA around the world, innovate to better integrate open access, move towards more interoperability, collaborate more closely, articulate open access and research evaluation, think about the sustainability of this model, its modes of governance... Research policies must keep in mind that open access journals can also be high quality journals, with contents that are often more robust than those of commercial journals.

It is also important to integrate Diamond OA in libraries, research tools and educational platforms, whose users must not be forgotten when talking about open access to scientific knowledge. It is necessary that all collaborate to disseminate scientific publications more widely throughout the world.

It is fundamental to recognize that it is possible to have a quality scientific publication on a non-profit basis, without charging APCs, in a cooperative approach where everyone benefits from each other's investments. The financial funds invested are for the benefit of all, not just the person who invested the funds. 15% of the articles indexed in Redalyc are funded by donors from other parts of the world.
The transition to Diamond OA can also allow reinventing scientific communication, not only to fill existing gaps, but to build or strengthen knowledge as a common good.

This is an ideal time to return to the essence of science, which promotes epistemic justice and methodological, linguistic, geographic and content diversity, to combat the current biases and inequalities in the system and to envision a future where Diamond OA piloted by the academic community has much to contribute.

## To go a step further

> "If we want to encourage countries where open access is still beginning, the challenges to be met concern first and foremost research itself, which must be developed in a multilingual perspective. In Latin America, too, the resources devoted to scientific research and publication are low, and the commercial stakes are sometimes very delicate. It is important not to ignore them, even in countries where scientific communities are only beginning to be structured."

> "Continuous innovation is a necessity if we want to make science a real common good. It is also crucial to ensure that academic publishing is better valued, at a time when the supposed prestige of some commercial journals continues to capture the attention of researchers."

## References

1. "Latindex", accessed 16 March 2022, https://peercommunityin.org/pci-network.

2. "Sistema de Información Científica Redalyc", accessed 16 March 2022, https://www.redalyc.org/.

3. "Scientific Electronic Library Online", accessed 16 March 2022, https://scielo.org.

4. "Consejo Latinoamericano de Ciencias Sociales", accessed 16 March 2022, https://www.clacso.org.
# An Action Plan for Diamond Open Access

Lidia Borrell-Damián
Science Europe

Johan Rooryck
cOAlition S

**Lidia Borrell-Damián**: We came up with this idea of an Action plan based on a study commissioned by cOAlition S, funded by Science Europe, and run by OPERAS (cf. infra), because we thought that the moment was right to create a community that would strengthen the Diamond Open Access (OA) sector. We thought that we could support this community to achieve a system that is better organised, coordinated and sustainably funded to support these researchers in their dissemination efforts. Moreover, we wished to provide the means for the diamond sector to develop its full potential in the context of open science and in the context of the scholarly communication system at large. We also thought that the moment was right at a practical level to develop this community to provide a common set of actions to better define a communication ecosystem and the infrastructure for scholars.

We wish to further aid Diamond OA journals and platforms that share the same principles, guidelines and quality standard. We want to advance the capacity of those journals and platforms to provide increased and improved services. It is a well-rounded initiative that we think is perfect to advance scholarly communication.

It has been defined by many speakers before me. Diamond OA means free of charge for authors and readers. It is community-driven, academic-led and –owned, which is very important, and since there is no fee, it provides equitable access for all, which is of tremendous importance for the results of research. It also involves the concept of bibliodiversity that is in development, and it respects all large- and small-scale multilingual and multicultural scholarly communities.

We now know much more, but with this in mind, about one year ago Science Europe issued a call for tenders to conduct a study on Diamond OA because we wanted to understand the landscape. We wanted to know more about this system that is so fragmented, and therefore so difficult to bring together in a way that provides a space
and a coherent voice to those concerned in the progress of the scholarly publishing
system. A consortium led by OPERAS¹ was the winner of that. I will mention a few key
results of this very extensive study².

What is important in this study is that they surveyed 1 619 journals, together with focus groups and interviews, which indicated that there are around 17 000 to 29 000 journals. 60% of them are in social sciences and the humanities, 17% in medicine and 22% in other scientific disciplines. Also, what is important to note from this study is that about 44% of the articles published in open access are published in diamond, so it is a non-neglectable size. Moreover, that represents 8-9% of the total publishing volume, compared to 10% of what represents the Gold Article Processing Charges. I think this is a very important figure because it signals the importance of a sector that normally has been underestimated. As Pierre himself confessed, this is not even a fully exhaustive study taking account of everything. It is possible that there is even more data that would give us increased and improved information, and we know that there are other researchers now taking this forward. This puts the importance of this sector of scholarly publishing into context.

What this study identified is that there are four main challenges. One is purely
technical. You will hear more about it. Another is about the management. A third one
is about the visibility, which we just addressed. A fourth one is about sustainability,
which some of my colleagues have also addressed. This is a comprehensive way of
presenting the challenges of the Diamond OA sector.

There are several recommendations to address these challenges. One major one is that indeed we need more dialogue and commitment between researchers, research funding organisations, research-performing organisations, university libraries, university presses and departments, scholarly societies, and ministries for all the reasons that you have heard very well presented by Arianna. It was on this basis that we decided to convene a group of experts in a meeting two days ago, with the support of the Ministry of Higher Education, Research and Innovation in France to precisely provide a shape to these four challenges. The study is available, and it is of course in open access. You can seek it out for more information. There are many more recommendations. It is a very in-depth study and I highly recommend that you read it. My colleague Johan will now take over and present to you the principal activities that we have decided to undertake to try to meet these four main challenges.

**Johan Rooryck**: As Lidia said, we came together a few days ago to discuss an action plan for Diamond Open Access and the discussion among the experts was specifically
focused on four central elements: efficiency, quality standards, capacity-building and sustainability. I will review those to tell you what we alighted on.

The first element is efficiency. We think that Diamond OA journals and platforms would benefit from sharing several common resources. That means aligning quality standards, in a flexible way of course. This is because you must to take into account various traditions across disciplines and various cultures, and research cultures as well. We need to create sustainability and build trust for all stakeholders via shared infrastructure standards, policies, practices and funding streams. That also means making the technical services more accessible, more interoperable and more streamlined. We know that the technical capacity of the existing diamond journals varies a lot from one journal to the next, and this is something that we would like to assist editors and journals with.

We would also like of course to create synergies between diamond journals in the same disciplines, in the same languages, in the same geographical locations, according to their needs via existing organisations, groups and societies. Basically, what we want is to federate and align in a flexible way without imposing standards, but by inviting everyone to participate and co-create these new standards.

The second element is quality standards. Quality of course needs to be flexibly aligned, like I said. It is not something where one standard fits everyone. However, we need to specify quality standards for the core components of publishing operations and services. That is in terms of, for instance, reviewer policies or author guidelines, or in terms of what a reviewer can expect, what a reader can expect, what an author can expect when they submit a paper to a diamond journal. At least there we should have some guidelines that are relatively identical across disciplines and languages and that can be flexibly aligned. That is why we want to align nontechnical standards, non-technological standards, and best practices for open access publishing into what we have called temporarily a European Quality Standard for Institutional Publishing. We also want to develop a self-assessment tool that can allow journals and editors to assess the quality standards for diamond journals to assist progress, to allow journals to progress towards this quality standard.

Thirdly, we want to focus on capacity-building. We want to enhance editorial and management skills across the various disciplines, languages, and countries by creating tools for open access publishing in diamond, training materials for institutional editors and service providers, quality standards for journals, authors and review policies and guidelines that will all become available from a common access point. To that end we
would like to create a capacity centre for diamond publishing that provides technical, financial, and training services for these journals and platforms.

Finally, there is the issue of sustainability, and this was already mentioned by various speakers before me, namely we need to investigate the revenue streams, the legal status and the governance of Diamond OA journals and platforms. That needs to be stabilised. For instance, we need to ensure that the operational costs of diamond publishing are carried by a network of institutional actors, like Arianna mentioned in the presentation before me. We need to develop a coordinated financial mechanism that will deliver various sources of funding both to the central capacity centre and the individual journals, infrastructures, and platforms. We must make sure that the legal status of these various journals clearly states that they are scholar-led, scholar-owned and –governed and that their titles are protected in a legal manner. This also requires that we seek a better understanding of the costs of Diamond OA and promote transparent financial and administrative good practices to inform all stakeholders. Clearly it is not just about open access, but it is also about open governance here.

In conclusion, as mentioned earlier, this action plan was developed by the participants in the workshop on 2 February. This workshop was organised by Science Europe in collaboration with cOAlition S³, OPERAS and ANR⁴. It was sponsored by the French Ministry of Higher Education, Research, and Innovation in the context of this Open Science European Conference. What is important and encouraging news is that a consortium of 23 organisations was awarded a capacity-building grant for 3 million by Horizon-Widera⁵ to develop institutional publishing and diamond publishing in Europe. That will certainly help towards realising this action plan. However, of course we need much more participation from various organisations, not only across Europe, but also across the world, and this is why we will launch an appeal in the next few weeks to sign this action plan to the various organisations and journals who are willing to support it. We certainly need a community of organisations worldwide to bring this action plan for Diamond Open Access forward. We are planning a conference in June 2022 to discuss the implementation of that action plan.

## To go a step further

"The agenda for the next two years is to achieve a consolidated network of research centres, funders, researchers, research infrastructures and service providers willing to contribute to the consolidation of the diamond sector. This is a very healthy option for academic scholarly publishing." (L. Borrel-Damián)
"We would also hopefully have a much better overview of the data of all the journals that exist, what their quality criteria are, how they are subsidised and how they function so that we are able to have a better understanding of this intricate landscape." (J. Rooryck)

"A quality-certified journal is first and foremost a researcher-run and owned journal that does not require money from either readers or authors. Some services such as the DOAJ can provide a more accurate assessment of such journals." (J. Roorick)

"All disciplines are affected by Diamond Open Access. It is simply a matter of researchers changing the way they think about the dissemination of their work and taking ownership of that dissemination." (L. Borrell-Damián)

## References

1. "OPERAS", accessed 16 March 2022, https://www.operas-eu.org/.

2. Jeroen Bosman et al., "OA Diamond Journals Study. Part 1: Findings" (Zenodo, 9 March 2021), https://doi.org/10.5281/zenodo.4558704; Becerril, Arianna et al., "OA Diamond Journals Study. Part 2: Recommendations" (Zenodo, 9 March 2021), https://doi.org/10.5281/ZENODO.4562790.

3. "cOAlition S", accessed 16 March 2022, https://www.coalition-s.org.

4. "Agence nationale de la recherche", accessed 16 March 2022, https://anr.fr.

5. "Horizon-Widera", accessed 16 March 2022, https://ec.europa.eu/info/funding-tenders/opportunities/portal/screen/opportunities/topic-details/horizon-widera-2021-access-03-01.
<figure><img src="image_31.png" /></figure>
Research Assessment
– First Session
<figure><img src="image_32.png" /></figure>
# The Intersections between DORA, Open Scholarship and Equity

Stephen Curry

Imperial College London/DORA

In 1969, the BBC broadcast a landmark television series called *Civilisation*¹. It was one of the first television series to be broadcast in colour. In it, the art historian Kenneth Clark traced the rise of Western European civilisation through an examination of its art through the centuries. In an early episode he observed that there had been three or four times in history where Man had made a leap forward that would have been unthinkable under ordinary evolutionary conditions. He is thinking of the 12th Century in Europe as one of these times, when there was an eruption of architectural creativity, which created treasures like Notre Dame. He also observed that this forward leap was due to the rise of confidence in the future that is needed to push through a long-term project. Without wishing to put too much weight on it, I think we are at a similar moment in history regarding research assessment and open science.

The evolution of this opportunity that I think we face now and that this conference is discussing was identified even back in 1999 in a paper by Michael Gibbons where he was analysing the interactions between science, industry and government². He recognised back then the need for a new contract to ensure that scientific knowledge was socially robust and relevant to society. That required its production to be both transparent and participative. This paper was published before the terms ‘open access’ or ‘open science’ were in common parlance, but in fact the whole paper is infused with the idea of openness. It is the realisation of that vision that I think we are here today to try and push forward.

The ideas that Gibbons talked about have started to enter the development of public policy. There have been several key documents since then that have talked about research assessment and open science. In the beginning, and the San Francisco Declaration on Research Assessment³ (DORA) was one of the first movers here, policy and recommendations around research assessment did not explicitly talk about open science. It is not really mentioned in DORA, in the Leiden Manifesto⁴, the Metric Tide⁵, or in the Hong Kong Principles⁶. However, it is in more recent pronouncements and policy documents, such as in a paper for the Global Research Council⁷, that the European Commission Scoping Report⁸ helped to set the scene for this conference. In
the UNESCO recommendations on open science⁹ that we heard about this morning, the intertwining of research assessment and open science is seen as essential. I think we need to develop a new vision in evaluating science if we are to move forward with this.

When we talk about what we value about science, and we have already heard quite a bit on this subject today, then I think it is interesting to take on board the thinking of philosopher Michael Sandel¹⁰. In a talk he gave in 2018, he primarily reflects on politics and society as it is now, but he has observed, and I think he is right to do so, the outsourcing of public policy by many governments to commercial organisations. This has been particularly intense in the 1980s in the UK and the US, but I think this is also reflected in the EU project. This has led to a diminishment of the idea of a public good, and a loss of faith in or sight of the collective responsibility for overall wellbeing. I think it is these ideas that we now want to see brought back, not just into society, but into the academy.

We have seen that what has market value in academia now is very much driven by metrics. This is an infusion of the ideas of marketisation into public policy around higher education and research. There is a legitimate demand from governments and public funders for accountability from the academy. But market value in academia has got tied up with the over over-metrication of research assessment and that has led to many unintended and harmful effects. We have a hypercompetitive environment that slows down publication and that can lead to biases in important clinical trial analyses, and we see a culture in which many academic activities are devalued. There is stress on academics as a result. There is a focus on the “what”, not the “who” or the “how” or indeed or the “why”. There is incentivisation within the system of fraud which is undermining public trust. That is something that we really need to guard against.

Within DORA we have a different vision of what research values should look like. I think these values are widely shared. We have heard many similar expressions, and one can see that this resonates very strongly with what we read in the UNESCO report. We are looking for reliable, rapidly communicated, high-quality research that transforms our understanding of the world and can change it for the better. For this, we need researchers who collaborate and feel a duty to care for their group members. We need a research system that values the people within it, cares about their quality of life and seeks out the creative vigour of diversity.

Now, this is a very grand vision. The hard part is to think about how we implement this vision. This is where I see the intersection between research assessment and open science. I do personally believe that we need to talk about how open science can