# Open comments on the "Open Letter: Establishing a national research software award"

Teresa Gomez-Diaz, Tomas Recio

<figure><img src="image_1.png" /></figure>

## To cite this version:

Teresa Gomez-Diaz, Tomas Recio. Open comments on the "Open Letter: Establishing a national research software award". 2024. hal-04545553

HAL Id: hal-04545553

https://hal.science/hal-04545553

Preprint submitted on 14 Apr 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
Open comments on the “*Open Letter:*
*Establishing a national research software award*”

Teresa Gomez-Diaz (University Gustave Eiffel-LIGM-CNRS, France),
Tomas Recio (University Antonio de Nebrija, Spain)

Contact: Teresa.Gomez-Diaz@univ-eiffel.fr, trecio@nebrija.es

This work is licensed under the Creative Commons Attribution 4.0 International License (CC-BY 4.0)

http://creativecommons.org/licenses/by/4.0/

January 11th, 2024, V1

# 1 Foreword

The goal of this document is to openly contribute with our comments to the Open Letter [1] *Establishing a national research software award* published in Open Research Europe, that has received two Referee reports [2, 3].

We hope that they can contribute to enrich the discussion on the involved subjects. Moreover, we think that these suggestions could be also useful for the authors to react to the Referees' comments when they discuss and ask for more precision regarding both the concept of research software presented in the Open Letter, and the proposed evaluation criteria and steps.

# 2 Introduction

As the authors of the references [11, 12, 14, 15, 13, 16, 17], we would like to express our acknowledgment of this interesting French initiative that will undoubtedly help to build an Open Science ecosystem in which research software finds its right place as a research production.

This award will help to convey best dissemination practices [11, 12] such as, for example, to put a free/open source license in the software before its dissemination. It will also help to increase the visibility of research software produced within French (research) institutions and establish research software models to be followed, which will help to install and expand an Open Science culture.

It will also establish a model to be followed, which is already in its beginning pace with the Australian award mentioned at the end of the Open Letter [1].

On the other hand, we would also like to offer some comments on this Open Letter. We hope that they can contribute to enrich the discussion on the involved subjects. Moreover, we think that these suggestions could also be useful for the authors to react to the Referees' comments when they discuss and ask for more precision regarding both the concept of research software presented in the Open Letter, and the proposed evaluation criteria and steps.

# 3 Comments

**Definition of Research Software** In the *Defining research software* section of [1], it is stated:

> research software is software designed, maintained, and/or used by scientists and/or research institutions. It is developed to meet a specific need of science, hence it results from research work and/or enables scientific work, which is notably valued by publications before/on/around/with the software.
Open comments on the “*Open Letter:*
*Establishing a national research software award*”

Teresa Gomez-Diaz (University Gustave Eiffel-LIGM-CNRS, France),
Tomas Recio (University Antonio de Nebrija, Spain)

Contact: Teresa.Gomez-Diaz@univ-eiffel.fr, trecio@nebrija.es

This work is licensed under the Creative Commons Attribution 4.0 International License (CC-BY 4.0)

http://creativecommons.org/licenses/by/4.0/

January 11th, 2024, V1

# 1 Foreword

The goal of this document is to openly contribute with our comments to the Open Letter [1] *Establishing a national research software award* published in Open Research Europe, that has received two Referee reports [2, 3].

We hope that they can contribute to enrich the discussion on the involved subjects. Moreover, we think that these suggestions could be also useful for the authors to react to the Referees' comments when they discuss and ask for more precision regarding both the concept of research software presented in the Open Letter, and the proposed evaluation criteria and steps.

# 2 Introduction

As the authors of the references [11, 12, 14, 15, 13, 16, 17], we would like to express our acknowledgment of this interesting French initiative that will undoubtedly help to build an Open Science ecosystem in which research software finds its right place as a research production.

This award will help to convey best dissemination practices [11, 12] such as, for example, to put a free/open source license in the software before its dissemination. It will also help to increase the visibility of research software produced within French (research) institutions and establish research software models to be followed, which will help to install and expand an Open Science culture.

It will also establish a model to be followed, which is already in its beginning pace with the Australian award mentioned at the end of the Open Letter [1].

On the other hand, we would also like to offer some comments on this Open Letter. We hope that they can contribute to enrich the discussion on the involved subjects. Moreover, we think that these suggestions could also be useful for the authors to react to the Referees' comments when they discuss and ask for more precision regarding both the concept of research software presented in the Open Letter, and the proposed evaluation criteria and steps.

# 3 Comments

**Definition of Research Software** In the *Defining research software* section of [1], it is stated:

> research software is software designed, maintained, and/or used by scientists and/or research institutions. It is developed to meet a specific need of science, hence it results from research work and/or enables scientific work, which is notably valued by publications before/on/around/with the software.
This definition is unclear as it seems to us that it does not delimit a precise subset of software, as
software used by scientists and/or research institutions include many different layers of software at
the very instant a scientist opens a computer or a phone for its every day work. Would every word
processor, used to write scientific papers, qualify as research software? A much more precise definition
would be something like:

> research software is software designed and developed by scientists to meet a specific need of science, hence it results from research work and/or enables scientific work, which is notably valued by the publication of the scientific results obtained with the software.

This means that the software that is qualified as research software is a product of the research work,
coinciding thus with the research software definition provided in section 2.1 of [14]:

> research software is a well identified set of code that has been written by a (again, well identified) research team. It is software that has been built and used to produce a result published or disseminated in some article or scientific contribution. Each research software encloses a set (of files) that contains the source code and the compiled code. It can also include other elements as the documentation, specifications, use cases, a test suite, examples of input data and corresponding output data, and even preparatory material.

The authors of the Open Letter should clarify the precise formula proposed for the research software
concept and put it in perspective with the definition provided in [14]. We would like to remind here
that much of the software developed by scientists or by research institutions is simply not maintained,
then this qualification should be avoided in the definition, otherwise it will be gambling to define an
almost empty set.

As not every software produced by scientists is disseminated as free/open source software, or simply disseminated, this should not be a characteristic that enters in the definition of research software, but in the software that is considered within the award context. As remarked by the Referee [3], this point could benefit of more insight about how it was dealt with the software without a license.

**Evaluation criteria** The Referee [3] comments: * *Establishing a national award: I see opportunities for facilitating the evaluation of activities related to the production of research software as a research activity, which could be important for a more fair evaluation of research activities in the context of open science.* This point is also remarked by the Referee [2]: *The Evaluation Criteria section (page 6) could benefit from more explicit information.* Which also coincides with the point: * *Evaluation criteria* of [3].

We also agree with both Referees, and this presentation of the French national research software award will gain in interest if the evaluation criteria could be disclosed. Among other benefits, it will provide a more transparent context that will help potential candidates to improve the conditions to present new research software to the future editions of the award.

In particular we would like to compare these award criteria with the CDUR evaluation protocols proposed in section 4 of [14]. CDUR stands for Citation, Dissemination, Use and Research, and proposes these four steps in order to evaluate different aspects of the research software. We submit here a precise question to the Open Letter authors: how the three considered levels “strong support, support, do not support” are related to the *basic/good/excellent levels* that we do study in the section 4.3 CDUR: a use case step-by-step of [14].

**Standing on the shoulders of Giants** As this French research software award shows, the situation of research software in France is in a privileged status, where many important points related to this research production are in discussion at the Ministry level. But to arrive to this situation has taken different and previous stages that have not occur in other countries or that are currently in construction, which also shows the French idiosyncratic situation.

Among all the pre-existing work, we would like to mention the French DevLOG network<fnref n="1" /> that gathers
the software development community. It was launched in 2009 with a website and a mailing list, and

<footnote n="1">¹https://www.devlog.cnrs.fr/</footnote>
This definition is unclear as it seems to us that it does not delimit a precise subset of software, as
software used by scientists and/or research institutions include many different layers of software at
the very instant a scientist opens a computer or a phone for its every day work. Would every word
processor, used to write scientific papers, qualify as research software? A much more precise definition
would be something like:

> research software is software designed and developed by scientists to meet a specific need of science, hence it results from research work and/or enables scientific work, which is notably valued by the publication of the scientific results obtained with the software.

This means that the software that is qualified as research software is a product of the research work,
coinciding thus with the research software definition provided in section 2.1 of [14]:

> research software is a well identified set of code that has been written by a (again, well identified) research team. It is software that has been built and used to produce a result published or disseminated in some article or scientific contribution. Each research software encloses a set (of files) that contains the source code and the compiled code. It can also include other elements as the documentation, specifications, use cases, a test suite, examples of input data and corresponding output data, and even preparatory material.

The authors of the Open Letter should clarify the precise formula proposed for the research software
concept and put it in perspective with the definition provided in [14]. We would like to remind here
that much of the software developed by scientists or by research institutions is simply not maintained,
then this qualification should be avoided in the definition, otherwise it will be gambling to define an
almost empty set.

As not every software produced by scientists is disseminated as free/open source software, or simply disseminated, this should not be a characteristic that enters in the definition of research software, but in the software that is considered within the award context. As remarked by the Referee [3], this point could benefit of more insight about how it was dealt with the software without a license.

**Evaluation criteria** The Referee [3] comments: * *Establishing a national award: I see opportunities for facilitating the evaluation of activities related to the production of research software as a research activity, which could be important for a more fair evaluation of research activities in the context of open science.* This point is also remarked by the Referee [2]: *The Evaluation Criteria section (page 6) could benefit from more explicit information.* Which also coincides with the point: * *Evaluation criteria* of [3].

We also agree with both Referees, and this presentation of the French national research software award will gain in interest if the evaluation criteria could be disclosed. Among other benefits, it will provide a more transparent context that will help potential candidates to improve the conditions to present new research software to the future editions of the award.

In particular we would like to compare these award criteria with the CDUR evaluation protocols proposed in section 4 of [14]. CDUR stands for Citation, Dissemination, Use and Research, and proposes these four steps in order to evaluate different aspects of the research software. We submit here a precise question to the Open Letter authors: how the three considered levels “strong support, support, do not support” are related to the *basic/good/excellent levels* that we do study in the section 4.3 CDUR: a use case step-by-step of [14].

**Standing on the shoulders of Giants** As this French research software award shows, the situation of research software in France is in a privileged status, where many important points related to this research production are in discussion at the Ministry level. But to arrive to this situation has taken different and previous stages that have not occur in other countries or that are currently in construction, which also shows the French idiosyncratic situation.

Among all the pre-existing work, we would like to mention the French DevLOG network<fnref n="1" /> that gathers
the software development community. It was launched in 2009 with a website and a mailing list, and

<footnote n="1">¹https://www.devlog.cnrs.fr/</footnote>
it organised the JDEV2011, the *1ère édition des Journées nationales du DEVeloppement logiciel*<fnref n="2" /> in
Toulouse (see also the other JDEV editions<fnref n="3" />). This network was launched by some developers, members
of the research community that were already part of the team of the French PLUME project (2007-
2013) [15]. The platform where this project published software descriptions was closed in 2022, which
is surely the reason for not to mention it in the Open Letter, nevertheless, descriptions of the project
goals, and of the work realized by the PLUME team can be found in [14, 15]. Readers interested to have
more insight about this project can consult [5, 6, 7, 8, 9], and [10] (work related to years 2007-2013),
and the video [4]. Among all the publications produced within the PLUME project we would like to
highlight [13], and the already mentioned [11], others are available at the mentioned link [10].

**Data award** A possible new version of this Open Letter could also mention the Data award also organized by the French Minister of Higher Education and Research in 2022⁴.

Again, we would be interested to have more knowledge about the review criteria process that has been
used in this Data award, as we would like to compare it with our proposed evaluation protocols for
research data [16, 17].

**Note** The present comments are also submitted to the Comment section of the Open Letter [1].

**Acknowledgements** We would like to thank the authors of the Open Letter for such an inspiring work.

## References

[1] Blanc Catala I, Di Cosmo R, Giraud M et al., (2023). Establishing a national research software award [version 1; peer review: 2 approved] Open Research Europe 2023, 3:185 https://doi.org/10.12688/openreseurope.16069.1

[2] Ram K, (2023). Peer Review Report For: Establishing a national research software award [version 1; peer review: 2 approved]. Open Research Europe 2023, 3:185 https://doi.org/10.21956/openreseurope.17349.r35884

[3] González Barahona JM, (2023). Peer Review Report For: Establishing a national research software award [version 1; peer review: 2 approved]. Open Research Europe 2023, 3:185 https://doi.org/10.21956/openreseurope.17349.r35882

[4] Archimbaud JL, (2007). Video de la présentation du projet Plume au CC-IN2P3, Lyon. Webcast CC-IN2P3. https://webcast.in2p3.fr/video/presentation-du-projet-plume

[5] Archimbaud JL, Romier G, (2009). La place des logiciels libres dans l'Enseignement Supérieur et la Recherche (ESR), état des lieux à travers PLUME. Et que font les autres ? https://hal.science/hal-00559446

[6] Archimbaud JL, Romier G, (2010). Bilan à trois ans de PLUME : les services rendus aux laboratoires de recherche et aux universités. https://hal.science/hal-00559460

[7] Archimbaud JL, (2011). PLUME : Promouvoir les Logiciels Utiles Maîtrisés et Économiques dans l'Enseignement Supérieur et la Recherche. 10e colloque national en calcul des structures, May 2011, Giens, France. https://hal.science/hal-00592935

[8] Archimbaud JL, (2012). La place du logiciel libre au CNRS : témoignage d'un ingénieur en informatique au CNRS depuis 1984. https://hal.science/hal-00850874

<footnote n="2">²https://archive-devlog.cnrs.fr/jdev2011</footnote>

<footnote n="3">³https://www.devlog.cnrs.fr/jdev/</footnote>

<footnote n="4">⁴https://www.enseignementsup-recherche.gouv.fr/fr/prix-science-ouverte-des-donnees-de-la-recherche-86179, https://www.ouvirlascience.fr/remise-des-prix-science-ouverte-des-donnees-de-la-recherche-2023/</footnote>
it organised the JDEV2011, the *1ère édition des Journées nationales du DEVeloppement logiciel*<fnref n="2" /> in
Toulouse (see also the other JDEV editions<fnref n="3" />). This network was launched by some developers, members
of the research community that were already part of the team of the French PLUME project (2007-
2013) [15]. The platform where this project published software descriptions was closed in 2022, which
is surely the reason for not to mention it in the Open Letter, nevertheless, descriptions of the project
goals, and of the work realized by the PLUME team can be found in [14, 15]. Readers interested to have
more insight about this project can consult [5, 6, 7, 8, 9], and [10] (work related to years 2007-2013),
and the video [4]. Among all the publications produced within the PLUME project we would like to
highlight [13], and the already mentioned [11], others are available at the mentioned link [10].

**Data award** A possible new version of this Open Letter could also mention the Data award also organized by the French Minister of Higher Education and Research in 2022⁴.

Again, we would be interested to have more knowledge about the review criteria process that has been
used in this Data award, as we would like to compare it with our proposed evaluation protocols for
research data [16, 17].

**Note** The present comments are also submitted to the Comment section of the Open Letter [1].

**Acknowledgements** We would like to thank the authors of the Open Letter for such an inspiring work.

## References

[1] Blanc Catala I, Di Cosmo R, Giraud M et al., (2023). Establishing a national research software award [version 1; peer review: 2 approved] Open Research Europe 2023, 3:185 https://doi.org/10.12688/openreseurope.16069.1

[2] Ram K, (2023). Peer Review Report For: Establishing a national research software award [version 1; peer review: 2 approved]. Open Research Europe 2023, 3:185 https://doi.org/10.21956/openreseurope.17349.r35884

[3] González Barahona JM, (2023). Peer Review Report For: Establishing a national research software award [version 1; peer review: 2 approved]. Open Research Europe 2023, 3:185 https://doi.org/10.21956/openreseurope.17349.r35882

[4] Archimbaud JL, (2007). Video de la présentation du projet Plume au CC-IN2P3, Lyon. Webcast CC-IN2P3. https://webcast.in2p3.fr/video/presentation-du-projet-plume

[5] Archimbaud JL, Romier G, (2009). La place des logiciels libres dans l'Enseignement Supérieur et la Recherche (ESR), état des lieux à travers PLUME. Et que font les autres ? https://hal.science/hal-00559446

[6] Archimbaud JL, Romier G, (2010). Bilan à trois ans de PLUME : les services rendus aux laboratoires de recherche et aux universités. https://hal.science/hal-00559460

[7] Archimbaud JL, (2011). PLUME : Promouvoir les Logiciels Utiles Maîtrisés et Économiques dans l'Enseignement Supérieur et la Recherche. 10e colloque national en calcul des structures, May 2011, Giens, France. https://hal.science/hal-00592935

[8] Archimbaud JL, (2012). La place du logiciel libre au CNRS : témoignage d'un ingénieur en informatique au CNRS depuis 1984. https://hal.science/hal-00850874

<footnote n="2">²https://archive-devlog.cnrs.fr/jdev2011</footnote>

<footnote n="3">³https://www.devlog.cnrs.fr/jdev/</footnote>

<footnote n="4">⁴https://www.enseignementsup-recherche.gouv.fr/fr/prix-science-ouverte-des-donnees-de-la-recherche-86179, https://www.ouvirlascience.fr/remise-des-prix-science-ouverte-des-donnees-de-la-recherche-2023/</footnote>
[9] Gomez-Diaz T, (2007-2013). Le thème PLUME Patrimoine logiciel d'un laboratoire, Zenodo Community, https://zenodo.org/communities/plume-patrimoine-logiciel-laboratoire/

[10] Gomez-Diaz T, (2007-2024). Page web with presentations, articles, posters and other productions since 2007, Laboratoire d'informatique Gaspard-MOnge (LIGM), http://igm.univ-mlv.fr/~teresa/logicielsLIGM/

[11] Gomez-Diaz T, (2010). Diffuser un logiciel de laboratoire : recommandations juridiques et administratives, Publication for the French PLUME project (2007-2013), https://zenodo.org/record/7096216

[12] Gomez-Diaz T, (2014). Free software, Open source software, licenses. A short presentation including a procedure for research software and data dissemination, Zenodo, Preprint, Septembre 2014, http://zenodo.org/record/11709/. Presented at the Workshop on open licenses: Data licencing and policies, EGI Conference 2015, Lisbon, May 2015, https://indico.egi.eu/event/2452/sessions/1522/#20150520. Spanish version: Software libre, software de código abierto, licencias. Donde se pone un procedimiento de distribución de software y datos de investigación, Zenodo, Preprint, Septembre 2015, https://zenodo.org/record/31547/

[13] Gomez-Diaz T, (2015). Article vs. Logiciel : questions juridiques et de politique scientifique dans la production de logiciels. 1024 - Bulletin de la société informatique de France, N. 5, 2015, http://www.societe-informatique-de-france.fr/wp-content/uploads/2015/04/1024-5-gomez-diaz.pdf, also available at https://hal.science/hal-01158010. First version published for the French PLUME project (2007-2013) in 2011, https://zenodo.org/record/7063154

[14] Gomez-Diaz T and Recio T, (2019). On the evaluation of research software: the CDUR procedure [version 2; peer review: 2 approved]. F1000Research 2019, 8:1353, https://doi.org/10.12688/f1000research.19994.2

[15] Gomez-Diaz T, (2019). Le Projet PLUME et le paysage actuel des logiciels de la recherche dans la science ouverte. Zenodo, Preprint, 2019. https://zenodo.org/record/2591474

[16] Gomez-Diaz T and Recio T, (2022). Research Software vs. Research Data I: Towards a Research Data definition in the Open Science context, F1000Research 2022, 11:118, https://f1000research.com/articles/11-118/v2

[17] Gomez-Diaz T and Recio T, (2022). Research Software vs. Research Data II: Protocols for Research Data dissemination and evaluation in the Open Science context, F1000Research 2022, 11:117, https://f1000research.com/articles/11-117/v2
[9] Gomez-Diaz T, (2007-2013). Le thème PLUME Patrimoine logiciel d'un laboratoire, Zenodo Community, https://zenodo.org/communities/plume-patrimoine-logiciel-laboratoire/

[10] Gomez-Diaz T, (2007-2024). Page web with presentations, articles, posters and other productions since 2007, Laboratoire d'informatique Gaspard-MOnge (LIGM), http://igm.univ-mlv.fr/~teresa/logicielsLIGM/

[11] Gomez-Diaz T, (2010). Diffuser un logiciel de laboratoire : recommandations juridiques et administratives, Publication for the French PLUME project (2007-2013), https://zenodo.org/record/7096216

[12] Gomez-Diaz T, (2014). Free software, Open source software, licenses. A short presentation including a procedure for research software and data dissemination, Zenodo, Preprint, Septembre 2014, http://zenodo.org/record/11709/. Presented at the Workshop on open licenses: Data licencing and policies, EGI Conference 2015, Lisbon, May 2015, https://indico.egi.eu/event/2452/sessions/1522/#20150520. Spanish version: Software libre, software de código abierto, licencias. Donde se pone un procedimiento de distribución de software y datos de investigación, Zenodo, Preprint, Septembre 2015, https://zenodo.org/record/31547/

[13] Gomez-Diaz T, (2015). Article vs. Logiciel : questions juridiques et de politique scientifique dans la production de logiciels. 1024 - Bulletin de la société informatique de France, N. 5, 2015, http://www.societe-informatique-de-france.fr/wp-content/uploads/2015/04/1024-5-gomez-diaz.pdf, also available at https://hal.science/hal-01158010. First version published for the French PLUME project (2007-2013) in 2011, https://zenodo.org/record/7063154

[14] Gomez-Diaz T and Recio T, (2019). On the evaluation of research software: the CDUR procedure [version 2; peer review: 2 approved]. F1000Research 2019, 8:1353, https://doi.org/10.12688/f1000research.19994.2

[15] Gomez-Diaz T, (2019). Le Projet PLUME et le paysage actuel des logiciels de la recherche dans la science ouverte. Zenodo, Preprint, 2019. https://zenodo.org/record/2591474

[16] Gomez-Diaz T and Recio T, (2022). Research Software vs. Research Data I: Towards a Research Data definition in the Open Science context, F1000Research 2022, 11:118, https://f1000research.com/articles/11-118/v2

[17] Gomez-Diaz T and Recio T, (2022). Research Software vs. Research Data II: Protocols for Research Data dissemination and evaluation in the Open Science context, F1000Research 2022, 11:117, https://f1000research.com/articles/11-117/v2