[Citation needed] Data usage and citation practices in medical imaging conferences

Authors: théo  sourget
ahmet  akkoç
stinna  winther
christine lyngbye galsgaard
amelia  jiménez-sánchez
dovile  juodelyte
caroline  petitjean
veronika  cheplygina
Venue: MIDL_io_2024_Conference
Published: 2024-01-29 15:32:58

Abstract: Medical imaging papers often focus on methodology, but the quality of the algorithms and the validity of the conclusions are highly dependent on the datasets used. As creating datasets requires a lot of effort, researchers often use publicly available datasets, there is however no adopted standard for citing the datasets used in scientific papers, leading to difficulty in tracking dataset usage.
In this work, we present two open-source tools we created that could help with the detection of dataset usage, a pipeline using OpenAlex and full-text analysis, and a PDF annotation software used in our study to manually label the presence of datasets. We applied both tools on a study of the usage of 20 publicly available medical datasets in papers from MICCAI and MIDL. 
We compute the proportion and the evolution between 2013 and 2023 of 3 types of presence in a paper: cited, mentioned in the full text, cited and mentioned. Our findings demonstrate the concentration of the usage of a limited set of datasets. We also highlight different citing practices, making the automation of tracking difficult.

Main paper: 
 introduction: 
 While the increased usage of open data is a positive development, we hypothesize it might introduce a shift in the targeted applications. For example, (Varoquaux and Cheplygina, 2022) show that since the Kaggle lung cancer challenge in early 2017 (Buckeye et al., 2017) , there has been a disproportionate increase in machine learning papers on lung cancer, while many of the proposed solutions do not differ in practice. A similar concentration on fewer datasets has also been found in machine learning (Koch et al., 2021 ). Another medical imaging example is the segmentation of cardiac ventricles, addressed with multiple competitions (Bernard et al., 2018b; Suinesiaputra et al., 2012; Petitjean et al., 2015; Campello et al., 2021) . The latest competition achieved highly accurate results and commercially available software exists (Wu et al., 2024) , yet the application still remains popular for evaluating novel algorithms. Moreover, while the availability of a public dataset is a positive step towards getting a problem addressed by the community, the choice of a single dataset for evaluation also results in an overestimation of performances leading to a gap when applied on a different one (Wu et al., 2021) .There is a need to analyse research within a field to understand the progress being made, but next to surveys focused either on methods (Litjens et al., 2017; Cheplygina et al., 2019; Budd et al., 2021) or on datasets within a specific application (Daneshjou et al., 2021; Wen et al., 2022) , we find few studies on understanding dataset use beyond their initial release in the field. We believe this is in part due to identifying dataset usage, as datasets may be used without corresponding citations, and vice versa. Our contributions, aiming to achieve this identification of dataset usage are as follows: (1) We present a fully automated pipeline for quantifying dataset usage based on the analysis of references and the paper full text. (2) We present an open-source annotation tool which allows for validation of the method, and can aid in tracking dataset usage in research papers. (3) We apply both tools to study the usage of several popular segmentation and classification datasets and their usage in MICCAI and MIDL conference papers between 2013 and 2023. (4) We discuss the limitations of our study and tools, display additional practices we found during our study, and provide recommendations to ease the tracking of datasets. 
related work: 
 Meta-research papers in medical imaging often focus on methods, for example surveys on deep learning (Litjens et al., 2017) , different types of supervision (Cheplygina et al., 2019) , human-in-the-loop methods (Budd et al., 2021) and so forth. As a by-product of annotating and categorizing papers, some surveys also provide lists of commonly used datasets (Çallı et al., 2021b) .More recently some dataset-focused reviews started to emerge, in particular for dermatology (Daneshjou et al., 2021; Wen et al., 2022) and ophtalmology (Khan et al., 2021) . These reviews focus on the type of data that is available, and find various biases in the patient populations, and/or that metadata about the patient demographics is missing. However these papers do not examine dataset use.Perhaps at the intersection of datasets and methods, there is work focusing on challenges (Eisenmann et al., 2022) which review participation in medical image competitions at MIC-CAI and ISBI. Such competitions are often seen as one of the drivers of publicly available datasets, but the impact of these datasets beyond these competitions is not known.The closest to our work are studies that examine dataset usage in other published works. (Koch et al., 2021) analyse dataset usage on PapersWithCode across various applications of machine learning, and find that the diversity of datasets used is decreasing. Within medical imaging, Heller et al (Heller et al., 2019) examined the role of publicly available data in MICCAI papers between 2014 and 2018, and found among others that over 20% of papers using public data did not cite the dataset. Simkó et al (Simko et al., 2022) examined reproducibility in MIDL papers between 2018 and 2022 and found that papers using public datasets are becoming more common but without proper citations or links. 
quantifying medical image dataset use: 
 We propose tools to evaluate the presence of datasets using the following definitions: a dataset is cited if its paper is present in the reference section, mentioned if its name, aliases or URL are in the abstract or in a section of the paper associated with the method or results (i.e., not in a related work or discussion sections), in a table or figure captions or in a footnote, showing an actual usage. We show our pipeline in Figure 1 . There are four main components: user input about the list of venues and datasets to track, the open citation index tool OpenAlex (Priem et al., 2022) , the full texts of the papers and GROBID, a tool to extract information from scholarly documents. We used OpenAlex because it has an official freely accessible API aggregating and standardizing information from multiple sources such as arXiv, Crossref and Pubmed, and we aimed to create a generalizable process that could be complemented with other tools. We compared it with OpenCitations to evaluate their completeness using the citations returned for a set of cardiac segmentation datasets. We found that 97% of the citations returned by OpenCitations were also returned by OpenAlex while only 84% returned by OpenAlex were returned by OpenCitations; thus we chose OpenAlex for our pipeline. First, we ask the user to specify the list of venues and datasets to track. This includes a dataset name, any aliases referring to the same datasets, and the titles and DOIs of papers associated with these datasets.We use the venue list to fetch the list of papers from these venues using the DBLP API and store the papers' titles and DOIs. We then use the paper DOI (or title if the DOI is not available) to query the OpenAlex API to get the following: (i) list of referenced papers, (ii) list of words in the abstract, and (iii) open access link to the paper's full text, an example of data from OpenAlex is shown in Appendix D. We then try to fetch the paper's full text. If this step fails, we complement this step with a custom scraping tool. This step can be easily replaced for different venues, as long as the paper PDFs are stored in the same folder, In this step some data cleanup may be necessary to remove duplicates created from the combination of PDF extracted with OpenAlex and with the scraping tool. We then convert the PDF to an XML file using GROBID, a library applying a cascade of machine learning models that first segment the document in different structures like header, full text or reference and then specific models tuned for each type of structure to extract content. This allows to detect different paper sections, while keeping information about figures, tables and footnotes, which were lacking in alternative tools such as PyPDF, an example of the data obtained with GROBID is shown in Appendix E.We use the dataset list to detect their citations and mentions. We detect citations in two ways: based on the dataset's DOI converted to an OpenAlex ID, and based on matching the dataset paper titles to the references sections of the full text. We detect dataset mentions by searching for the dataset's name, aliases or URL in certain fields of the full text. Examples of different types of citations and mentions can be seen in Appendix C.Finally, we assign each dataset presence to one of the following types: only cited, only mentioned, and both cited and mentioned. 
annotation tool for paper pdfs: 
 We also present our PDF annotation tool made with Streamlit, a Python library to easily create web apps. We used it to verify our detection process and therefore we designed it to fulfil two needs: having multiple users annotate the same project easily and being able to handle a large number of PDF files. While it was used to annotate datasets' presence in scientific papers, it can be extended to any PDF annotation task.Once the software is installed on a local server, a user can create an annotation project by uploading the PDFs and choosing up to two initial sets of labels. In our study, the first set of labels corresponds to the list of datasets to detect and the second set is the list of locations a mention could be classified into (E.g. Abstract, Introduction, Method). While the second set of labels is fixed, the first one is not and new values can be added at any point during the labelling.We also wanted to ease the annotation by multiple users. At the creation of the project, the owner can upload a file containing the division of the papers into different groups. This way, users can find the papers they were assigned to by selecting the right group on the annotation page. Finally, when the annotations are downloaded from the server, a file per person is obtained allowing more data processing afterwards. 
data selection: 
 We apply our tools to a set of 20 publicly available medical datasets for both classification and segmentation of various organs shown in Appendix A. We initially tried a systematic procedure of identifying datasets via Google datasets and OpenAlex. However, this resulted in many poorly documented datasets (particularly on COVID-19) which did not have distinct names, and of which we could not trace whether they were in part duplicated from other datasets. Therefore, we selected datasets based on a combination of prior knowledge of the authors and consulting recent surveys in medical imaging which provided a table or list of datasets (Hesamian et al., 2019; El Jurdi et al., 2021; Niyas et al., 2022; Qureshi et al., 2022; Çallı et al., 2021a; Guan and Liu, 2021) . In order to obtain enough data to analyse, we took the following aspects into account for the selection: presence of a paper linked to the dataset available in OpenAlex, year of publication, having some citations in OpenAlex, and having a unique name or acronym to make the detection process more reliable. We chose to analyse papers from two major conferences about medical image analysis, MICCAI and MIDL, so that the papers are more likely to contain the presence of such datasets.We identified 4835 papers in total (4569 from MICCAI and 266 from MIDL), however, 44 were discarded as we could not obtain information on the content of the paper or the list of references. We categorize the remaining papers in three groups, where for each group we slightly adjusted our processing due to the missing data:• Group 1 with n=2327 papers either have all all information, or only miss the abstract from OpenAlex. In this case, we analyze the abstract from the full text of the paper.• Group 2 with n=2237 where full text is not available, but we can still detect dataset mentions using the OpenAlex abstract. This is an important limitation, as the abstract does not always mention the datasets used. All the papers in this group are from MICCAI, showing the usefulness of the modular part to obtain the full texts. Unlike MICCAI papers, the structure of the PMLR website and the complete open access of PDFs made possible the development of the scraping tool for all MIDL full texts while they were not accessible from OpenAlex.• Group 3 with n=227 papers which do not have references in OpenAlex. We therefore detect citations only with our simpler matching of dataset papers' titles with Grobid, which may result in less accurate detection. A majority of papers from this group are from MIDL as the information for papers from this conference is almost absent from OpenAlex. This shows that the download and analysis of the full text is a crucial and needed aspect of our method.Although we considered the number of citations in OpenAlex to make the first selection of datasets, some datasets had very low numbers of citations and mentions in MICCAI and MIDL. We only present in Figures 2 and 3 results for eight datasets with the highest usage or that exemplify one of our conclusions, a more complete version including all the datasets can be found in Appendix B. This result may highlight the focus on some particular datasets also shown in (Koch et al., 2021) when using publicly available data, especially for datasets for the same task (cardiac segmentation and chest classification) as ACDC and M&Ms. This is also visible in Figure 2 with the large gap between the count of citations and mentions for BRATS and the rest of the most present datasets. Note the difference in growth between the datasets, which might suggest a snowball effect where popular datasets become even more popular. This seems to be the case for BRATS, ACDC or Chexpert which have a very strong growth in citations and mentions. For other datasets like LIDC-IDRI or DRIVE, the number of citations and mentions is more gradual and even stagnates for DRIVE. Multiple factors can impact the popularity of a dataset, one of the most straightforward is the year of publication but while Chexpert and PadChest have been released at the same time, the second is almost absent from our list of papers. Therefore, other aspects such as how the dataset is updated or has a competition been organized with the dataset could be an explanation for such differences. A dataset's citation doesn't necessarily imply actual usage and not all used datasets are cited in the references section. We further analyze this difference between mention and citations with Figure 3 in which we assign each presence of a dataset in a paper to one of the categories described in Section 3. We found out that even if there is variability in the groups' proportion for each subset, we can observe that almost every subset has more than 25% of datasets being only cited and around 10% being only mentioned. We considered papers from the "Only Cited" group as not using the dataset while citing it in the introduction or related works, mostly for general statements about machine learning usage in the medical sector. However, 132 papers out of 233 miss the full text and therefore only the abstract is used to detect the mention, a fraction of these papers could therefore mention the dataset and use it but the lack of information prevents our tool from detecting it. On the other hand, the "Only Mentioned" group mostly represents papers that are using a dataset without citing the associated paper. These two groups display the lack of standards to indicate the usage of a dataset such that it can be easily tracked. It also supports our approach to analyze part of the full text to determine such a usage. The "Only Cited" group in blue represents papers that cite a dataset without having a mention detected and therefore may not use it. The "Only Mentioned" group in orange represent the bad citation practice as the usage would not be detected by tools tracking the citations. The "Cited and Mentioned" group in green represent the best practice. 
discussion: 
 We presented two open-source tools for the detection of dataset usage in scientific papers and applied them to a case study on publicly available medical datasets. We show that papers in major medical conferences tend to use a limited set of datasets, especially for papers addressing the same task. We also found that the lack of citation standards for dataset usage makes tracking such usage difficult, in particular due to (i) papers citing a dataset's paper without mentioning it in particular sections, indicating a non-usage, and (ii) papers mentioning a dataset without citing its paper, which classical bibliometric tools like OpenAlex can not detect.Our study is limited to a set of datasets and venues manually selected and may therefore be biased by this selection. We also did not try to disambiguate between different datasets versions (for example different years of BRATS or datasets with similar names) due to already having difficulties with identifying these more-easily-identifiable datasets, it could however be valuable information to not overestimate the usage of a dataset or distinguish various tasks present across different version of a dataset. Doing a study on more datasets, venues and tasks would strengthen the outcome of our work. While datasets can be cited but not an associated paper, OpenAlex only keeps track of citations to papers. It is an important limitation and therefore a more precise matching of citations using GROBID could be a solution to track citations without a paper like it can be for Kaggle datasets.Our method relies on regex matching and their location, it makes our tool usable to other data easier as only some information needs to be changed. We did not use text classification methods based on deep learning, such as fine-tuning a model pre-trained on scientific data like (Beltagy et al., 2019) . While this could result in better performances, it implies a fine-tuning for every new set of datasets, reducing the applicability of our tools to new settings.While doing this study we had some anecdotal findings that we do not report on in the paper, but which we feel may warrant further study.• We saw the number of citations a paper has doubled, from 10 to 20, in 2019. This is likely because until 2019 MICCAI used to include citations in the 8-page limit.Relaxing such page restrictions may incentivize authors to not omit dataset citations.• We found many instances of papers associated with Github repositories that were promising the code to be available upon acceptance, but never actually did this.• We found cases where a "backup" of a dataset on Kaggle was cited as if it were the original source. The dataset was often stripped of its metadata and license information, and it was not clear whether the data was exactly the same or a derivative of the original, for a longer discussion please see (Jiménez-Sánchez et al., 2024) .• We discovered that ACDC is a popular name, as it can refer to the Automated Cardiac Diagnosis Challenge (Bernard et al., 2018a) but also to the Adverse Conditions Dataset with images of streets (Sakaridis et al., 2021) or to the Automatic Cancer Detection and Classification in Whole-slide Lung Histopathology challenge (Li et al., 2018) .We believe that better knowledge and therefore easier access to dataset usage information are needed. In addition to giving due credit to the creators of the dataset, it can raise awareness of the overuse of a particular dataset for a task, which could have a negative impact on real performance, but also an over-representation of a task in regards of real clinical needs. Working towards the adoption of a standard for indicating the usage of a dataset seems to be an essential step to achieve this objective. As examples, NeuroImage has a specific section on data availability at the end of each manuscript, and in 2023, MICCAI added the obligation to declare "the data origin, data license, and (when appropriate) ethics application number for any public or private data used in the preparation of the paper". While such requirements will not solve all the issues at hand, we believe that including a "Data availability" section could be an easy solution to put in place that would pave the way towards more systematic ways of determining the usage of a dataset. There are of course still many unanswered questions as to how exactly we want to implement this, for example what to do in cases of derivative datasets, synthetic data, and so forth, which we hope we can discuss together as a community.  
 
 Appendix:  


Decision: The reviewers believe (and I agree) that this paper addresses an important topic for our community. The authors have done a good job addressing the initial reviewer comments, and after the revisions all reviewers have reached a consensus and recommend acceptance of the paper.

Reviews: review: paper_summary: paper_summary: The authors studied citation and mention patterns of medical AI papers and commonly used datasets. The authors analyzed both citations and text of publications to identify datasets that are being used or mentioned.
main_review: main_review: I appreciate the authors' update and explanations to my questions and suggestions. Overall I found the project very interesting and an important research area. I think additional implications of the research results and actionable next steps are important for authors to consider and continue the work on.
strength_weakness: strength_weakness: Whether mentioning a dataset vs using the dataset for train/validation/test could also be an interesting topic that is underexplored in the project.
What would be interesting is also including a way to track emergence of new dataset and their impact on healthcare AI.
questions: questions: Evaluation of OpenAlex's accuracy and completeness in this setting? I may have missed this but seem an important prerequisite.
Evaluation of correlation of multiple datasets used in the same publicatoin?
paper_summary: paper_summary: In this meta-research, the authors investigate the adequacy of data citation and mention practices in medical imaging research papers, proposing a novel pipeline and open-annotation tool for this analysis. Focusing on publications from the MICCAI and MIDL conferences between 2013-2023, they provide an evaluation of citation behaviors. The findings highlight significant insights into the current state and potential areas for improvement in research integrity and data usage acknowledgment within the field.
main_review: main_review: The authors have addressed each of my comments and provided discussions. I've decided to increase my initial score, believing it accurately reflects the paper's quality and merits within the context of its field.
strength_weakness: strength_weakness: Although the terminology starts off seeming straightforward, clarity becomes an issue as the paper progresses, leading to confusion. 

The explanation of external tools used in the study is not sufficient, making the full pipeline difficult to understand.
questions: questions: It would be helpful to the reader if the authors: 
- explicitly define key terminologies such as "usage," "mention," and "citation," outlining their distinct meanings within the context of this paper. 
- Section 5.1 Data Selection par. #4 "All the papers ...", clarify how the modular part (I think) in the additional scraping tool help MIDL paper and why this was not used for MICCAI papers. 
- Figure 2, if possible add "actual usage" to the figures.
- explain briefly in the methods section how OpenAlex and GROBID work. 

What are some of your insights on:
- data derivatives (if the data used is not coming from the original data source) and how they should be documented/cited?
- "dataset accessibility" as a key factor?
paper_summary: paper_summary: The authors present  open-source tools  that could help with the detection of dataset usage; (1)  OpenAlex: a regex based full-text analysis tool, and (ii) a PDF annotation software to manually label the presence of datasets. They applied both tools to analyze MICCAI and MIDL papers in terms of citations of publicly available datasets. They aim to understand the standardization of the citations of public databases.
main_review: main_review: All my review comments are responded by the authors. They also updated the respective section of the paper according to review comments. As all my review comments are answered, I am increasing my score in th
strength_weakness: strength_weakness: The paper does not address the current trend of using DOIs to standardize dataset references.

It is difficult to ascertain the extent of necessary data cleanup and the level of automation achievable within the described pipeline.

The results lack clarity regarding the significance of automating the entire process.
questions: questions: To what extent can the entire process be automated, and how much data cleanup is necessary for the results?
paper_summary: paper_summary: The paper focuses on the usage and citation practices of datasets in medical imaging conferences. The findings demonstrate the concentration of dataset usage on a limited set of datasets and highlight different citing practices, making tracking difficult. The paper also discusses challenges in dataset identification and the impact of publicly available datasets beyond competition. The significance of the research lies in addressing the lack of an adopted standard for citing datasets, providing tools for dataset detection, and shedding light on dataset usage patterns in medical imaging research.
main_review: main_review: Medical imaging papers often focus on methodology, but the quality of algorithms and the validity of conclusions heavily rely on the datasets used. Creating datasets requires substantial effort, so researchers often use publicly available ones. However, there is currently no universally adopted standard for citing these datasets in scientific papers, which makes it challenging to track their usage. This study reveals a concentration in the usage of a limited set of datasets and highlights varying citing practices, complicating the automation of tracking dataset usage. The authors have addressed all of my concerns. I have no further comments.
strength_weakness: strength_weakness: - The study is limited to a manually selected set of datasets and venues, which may introduce bias into the findings.

- The paper acknowledges the difficulty in disambiguating between different versions of datasets, which could impact the accuracy of dataset usage tracking.

- The tools used in the study, such as OpenAlex and the PDF annotation software, may have limitations in detecting dataset usage, as they rely on specific criteria for citation and mention detection.

- The paper does not provide a comprehensive analysis of the reasons behind different citing practices, such as citing a dataset's paper without mentioning it or mentioning a dataset without citing its paper.

- The research does not explore the potential impact of dataset usage on the validity and reproducibility of medical imaging studies
questions: questions: 1 How does one identify between citing a dataset's paper without mentioning it or mentioning a dataset without citing its paper?

2 What is the potential impact of dataset usage on the validity and reproducibility of medical imaging studies

3 What are the challenges in dataset identification?

4 What are the different citing practices?

5 What is the impact of publicly available datasets beyond competition?

6 How does disambiguation affect dataset tracking?

Comments: title: Response to Rebuttal
Thank you
Concerns addressed?
comment: Thank you for your response addressing my concerns and discussions on derivatives.
Thank you for your thoughtful responses to my questions and comments. I do not have further questions.
Thanks for the responses to the review comments. As the authors added the details to the submission I am increasing my score.
The authors have addressed all of my concerns. I have no more comments.
Dear reviewers,

Could you please briefly comment whether your concerns were adequately addressed by the rebuttal, and if necessary or helpful, engage in a discussion with the reviewer? 

Thank you very much, and best wishes,
AC
Dear authors,

Thanks for reaching out, and for posting your rebuttals with ample time. Please note that this year, the "rebuttal" and "discussion" phase are separated, so the discussion phase, where reviewer can interact with authors, has only started four days ago, and will last until 27 March.

However, I do encourage the reviewers to respond to the author comments as soon as possible to enable a constructive discussion.

All the best,
AC
Dear area chair and reviewers,

Thanks again for your input on the paper. It's been 10 days since we revised the manuscript and answered the reviewers' questions. With the discussion period ending next week, we were wondering if there are any responses to our changes, is there anything else we need to clarify?
Thank you for your review. We addressed the comments as follows:

*How does one identify between citing a dataset's paper without mentioning it or mentioning a dataset without citing its paper?* and *What are the different citing practices?*

As also pointed out by another reviewer, we clarified the definitions at the beginning of Section 3. For more intuition on the different citation practices and mentions, we also added examples in Appendix C.

*What are the challenges in dataset identification?* and *How does disambiguation affect dataset tracking?*

The automatic identification of datasets is made difficult by a combination of large variability in names and context even for a single dataset with multiple aliases, and sometimes some close names between different datasets for example as we mentioned for COVID-19 datasets. It’s therefore complicated to know if a name is just another alias of a dataset or actually a completely different one.

This problem also affects the disambiguation of datasets, a similar name between different datasets may lead to the overestimation of the usage of the one being tracked. Similarly, as different versions of a dataset may also contain different tasks or solve potential problems such as a lack of diversity in a previous version, the disambiguation could be valuable information. 

We extended this part of the discussion to include these remarks.

*What is the potential impact of dataset usage on the validity and reproducibility of medical imaging studies?* and *What is the impact of publicly available datasets beyond competition?*

The dataset can indeed impact multiple aspects, for example, the generalization of models trained and evaluated on a single dataset but with a gap in performances when applied to other datasets of the same task. Therefore, while the availability of a public dataset is a positive step towards getting a problem addressed by the community, if this dataset becomes the only point of evaluation, then performance may eventually be overestimated.

We modified the introduction to add this problem.
Thank you very much for your review. 

Regarding the automation of the process, it was designed to be easily used for other datasets and venues without modifying the code, therefore the main parts of the data collection and the detection of mentions/citations are automated.

The following information is gathered automatically:

- Get the list of paper of selected papers from DBLP
- Get information from OpenAlex
- Download full texts using links from OpenAlex
- Convert PDF to XML with GROBID
- Detect citations & mentions from OpenAlex data & GROBID

The following actions need to be performed by the user (also answering your question about data cleanup):

- Input the list of datasets and venues
- Gather additional full texts for which no links were available in OpenAlex, or removing duplicate PDFs.

To give an example, if users want to perform the same analysis but including the papers from MIDL 2024, they would have to change the maximum year in venues.csv and could use the same scraping tool without code modification but by adding the volume number of MIDL 2024 in the command line to launch it.

Overall, no code modification is needed for the data collection and processing except for an optional scraping tool that users could define to fit their needs. Note that OpenAlex may be sufficient to obtain the full text and therefore such a scraping too may not be necessary.

We added details about the data cleanup in section 3.
Thank you very much for your review. 

*It would be helpful to the reader if the authors:*

- *explicitly define key terminologies such as "usage," "mention," and
"citation," outlining their distinct meanings within the context of this paper.*
- *Section 5.1 Data Selection par. #4 "All the papers ...", clarify how the modular part (I think) in the additional scraping tool help MIDL paper and why this was not used for MICCAI papers.*
- *Figure 2, if possible add "actual usage" to the figures.*
- *explain briefly in the methods section how OpenAlex and GROBID work.*

To improve clarity we did the following:

We rearranged Section 3 to include the definitions at the beginning and we also added some examples of citations and mentions in Appendix C.

We also added more explanation on the modular part and why we did it for MIDL and not MICCAI in section 5.1

Unfortunately, the number of papers being too high (2521 papers) we did not screen all of them it’s therefore not possible to add the actual usage in the figure, however while developing the method we saw that the heuristics we use (for example a dataset mentioned in the methods section) often indicates actual use. 

We added a small description of OpenAlex (aggregating information from other sources) and GROBID (a cascade of models to extract information) in Section 3. We also added an example of data from both tools in Appendix D and E in case it helps to understand the process. 

We hope the changes make the paper and the process easier to understand. 

*What are some of your insights on:*

- *data derivatives (if the data used is not coming from the original data source) and how they should be documented/cited?*

This is a great question, it is difficult to say something general about this because there are many different degrees of derivatives. For example, the LUNA16 challenge uses selected cases from the LIDC-IDRI dataset, however, papers may still talk about the “LUNA16 dataset” and only refer to the challenge website. Since the challenge website clearly documents that the data is based on LIDC-IDRI, and the data license allows it, only describing the actual data used and then citing LUNA16 seems sufficient. 

In other cases it is less clear. Suppose the experiments use two independently released derivatives B and C, of some original dataset A. Then it would be important to document that both derivatives were based on A (and therefore to cite it), since there might be overlaps in B and C. But at some point, if multiple original datasets were involved, this might become impractical.  

Due to space limitations it is difficult to go in-depth into this point at the paper but it is clearly an important question that we should discuss as a community, we added a note about this to the Discussion section. 

- *"dataset accessibility" as a key factor*

One of our original questions was to study what factors lead to increased dataset use, and likely availability/accessibility is strongly linked to this. Downloading a (copy of) a dataset from Kaggle is simpler than filling in a data agreement. But while this increases accessibility/use, we feel it might decrease our capacity to build knowledge, since in this process many details about the data can be lost - see for example another recent work from our lab https://arxiv.org/pdf/2402.06353.pdf, we added a note about this in the Discussion.
Thank you very much for your questions. To start with the second one about the correlation of multiple datasets, we indeed computed it with a co-occurrence matrix. We found very low numbers of papers with such co-occurrences, and both due to the low sample size and space limitations, we decided to not include this result in the paper.

Regarding the evaluation of OpenAlex, it’s indeed an important prerequisite, especially for the detection of citations. However, we mitigate this dependence with the citation detection in our tool using GROBID and it could be more developed to reduce this risk. 

We also compared OpenAlex with another open citation index tool named OpenCitations. We compared the citations returned for our set of datasets on cardiac segmentation. From this comparison, 97% of the citations returned by OpenCitations were also returned by OpenAlex. On the other hand, only 84% of the citations returned by OpenAlex were returned by OpenCitations.

We updated the paper to include the comparison with OpenCitation at the end of the first paragraph of section 3.
Thank you to all the reviewers. We uploaded a revised version of the paper and we will shortly answer your individual questions. To summarize, the main changes are:

- We clarified the different terms (citation, mention and usage) in Section 3. We also added some examples of citations and mentions in the Appendix.
- We added a brief description of OpenAlex and GROBID and explained why we chose OpenAlex. We also included an example of data from OpenAlex and GROBID in the appendix.
- We added some notes in the discussion about derivative datasets and the impact of disambiguation.
- We removed Table 1 with the number of papers in different groups, and described this within the text, to comply with the page limit.
- Other minor text changes to comply with the page limit.