TITLE: Semi-automated extraction of information  on open datasets mentioned in articles
AUTHORS: Anastasiia Iarkaeva, Evgeny Bobrov, Jan Taubitz, Benjamin Gregory Carlisle, Nico Riedel

[DESCRIPTION]
This protocol describes how to determine for a body of research articles, whether underlying datasets have been openly shared. Statements on shared data are detected within articles using the ODDPub text mining algorithm, and are then further processed using an openness extraction form implemented in Numbat. This extraction form was developed to guide and document the manual validation of automatedly detected Open Data statements. For one article, several datasets are checked, one per dataset location. The extraction form consists of checks of data availability and reusability, loosely inspired by the FAIR principles. The resulting table gives an overview of, amongst others, dataset location, applied license, and data formats. Data sharing in supplements, data reuse and restricted data sharing are also documented as alternatives to open data.
Operationalization of Open Data criteria has been described in the article preprint Operationalizing Open Data – Formulating verifiable criteria for the openness of datasets mentioned in biomedical research articles (Bobrov et al., 2023).

[STEPS]
SECTION: Automated validation of publications for statements indicating Open Data using ODDPub
1. Collate a list of article identifiers
Begin with a list of articles identifiers (usually DOIs), for which you want to assess the openness of the datasets underlying these articles. These idetifiers can typically be obtained by searching publication databases (e.g. Pubmed, Web of Science, OpenAlex, Dimensions, Embase) using the relevant search criteria and exporting the results with the relevant metadata fields.

Search criteria may include institutional affiliations of authors, specific research fields, specific journals, and typically, a publication date range. If the focus is on articles from a particular institution, a curated list of publications might be available directly from the institution (often provided by the institutional library).

The development and optimization of ODDPub (Open Data Detection in Publications) algorithm have been aligned with Open Data criteria, similar to those cited below. However, some changes in the criteria have been introduced over time.
 

SECTION: Automated validation of publications for statements indicating Open Data using ODDPub
2. Obtain the article full texts
There are several options for this:
via PubMed Central (Open Access articles)
via  unpaywall API: the full-text links (Open Access articles).
via Publisher APIs: for full-text retrieval of subscription-based articles, offered by several major publishers, like Elsevier, Wiley, or Springer/Nature
via full-text R package: a solution that combines several of those data sources for downloading full texts. 
These retrieval options are limited to articles that can be accessed as Open Access or through the subscription provided by the institution where the article are retrieved.

Store all retrieved article full-texts in a single folder either as PDFs (preferred for current ODDPub version) or as text files (e.g. XML).
SECTION: Automated validation of publications for statements indicating Open Data using ODDPub
3. Apply ODDPub to article full texts
 

The following steps describe how to use the ODDPub package in R.
SECTION: Set up extraction workflow in Numbat - Admin access
7. Load the article DOIs and detected statements into Numbat

SECTION: Manual validation using Numbat - User access
5. Preparation for manual extraction of Open Data status

For a detailed walkthrough of the manual extraction process of Open Data, see the instruction below under 5.1-5.4.

To access the publication, use the DOI directly on doi.org website or enter the URL http://doi.org/ + DOI into browser's address bar, e.g. http://doi.org/10.1128/mBio.02755-20.

Please note that while this example may be straightforward, actual extraction can be time-consuming. Some cases may require extensive searching and checking, and occasionally more than one dataset may need to be extracted per article.
     

SECTION: Different issue handling in Numbat - Admin access
9. The extraction process up to this point has been linear. Steps 9.1 to 9.4 outline optional procedures. Sometimes, it may be necessary to correct errors (9.1), or add one or more datasets to the extraction list. A new article entry can be uploaded to the existing Numbat list (9.2). If an extractor is unsure about the Open Data status of a dataset, the extraction can be reassigned to another extractor (9.3). Assessments from two or more extractors - whether for unclear cases or for quality assurance - can then be reconciled (9.4).
SECTION: Automated validation of publications for statements indicating Open Data using ODDPub
3.1. First, install ODDPub package in R with the following command:
 

SECTION: Automated validation of publications for statements indicating Open Data using ODDPub
3.2. Next, set the working directory to the folder containing all the publications to be examined. In this example, the folder is named 'PDFs':
 
Create a new folder to store the converted text files from PDFs after algorithm application.
SECTION: Automated validation of publications for statements indicating Open Data using ODDPub
3.3. After installing ODDPub and organizing the PDFs, run the following command to start evaluation for Openness:
 
After screening, an output file in .csv format will be created in the same folder as the ODDPub script. You can change the format to any other type if needed (e.g. .txt, .tsv). Below is an example of the output from the latest ODDPub version (7.0.0). This output will be further validated manually in Numbat (see from the step 4).

 

SECTION: Different issue handling in Numbat - Admin access
9.1. In case an extraction has to be corrected:

Go to the 'Do extractions' section.
Locate the dataset you need to correct.
Open the extraction form as usual by clicking 'Extract'.
Change the answer(s) as needed, then click on 'Completed' if required.
If the extractions are finished and an export has already occured, delete the older version of the table from your storage and export the updated output via 'Export data'.

 

SECTION: Set up extraction workflow in Numbat - Admin access
7.1. Select 'Manage reference sets' section from the Numbat menu, then click on 'Add new reference set'.

 
 

SECTION: Set up extraction workflow in Numbat - Admin access
7.2. Select the relevant columns from the dropdown menu (e.g., doi, is_open_data, open_data_statement, is_reuse) and assign a name to the set (e. g. 'Publications of 2020').

 
SECTION: Set up extraction workflow in Numbat - Admin access
7.3. Assign the new dataset to a user (or multiple users) via 'Manage extraction assignments' in the Main menu:
Select either all records or choose specific ones from the list.
In the 'For the following form' section, select the extraction form you want to use.
In the 'For the following user' section, choose the user(s) to whom you want to assign the tasks.
Click 'Assign to user' - a list will appear below, with successful assignments highlighted in green.

 
Explore different assignment options, such as assigning articles that have already been screened by one rater or randomly selecting some articles.

SECTION: Set up extraction workflow in Numbat - Admin access
6.3. Prepare article list (ODDPub results) for Numbat

Filter the output from ODDPub in the is_open_data column to include only TRUE statements. You can use a table calculation program like Excel or any text editor.

Save the filtered data in .tsv format (tab-delimited text file) as new input for Numbat. A text editor is more suitable than Excel for this task, as Excel spreadsheets are known to cause unexpected errors. For example, adding a small symbol or space can lead to incorrect file reading, even if the spreadsheet appears correct upon visual inspection. Additionally, copy and paste actions in Excel can result in lost or modified data. By using a text editor, you maintain control over every character.
SECTION: Set up extraction workflow in Numbat - Admin access
6. Preparing the ODDPub output for further evaluation in Numbat

To validate the presence of Open Data associated with the article, an extraction form was created in Numbat. This follows the Open Data Criteria for LOM (leistungsorientierte Mittelvergabe or performance-based allocation of funds). Each publication is reviews individually to determine whether the Open Data statement detected by ODDPub ("is_open_data" = TRUE) indeed refers to an openly accessible dataset.

 

SECTION: Manual validation using Numbat - User access
5.1. Select the 'Do extractions' option to begin extracting the Open Data status for publications where ODDPub has detected an Open Data statement ("is_open_data" = TRUE) by clicking on 'Extract'.

If there are multiple repositories, add a new sub-extraction for each repository after finishing the previous one.
 
The extraction of other categories, such as "is_reuse" =TRUE or "unknown url", is optional and
should be based on the specific goals of the screening.
SECTION: Manual validation using Numbat - User access
5.2. Open Data statement - Which ‘candidates’ are detected by ODDPub?

Review the detected statement(s) for details such as repository information or accession codes. Be sware of potential false positives, which may include open code or statements completely unrelated to data sharing.

The extraction form not only covers open data in the strict sense (freely accessible data) but also data under restricted access and the reuse of open data. It allows documentation of these practices without completing a full extraction.

Some reuse cases might lack a direct citation or referenceto the landing page in the article, even though they are indicated as a basis for the study provided by another institution or found in publicly accessible repositories. Where it is possible, answer all questions up to authorship. For cases where multiple criteria are challenging to assess, it is possible to document the reuse cases with the response "Not checked (reuse)".

To save time, you can skip extracting cases where it is clear that data have been reused or access is restricted. However, in many cases, this clarity emerges only during the detection process. Ignoring obvious cases may result in an incomplete overview. 
picture will not be complete.

Always click 'Complete' to finish the extraction.


 
 
SECTION: Manual validation using Numbat - User access
5.4.  Begin a new sub-extraction 

For each repository extract one dataset.

Add a new Sub-Extraction to the current extraction. 
If datasets are shared across multiple repositories, begin with the first repository listed in the article.
For each repository, if multiple dataset are shared, choose the first listed dataset for the extraction.
Repeat the extractions by adding a new sub-extraction for each additional repository. The total number of sub-extractions will correspond to the number of data repositories mentioned in the article, plus any additional sub-extractions for explicitly referenced supplemental data.
Typically, you will encounter one or two repositories and a few datasets each, but a larger number is possible.

 
How to add a new sub-extraction and delete any spurious ones:
 

SECTION: Different issue handling in Numbat - Admin access
9.2. How to upload new article to an existing reference set (table):

1. Create a new .tsv document in your text editor. This document should include the same column names as your existing reference set and contain all the references (DOIs) that must be added.
2. Go to 'Manage reference set' --> then 'Your set name XY' --> and 'new reference'. 
3. Verify that the columns in the updated table match the existing ones .
4. Assign the new records to one or more users via 'Manage user assignments'.
SECTION: Different issue handling in Numbat - Admin access
9.3. How to assign a questionable dataset to another extractor:

At the end of every extraction, there is an 'Assign to' option. Select the extractor(s) and the relevant extraction form (here: Openness), then finalize the assignment by clicking on 'Completed'. 

For our use case, we assigned datasets to other extractors whenever the Open Data status was marked 'unsure'. Conversely, we did not reassign datasets if the status was clear to the first extractor (i.e., 'yes' or 'no').
SECTION: Different issue handling in Numbat - Admin access
9.4. How to reconcile the answers of different extractors:

This function is essential when multiple extractors have worked on the same articles. If more than one extractor has completed the extraction, their answers can be compared and reconciled. Overlapping responses can be reconciled immediately, while differing responses may require further discussion or a decision based on the extractors' comments.

Go to the 'Reconcile finished extractions' section.
The extractors who completed the extractions will be listed under the 'Extractors' column.
Click on 'Reconcile extractions': 
compare the answers of two or more extractors, which will be displayed in adjacent columns.
a discussion among the extractors may be necessary to reach a clear conclusion, unless one extractor is designated as the 'master extractor' and undertakes the final decision. Pay attention to the commentaries, as they may include valuable insights for decision-making.
5. Save the selected answer in the final report by clicking 'Copy to final'. 
6. The final copy can still be edited and enriched with additional comments if none of the original answers is fully satisfactory.
7. Copy each sub-extraction individually into the final copy.
7. As in the article extraction, ensure to click 'Completed' after finishing the reconciliation.
SECTION: Manual validation using Numbat - User access
5.3. Search for other indications of Open Data missed by ODDPub

Although ODDPub is highly sensitive, it may still miss some Open Data statements, especially in fields less related to the biomedical field for which the workflow was developed and validated. To balance sensitivity and effort, it is recommended to review the article, focusing on the areas around the statements detected by ODDPub to check for any additional information that might have been missed. You can use keyword searches within the article to quickly locate the sections related to the ODDPub statements. If a statement appears to be a combination of different sections or sentences, use various keywords from the statement to access all detected sections.

As new repositories and standard statements emerge, the frequency of missed Open Data statements may increase over time. Therefore, it is recommended (though not strictly necessary, depending on your use case) to briefly scan the article itself for further indications of shared datasets. Follow these steps:

Search for keywords such as "data availability", "data sharing", and "data access". If you find a section named "data availability statement" or similar, review the entire section for indications on shared data.

  2. If no such section is found, use keywords like "dataset", "data set", "access*" and "availab*". Check all results with <=10 hits for each keyword. Dismiss results where there are >10 hits.

  3. If no statement is found, search for the keyword "data"; if it yields <=10 hits, review each result. If it yields >10 hits, dismiss them.

This procedure helps identify most datasets missed by ODDPub with minimal additional time investment. However, for a larger set of articles, this step may still be substantial, so consider implementing it only if missing datasets would significantly impact your use case.
SECTION: Set up extraction workflow in Numbat - Admin access
6.2. To set up the Openness extraction form in your Numbat workspace, select 'Edit extraction form' and click on the 'Import an extraction form' button. Ensure that the extraction form is in JSON format.
 
You may need to make additional adjustments, such as modifying conditions for when and how each question should appear.
SECTION: Overview of extracting form in Numbat
4. The attached file contains the Numbat extraction form (Version 4). This form was automatically generated from Numbat in markdown format and then transformed into a PDF using https://md2pdf.netlify.app/ for better human readability. It contains all questions related to Open Data criteria for manual openness verification.
______________________________________________________________________________________________________________
  
______________________________________________________________________________________________________________
SECTION: Post-process extracted table in Numbat - Admin access
8. How to export the final report table:

You can only export the data if more than two records have been checked.

Before downloading the results table, ensure that the answers have been reconciled if multiple users have checked the same data records. Otherwise, you may encounter duplicates.

Go to the 'Export data' section -> 'Export Openness extractions' to download the results table.
Clean up the table:
For Excel: go to the 'Data' tab, then 'Text in Columns' -> celect 'Delimited' + tab stop / comma (depending on your data) + Standard -> Finish. Save the file in your desired format.

Consider that the extraction dataset may contain several Open Data assessments per article. If you need an analysis at the article level, process the output accordingly. For example, when incentivizing data sharing at the article level, identify all publications in which at least one dataset was shared (openly or with restricted access).
 

  
        article   is_open_data   open_data_category   is_open_code   open_data_statements   open_code_statements   reference_to_data   comment_1_reference_to_data   identifier   own_or_reuse_data   comment_2_own_data   data_in_supplement   comment_3_data_in_supplement   findability   comment_4_findability   data_access   comment_5_data_access   is_machine_readable_format   comment_6_format   machine_readable_format_excel   machine_readable_format_csv   machine_readable_format_txt   machine_readable_format_spss   machine_readable_format_other_text_formats   machine_readable_format_video   machine_readable_format_audio   machine_readable_format_picture   machine_readable_format_fasta_fastq   machine_readable_format_raw   machine_readable_format_genetic_sequences   machine_readable_format_subject_specific_format   machine_readable_format_unsure   comment_7_machine_readable_format   illustrative_files   comment_8_illustrative_files   analytical_replication   comment_9_analytical_replication   assessment   comment_10_open_data_discussion      10.1038/s41467-020-16734-3   TRUE   general-purpose repository   TRUE   code availability all code used to analyze the dataset is   openly available within lead-dbs/-connectome software   (https://github.com/leaddbs/leaddbs).   code availability all code used to analyze the dataset is   openly available within lead-dbs/-connectome software   (https://github.com/leaddbs/leaddbs).   yes   NULL   supplement   own_open_data   NULL   no   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   no_open_data   only a supplement   available      10.1038/s41467-020-16929-8   TRUE   field-specific repository   FALSE   proteomics data have been deposited to pride server under   accession code pxd017341   [http://proteomecentral.proteomexchange.org/cgi/getdataset?   NA   yes   NULL   http://proteomecentral.proteomexchange.org/cgi/GetDataset?ID=PXD017341   own_open_data   NULL   yes   NULL   yes   NULL   yes   NULL   yes   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   1   NULL   NULL   NULL   NULL   yes   NULL   yes   NULL   open_data   NULL            10.1186/s12916-020-01851-z   TRUE   supplement   FALSE   additional file 5. kaplan-meier raw data   NA   yes   NULL   TriNetX   unsure   NULL   unsure   NULL   unsure   NULL   restricted   not publicly available, the data are available from the   authors upon reasonable request and with the permission of TriNetX.   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL           


SECTION: Numbat local installation - (System) Admin access
10. What is Numbat and how to install it
 

How to install Numbat (for Windows OS):

Requirements:
Apache HTTP Server
MySQL Database - provided by your company/organization
PHP

 


SECTION: Numbat local installation - (System) Admin access
10.2. Install the XAMPP package
It includes the Apache distribution and MySQL database.
 
This is what XAMPP looks like before any distributions are started:

 
 

SECTION: Numbat local installation - (System) Admin access
10.1. Clone the Numbat repository from GitHub locally
You can use a GitHub account for this, or simply download the Numbat folder without logging in, as it is free open-source software.
 
SECTION: Numbat local installation - (System) Admin access
10.3. Install Numbat in XAMPP
Copy the entire repository into the 'htdocs' folder within XAMPP.
In XAMPP, start the Apache module by clicking on 'Start'.
Click on 'Admin' button next to Apache - you should see the installation instruction if they are not yet completed.

The following image shows a successful installation:
 

SECTION: Numbat local installation - (System) Admin access
10.4. MySQL setup (probably provided by the company)
In XAMPP: start the MySQL module by clicking on 'Start'.
Click the 'Admin' button next to MySQL to access the database.
Create a new MySQL database for Numbat

 
Create a new user (yourself as a new admin). You can use the default preferences with your name and password, as shown in Figure 10.4.3

 
 

SECTION: Numbat local installation - (System) Admin access
10.5. 
In XAMPP: use the 'Admin' button next to Apache to complete the installation. 
In the window that opens, fill out the database username, password, name, and host, as well as the URL name - everything except the URL you created in step 10.4.
SECTION: Set up extraction workflow in Numbat - Admin access
6.1. New user registration:

To register as a new user in the existing Numbat instance, click on 'New here? Sign up'. You only need to provide your email address, password, and name. After registration, the Numbat admin will activate your account.

User administration:

As an admin of the Numbat instance, navigate to the 'User administration' section in the main menu. New users will appear with unverified email addresses. Verify their email addresses and assign the appropriate privileges - User or Admin. The descriptions of these privileges are provided on the same page.
SECTION: Overview of extracting form in Numbat
4.1. The table below provides an overview of all Open Data criteria presented in the extraction form.

 

A quick video tutorial on how to do the extractions can be found under Section 5.1.
SECTION: Overview of extracting form in Numbat
4.9. Embargoed data:
data that are not yet public but whose publication is planned and precisely specified on the existing dataset landing page.
they are documented through the "Data under embargo" button and can be validated after the embargo period ends.
SECTION: Overview of extracting form in Numbat
4.6. Handling the documentation of the data set publication date:
typically, use the last update date.
however, this can be confusing in some cases, as the last metadata update date may not correspond to the publication date. For example, in the Gene Expression Omnibus (GEO), the "Status (public)", "Submission date" and "Last update date" are usually three different dates. The entry most closely related to the publication date is the "Status (public)".
SECTION: Overview of extracting form in Numbat
4.8. Genetic sequences and case reports:
considered (open) data only when shared in a discipline-specific repository.
SECTION: Overview of extracting form in Numbat
4.7. Managing Access with Registration Requirements:
registration in a repository or databank should not be mandatory to access the data. If access is hindered in such way, it cannot be considered open access.
simply clicking on an agreement without requiring registration is sufficient to meet the definition of open and free access.
SECTION: Overview of extracting form in Numbat
4.5. What does count to the 'best identifier'?
the best option for an identifier is a persistent one, such as a DOI or Handle.
if both a URL and a DOI are available, the DOI should be documented as it is a more stable and persistent identifier. 
for discipline-specific repositories that do not provide persistent identifiers, documenting the URL along with the accession number is the best approach.
in cases like NCBI BioProject/SRA, link the project main page along with the corresponding SRA page leading to the data files. 
non-persistent URLs, such as those from GitHub or the Open Science Framework without a DOI, are not sufficient as identifiers or data sources because not just metadata, but also data files can be changed by the data owners.
supplements are generally considered insufficient unless they are stored independently of the article.
SECTION: Overview of extracting form in Numbat
4.2. Which types of articles might produce data and therefore should not be excluded from the screening as potential sources of data?
case report
study protocol
methodology
short reports
systematic review
SECTION: Overview of extracting form in Numbat
4.12. Own or Reuse data:
document both "Own (Open) Data" and "Reused (Open) Data" cases.
typically, dataset metadata include author names, which can be compared with the article's authors to determine data ownership. In some cases, such as some of the NCBI repositories, the data owner may be listed only under an institution name. In such cases, if any author is affiliated with that institution, the dataset is considered "own" data.
if neither author nor institution names are available, the article text (e.g., "data were collected during...") and context should help to clarify the data origin.
reuse statements are often found outside the "Data Availability Statement".
reuse of the datasets generated by other researchers is not considered Open Data per default, but need a separate verification for openness.


SECTION: Overview of extracting form in Numbat
4.11. Criteria for restricted or reuse cases:
criteria for repository requirements and accessibility may vary for restricted data sets. A URL to the location or page where the data can be requested (rather than a general "upon request" statement in the article) is sufficient.
the same approach applies to the reuse of restricted data sets.
SECTION: Overview of extracting form in Numbat
4.14. Are classifieds (models) for ML training raw data?
classifiers for machine learning algorithms are not considered raw data according to our definition.
SECTION: Overview of extracting form in Numbat
4.3. Systematic reviews:
should include more than just a list of referenced publications; they should also provide extracted information and clearly outline the exclusion and inclusion criteria.
SECTION: Overview of extracting form in Numbat
4.13. Are genomics summary statistics the same as other summary statistics?
generally, summary statistics are not considered raw or unprocessed data. However, genomics summary statistics stored in specialized repositories, such as the GWAS Catalog, differ from typical summary statistics. They represent a collection of studies and are not directly comparable to standard statistics.
SECTION: Overview of extracting form in Numbat
4.10. Pharma data:
researchers may collaborate with or be part of a pharmaceutical company, but the primary consideration is whether the data is stored on an academic platform/database/repository or at a pharmaceutical company (e.g., Vivli, Yoda). 
mention of pharmaceutical platforms in the article indicates that the data are more likely to be stored outside academic repositories, and such cases are treated separately from common (academic) open data cases.
SECTION: Overview of extracting form in Numbat
4.4. What does not count as a 'clear reference'?
according to the current criteria (Bobrov, et al. 2024, 10.1162/qss_a_00301) supplements are not considered Open Data, partly because they are not shared independantly from the article.
reference to data shared within the manuscript, without explicitely mentioning the word 'supplement', are also not sufficient.
references to data presented in tables within articles do not qualify as clear references to raw data.
reference to data upon reuqest by the authors is do not qualify as clear references to raw data.