DOI: 10.7910/dvn/8feguv
title: Dataset metadata of known Dataverse installations, August 2023

description: &lt;p&gt;This dataset contains the metadata of the datasets published in 85 Dataverse installations and information about each installation's metadata blocks. It also includes the lists of pre-defined licenses or terms of use that dataset depositors can apply to the datasets they publish in the 58 installations that were running versions of the Dataverse software that include that feature.

&lt;p&gt;The data is useful for reporting on the quality of dataset and file-level metadata within and across Dataverse installations and improving understandings about how certain Dataverse features and metadata fields are used. Curators and other researchers can use this dataset to explore how well Dataverse software and the repositories using the software help depositors describe data.

&lt;p&gt;&lt;strong&gt;How the metadata was downloaded&lt;/strong&gt;
&lt;p&gt;The dataset metadata and metadata block JSON files were downloaded from each installation between August 22 and August 28, 2023 using a "get_dataverse_installations_metadata" function in a collection of Python functions at &lt;a href="https://github.com/jggautier/dataverse-scripts/blob/main/dataverse_repository_curation_assistant/dataverse_repository_curation_assistant_functions.py"&gt;https://github.com/jggautier/dataverse-scripts/blob/main/dataverse_repository_curation_assistant/dataverse_repository_curation_assistant_functions.py&lt;/a&gt;. 

&lt;p&gt;In order to get the metadata from installations that require an installation account API token to use certain Dataverse software APIs, I created a CSV file with two columns: one column named "hostname" listing each installation URL in which I was able to create an account and another column named "apikey" listing my accounts' API tokens. The "get_dataverse_installations_metadata" function expects this CSV file in order to get metadata and other information from installations that require API tokens.
&lt;p&gt;&lt;strong&gt;How the files are organized&lt;/strong&gt;

&lt;pre&gt;
├── csv_files_with_metadata_from_most_known_dataverse_installations
│   ├── author(citation)_2023.08.22-2023.08.28.csv
│   ├── contributor(citation)_2023.08.22-2023.08.28.csv
│   ├── data_source(citation)_2023.08.22-2023.08.28.csv
│   ├── ...
│   └── topic_classification(citation)_2023.08.22-2023.08.28.csv
├── dataverse_json_metadata_from_each_known_dataverse_installation
│   ├── Abacus_2023.08.27_12.59.59.zip
│       ├── dataset_pids_Abacus_2023.08.27_12.59.59.csv
│       ├── Dataverse_JSON_metadata_2023.08.27_12.59.59
│          ├── hdl_11272.1_AB2_0AQZNT_v1.0(latest_version).json
│          ├── ...
│       ├── metadatablocks_v5.6
│          ├── astrophysics_v5.6.json
│          ├── biomedical_v5.6.json
│          ├── citation_v5.6.json
│          ├── ...
│          ├── socialscience_v5.6.json
│   ├── ACSS_Dataverse_2023.08.26_22.14.04.zip
│   ├── ADA_Dataverse_2023.08.27_13.16.20.zip
│   ├── Arca_Dados_2023.08.27_13.34.09.zip
│   ├── ...
│   └── World_Agroforestry_-_Research_Data_Repository_2023.08.27_19.24.15.zip
└── dataverse_installations_summary_2023.08.28.csv
└── dataset_pids_from_most_known_dataverse_installations_2023.08.csv
└── license_options_for_each_dataverse_installation_2023.09.05.csv
└── metadatablocks_from_most_known_dataverse_installations_2023.09.05.csv

&lt;/pre&gt;

&lt;p&gt;This dataset contains two directories and four CSV files not in a directory.
&lt;p&gt;One directory, "csv_files_with_metadata_from_most_known_dataverse_installations", contains 20 CSV files that list the values of many of the metadata fields in the citation metadata block and geospatial metadata block of datasets in the 85 Dataverse installations. For example, author(citation)_2023.08.22-2023.08.28.csv contains the "Author" metadata for the latest versions of all published, non-deaccessioned datasets in the 85 installations, where there's a row for author names, affiliations, identifier types and identifiers.
&lt;p&gt;The other directory, "dataverse_json_metadata_from_each_known_dataverse_installation", contains 85 zipped files, one for each of the 85 Dataverse installations whose dataset metadata I was able to download. Each zip file contains a CSV file and two sub-directories:
&lt;ul&gt;
&lt;li&gt;The CSV file contains the persistent IDs and URLs of each published dataset in the Dataverse installation as well as a column to indicate if the Python script was able to download the Dataverse JSON metadata for each dataset. It also includes the alias/identifier and category of the Dataverse collection that the dataset is in.
&lt;li&gt;One sub-directory contains a JSON file for each of the installation's published, non-deaccessioned dataset versions. The JSON files contain the metadata in the "Dataverse JSON" metadata schema. The Dataverse JSON export of the latest version of each dataset includes "(latest_version)" in the file name. This should help those who are interested in the metadata of only the latest version of each dataset.
&lt;li&gt;The other sub-directory contains information about the metadata models (the "metadata blocks" in JSON files) that the installation was using when the dataset metadata was downloaded. I included them so that they can be used when extracting metadata from the dataset's Dataverse JSON exports.
&lt;/ul&gt;
&lt;p&gt;The dataverse_installations_summary_2023.08.28.csv file contains information about each installation, including its name, URL, Dataverse software version, and counts of dataset metadata included and not included in this dataset 
&lt;p&gt;The dataset_pids_from_most_known_dataverse_installations_2023.08.csv file contains the dataset PIDs of all published datasets in the 85 Dataverse installations, with a column to indicate if the Python script was able to download the dataset's metadata. It's a union of all "dataset_pids_....csv" files in each of the 85 zip files.
&lt;p&gt;The license_options_for_each_dataverse_installation_2023.09.05.csv file contains information about the licenses that some installations let depositors choose when creating datasets. When I collected this data, 58 of the available 85 installations were running versions of the Dataverse software that allow depositors to choose a license or terms of use from a dropdown menu in the dataset deposit form. For more information, see &lt;a href="https://guides.dataverse.org/en/5.14/user/dataset-management.html#choosing-a-license"&gt;https://guides.dataverse.org/en/5.14/user/dataset-management.html#choosing-a-license&lt;/a&gt;.
&lt;p&gt;The metadatablocks_from_most_known_dataverse_installations_2023.09.05.csv file contains the metadata block names, field names and child field names (if the field is a compound field) of the 85 Dataverse installations' metadata blocks. This file is useful for reviewing and comparing the metadata fields and field structure used in each installation.

&lt;p&gt;&lt;strong&gt;Known errors&lt;/strong&gt;
&lt;p&gt;The metadata of a few datasets from several known installations could not be downloaded. In most cases, this is because of download timeouts caused by the datasets' relatively large metadata exports, which contain information about the datasets' large number of versions and files.

&lt;p&gt;&lt;strong&gt;About metadata blocks&lt;/strong&gt;
&lt;p&gt;Read about the Dataverse software's metadata blocks system at &lt;a href="http://guides.dataverse.org/en/latest/admin/metadatacustomization.html"&gt;http://guides.dataverse.org/en/5.14/admin/metadatacustomization.html&lt;/a&gt;
description: Python, 3.7
description: Dataverse
description: When I collected ODISSEI Portal's dataset metadata in late August 2023, all datasets listed in ODISSEI were actually published in other repositories, including DataverseNL and IISH Dataverse. The ODISSEI Portal's purpose is to help others find data published in other repositories. 

Because the dataset metadata wasn't harvested using OAI-PMH, the installation's Search API doesn't indicate that the datasets are harvested, so the Search API's parameters couldn't be used to exclude "harvested" datasets. The dataset PIDs attributed to ODISSEI Portal are also listed in the files and directories of DataverseNL and IISH Dataverse.