PolarFront data management status 2023-03
Creators: name: Helgeland, Conrad
Publisher: name: Zenodo
Year: 2023
Subjects: PolarFront

description: https://doi.org/10.5281/zenodo.7753095 Presentation text with speaker notes <em>Slide 1:</em><br> PolarFront <strong>data management status </strong>2023-03<br> Conrad Helgeland Notes:<br> Conrad Helgeland (CH/che@akvaplan.niva.no) is co-lead of the data management work package in the PolarFront project.The other work package leader is Geir Morten Skeie (GMS/gms@akvaplan.niva.no). <em>Slide 2:</em><br> PolarFront is <strong>open science</strong> Research outputs are <strong>open access </strong> Citable, open data in a high-trust repository Open source data management in version control Open/transparent data publishing strategy Notes: PolarFront is "built on <strong>open science</strong> principles" (proposal, 2021)<br> PolarFront data, research articles, and other output is <strong>open access</strong><br> We have open access guidelines<br> We have a data management plan<br> We have a safe data repository for all research outputs<br> We have a data management team (and work package)<br> We have a dedicated GitHub repository for data management<br> We encourage public sharing of all kinds of research output – from all project phases<br> "No raw data, no science": We require publishing of initital raw data in addition to analysed and/or polished data products, for increased trust and traceability Ref. Miyakawa, T. No raw data, no science: another possible source of the reproducibility crisis. <em>Mol Brain</em> <strong>13</strong>, 24 (2020). https://doi.org/10.1186/s13041-020-0552-2 <em>Slide 3:</em><br> <strong>Work done in 2022 / 2023</strong> Data repository review process Zenodo selected for all research outputs Ship &amp; glider data published Open access guidelines Cruise report 2022-05 Darwin Core trawl biodiversity Darwin Core sampling events PolarFront open access tracker Notes:<br> Here are some highlights from 2022/early 2023<br> Zenodo selected as data repository after comprehensive review process<br> Ship &amp; glider data from 2022 published<br> First 2021 datasets published (glider data)<br> Open access guidelines finalized<br> Cruise report 2022-05 published<br> Created Darwin Core trawl biodiversity datasets (ready for GBIF)<br> Created standardised Darwin Core sampling events (from raw ship log, the arrows are nonsense)<br> Created open access tracker <em>Slide 4:</em><br> Zenodo logo Notes:<br> We made a big move just before the end of 2022. We switched data repository to Zenodo, on our own community (ie. dedicated landing page) for all PolarFront datasets and other material. The number one reason for selecting Zenodo was the incredibly robust file storing backend.The file storage supports resuumable uploads and downloads making it quite easy to deposit and access our big datasets. In Zenodo every dataset have at least 2 DOIs, one eternal (always pointing to the latest version). And one DOI pointing to a specific set of files. Files are immutable, ie. you may never remove or change a file, only upload a new set and thereby creating a new DOI. Introduction to Zenodo:<br> LH Nielsen (2017). Sharing your data and software on Zenodo. https://doi.org/10.5281/zenodo.787445 Upcoming release:<br> https://blog.zenodo.org/2022/12/07/2022-12-07-zenodo-on-inveniordm/ <em>Slide 5:</em><br> Screenshot of Zenodo search interface for PolarFront: https://zenodo.org/communities/polarfront/search?q= Notes:<br> In Zenodo, we have our own community repository: https://zenodo.org/communities/polarfront<br> The PolarFront repository is searchable and available to harvest as a machine-readable dataset catalog (DCAT).<br> You may filter by type (dataset, presentation, publication) and other aspects like file type and keywords. <em>Slide 6:</em> <br> Self-publishing data <strong>Everyone on the project </strong>may publish datasets &amp; derived data products on Zenodo You just need an ORCID or a GitHub handle Register on Zenodo – and hit the green button <strong>Remember to add the dataset to the PolarFront community</strong> This is the best option for the long tail that may come in the late stage of the project Notes: Who here has (not) an ORCID or a GitHub handle?<br> With Zenodo, it is now possible for anyone on the project to publish data or other material.<br> This means that there is one less single lane / single point of failure, dataset publishing now does not require the data manager, except to <strong>approve </strong>the inclusion of the published resource in our Zenodo archive. <em>Slide 7:</em><br> Big data publishing<br> Screenshot of dataset landing page: https://doi.org/10.5281/zenodo.7463345 Automated pipeline Data integrity checks Resumable sync/upload Reads source data from disk or cloud Publishes data as multiple 50GB datasets A parent DOI with common files and documentation is created (see left) Notes: Autonomous vehicle data and data copied cruise network disks will be published by Conrad Helgeland. In PolarFront we have two main sources of big data, acticve and passive acoustics, that is echosounders and underwater microphones. Managing large datasets can be challenging. Zenodo has a 50 GB max size per dataset, so for long endurance / high volume data we need to split data into manaageable chunks. Notice that files are NOT zipped together. This is an advantage while uploading and downloading, both of which are resumable on Zenodo. Also validating data integrity is much easier with individual files contra having the files inside a zip archive. To download everything, you simply need a tool that will accept a list of file URLs as input. <em>Slide 8:</em><br> Screenshot of May 2022 cruise report landing page: Notes: The May 2022 cruise report was our first significant project output. Thanks to Malin and all contributors.<br> Cruise Reports are vital in providing early signs of what data is gathered and which equipment is used.<br> For many ship-based datasets, the description is currently copied verbatim from the report.<br> For future reports we would like to include DOIs to the data straight in the report. <em>Slide 9:</em><br> Open access guidelines Notes: PolarFront now has a set of open access guidelines, based on best practices. <br> In order to publish data, and for others to be able to cite a PolarFront work, you<strong> </strong>must at least provide: Title Description Rights. By default PolarFront data files are declared free of copyright restrictions, by applying the CC0 public domain dedication. <br> Alternative license for datasets, in particular hand-crafted: CC-BY 4.0 Author(s). Unless provided, <strong>PolarFront Consortium </strong>will be set as dataset author. Remember: Cite other/relevant PolarFront works by DOI. Include Research Council of Norway funding grant #326635 in addition to the name PolarFront <em>Slide 10:</em><br> <strong>Darwin Core</strong> <br> Occurrences (biodiversity) example<br> Sampling events example Notes:<br> Frida and Conrad have cooperated to create two Darwin Core biodiversity datasets, from fish and macrozooplankton trawls.These are now ready to be deposited in GBIF, the Global Biodiversity Information Facility. There is a lot to be learned from Darwin Core, for all of you that are compiling data "by hand": Use plain text files with UTF-8 encoding Use programmer friendly variable names Use standard format for date and time (and always use UTC) Use standard WGS84 GPS coordinates Consider using a data model like in Darwin Core, based on DRY principles (don't repeat yourself):<br> Create one file for sampling events, and keep actual observations/measurements in separate files. We have all seen what happens if you smudge all possible properties into the same sheet, your data becomes murky and prone to having differerences that should not be there, due to spelling and other mistakes. <em>Slide 11:</em><br> Open access tracker<br> Screenshot of https://polarfront.deno.dev/ Notes:<br> The PolarFront <strong>open access research</strong> tracker, enables us to monitor our research outputs – per field season.<br> Notice draft datasets are visible here (not available anywhere else on the public internet)<br> We have a lot of datasets. Make sure to cite the appropriate datasets when writing articles.<br> And help make sure that all relevant datasets for your work package is listed here (more on this soon). <em>Slide 12:</em><br> Call for data <strong>Is the dataset you are sitting on/preparing in the open access tracker?</strong> All datasets that are not in the open access tracker must be registered. Project leadership has requested publication of the 2021-05 data, but only CTD and ship log is available, formal request sent Please inform the data management team on upcoming datasets (just a preliminary title by email will do) Notes: Much data is simply unknown/unavailable to the data management team<br> We have only 2 sources of data as of now A disk from 2022-05 cruise All of the glider data in cloud storage (full resolution/delayed mode) for all deployments (2021,2022) Project leadership has requested publication of the 2021-05 data, but only CTD and ship log is available For ship/glider data we have a folder with data files, but absolutely no descriptors<br> Only 2 researchers have contributed datasets (that are not on the disk after cruise) Only CTD and ship log is currently available from 2021-05, a formal request for data is sent to the cruise leaderRemember link(s)to input data (if already published on a different DOI) <em>Slide 13:</em><br> <strong>Deliverables</strong> D4.1 DMP D4.2 DCAT D4.3 Open source data translation toolbox D4.4 Open source data visualization D4.5 Data paper Notes:<br> Deliverables – where are we?<br> This is the list of deliverables from the proposal D4.1 Machine processable Data Management Plan (DMP) using ELIXIR Norway’s Data Stewardship Wizard D4.2 Data set catalog in DCAT vocabulary linking to all primary data sets and derived data products. D4.3 Open source data translation toolbox for processing data into formats needed for archiving, analysis, models, etc. D4.4 Open source data visualization and communication components, including real-time data preview during cruise operations. D4.5 Data paper on the data management and key datasets And here's a rough status DMP is mostly done<br> DCAT is included in Zenodo <br> Toolbox: Big data pipeline in place [GMS on adapting data for risk models?]<br> Real time data viz: Prototype exists<br> Data paper: ? <em>Slide 14:</em><br> In 2023, work continues on Finish/publish big data pipeline software Publish all of 2021-05 Publish 2021-05 cruise report Finish up 2022 data Publish upcoming data Finalizing and publishing data management plan v1 Risk analysis data (GMS) Notes: Goal: Tighter collaboration with researchers in each work package<br> Ensure key data products are published in appropriate formats<br> Improve data descriptors – lots to be done<br> Work on data standardization <em>Slide 15:</em><br> We are not alone<br> 2023 is the year of open science :)<br> Screenshot: <strong>Open Science is Accessible, Reproducible &amp; Inclusive</strong> from Chelle (2022). Reference<br> Gentemann, Chelle. (2022-12-05). NASA's 2023 Year of Open Science. Zenodo. https://doi.org/10.5281/zenodo.7401880
description: Presentation given at PolarFront project meeting on 2023-03-20.