Registry Considerations for Manifest Files:

Configurable Data Curation System (CDCS) and FAIR Containers

Benjamin Long

CDCS Team: Guillaume Sousa Amaral Philippe Dessauw, Hamza Bouhanni
Introduction

2 2023-12-06

 Will discuss 

 an informatics platform: Configurable Data Curation System (CDCS) 

 that may be relevant to your context: interoperable, FAIR containers

 brief contextual intro

 Will show 

 some examples 

 that demonstrate its ability to support this concept in common formats (XML, JSON), with relevant 

API automation

 Will consider 

 how this might fit into use-cases such as 

 UI support, 

 automation, 

 file-format usage and translation

 even touching a little on consensus specification and evolution
Configurable Data Curation System (CDCS)

3 2023-12-06

 Informatics infrastructure / eco-system

 Developed at NIST

 For Materials Genome Initiative (MGI) – original context: materials science

 For input, output, transformation, search for, and storage of 

 information structures (data, their abstractions, associated files/resources)

 Supports FAIR (findable, accessible, interoperable, reusable) data

 Infrastructure for storage (repositories) and search/discovery (registries)

 Will discuss

 Your context: FAIR containers/manifests

 How this fits, potentially, into that context + those goals
Organizing and connecting resources

(Question: How to organize, connect, and coordinate an eco-system of manifests and specs over time?)

4 2023-12-06

Disconnected data

Data resource

CDCS repository

CDCS registry

Registry federation

Repository federation
Strategy – customize individually, connect globally

(Question: How to support rapidly growing, diverse groups, communities and their outputs in a unified way where all can benefit?)

5 2023-12-06
Strategy – organize communities of meaning

(Question: Will you need to do analysis and informatics on your manifests and containers themselves?)

6 2023-12-06

Shared meaning

Manifests

Persistent IDs

Linked data
The CDCS Process

7 2023-12-06

Meet with users and communities about their resources and data

Start structuring and sharing data one resource, one project, or one community at a time

• Customize online containers for structured data

• Create repositories of data (in XML and JSON format)

• Convert data into different formats as needed

• Customize how it is presented, displayed, and searched

• Share and use it in a FAIR manner

Start linking together data into a more global infrastructure

• Connecting repositories together

• Creating and connecting registries together

• Creating and connecting data via linking (persistent IDs, handle services, REST APIs)
Configurable Data Curation System (CDCS)

8 2023-12-06

Curators (Repositories) Registry (Search)
Federated Registries

(Question: How do you anticipate distributing the load of tracking, coordinating evolution in manifests and specs across a potentially global ecosystem like this one?)

9 2023-12-06

4 5 6 7

1 2 3

Registry Rules of Connection

• Interoperability Protocol

• Common Schema

Element 

(Registry)

Combination 

(Federated 

Registry)
Reason About Knowledge of Entire Domain

(Question: What knowledge regarding your manifests, containers, specifications, and eco-system would you like to reason about?)

10 2023-12-06

Representation Inference

Inference 

Engine

4 5 6 7

1 2 3

4 5 6 7

1 2 3

OVERALL GOAL:

USE FEDERATION TO ACCESS AND REASON 

ABOUT ALL GLOBAL, CURATED, KNOWLEDGE 

OF A DOMAIN
CDCS: General Objectives

(Question: Are there similar objectives in this context?)

11 2023-12-06
How CDCS could possibly support 

FAIR manifests and containers now

12 2023-12-06

 Define/manage component eco-system (such as through GitHub, DockerHub, 

manifest-files)

 Define manifest files in some supported format (XML, JSON)

 Register manifest files in CDCS system (registry) with PID

 Translate out in whatever formats (exporters and exporter extensions for translation)

 Automate processes for finding, manipulating, editing components and 

component-structures via CDCS registry REST API

 Have tools, services, processes, UIs, workflows – call this API to interact with 

the registry (or services that interact with the registry)

 Organize registry with several types, PIDs, and interlinkages

 Manifests

 Specification versions, evolution
Container manifest 

workflow representation example

13 2023-12-06
Manifest curation examples with CDCS

14 2023-12-06
Crop image example curation

15 2023-12-06
Threshold image example curation

16 2023-12-06
Search component manifests in CDCS

17 2023-12-06
Export/translate in other formats 

(JSON, XML, others)

18 2023-12-06
Access and manipulate via CDCS REST API 

(use to automate services, tools, UIs...)

19 2023-12-06
Each of these are linked with PIDs

20 2023-12-06

manifest manifest

spec spec

test

UI UI

test

Some useful considerations for specification 

evolution

• Can uniquely identify and store manifests, UIs, specs, and 

tests in distributed manner

• Any manifest can support 1+ specs (allow evolutionary 

compatibility of manifests based on supported specs)

• Specs can have lifecycle phases: draft, stable, legacy, 

retired

• Later specs can be backward compatible with 

previous ones

• Components/manifests can claim compliance with 1+ 

specs users might use

• Every spec has associated tests to verify compliance

• This enables users verify specification compatibility at 

a component/manifest level: important for 

interoperability

• A given manifest can refer to various supporting UIs

specification

• version: 1.0, 2.0, ...

• pid: 

https://host/pid/rest/local/cdcs/spec-1.0-stable

• status: draft, stable, legacy, retired

Test for specification

• version: 1.0, 2.0, ...

• pid: 

https://host/pid/rest/local/cdcs/test-1.0-stable

• status: draft, stable, legacy, retired
Conclusions

21 2023-12-06

 Needs

 Interoperable containers/manifests have a number of needs regarding 

 curation, formatting, manipulation, unique identification, association with UIs, specifications, tests, and needs to be discoverable and accessible through 

UIs and automated processes

 Challenges

 These needs are not typically addressed in a unified way through existing processes, components, or platform infrastructure (such as 

GitHub/DockerHub) by themselves

 Without support from platforms/methods like CDCS, there can often be significant varieties/duplication of format, information, and more

 Solutions

 CDCS core functionality, including support and applications of PIDs, integrated into lifecycle usage can help facilitate eco-system level reuse, 

interoperability, automation, and consensus specification coordination/evolution

 Lifecycle use of FAIR containers can be facilitated by CDCS platform infrastructure 

(CDCS registries and repositories)

 These follow best practices in software and data

 Example use-cases covered

 Curation - definition and evolution of components

 Export/translation – JSON, XML, other

 Persistent identification

 API-based automation and integration with services/UIs (REST)

 Reuse and interoperability 

 Of manifests, specifications, tests

 Within a shared eco-system
CDCS Project Information

22 2023-12-06