Package: ckanext-dcat-usmetadata
ckanext-dcat_usmetadataThis extension provides a new dataset form forinventory.data.gov. The form is tailored to managing metadata meeting theDCAT-US Schema.UsageDependenciesThis module currently depends on theUSMetadata appfor server-side validation and rendering.
Make sure it is enabled in CKAN's plugins.This extension is compatible with these versions of CKAN.CKAN versionCompatibility>=2.9yes>=3.0noInstallationAddckanext-dcat-usmetadatato your requirements.txt, and then pip installIn your CKAN .ini file adddcat_usmetadatato your enabled plugins:ckan.plugins = [YOUR PLUGINS HERE...] dcat_usmetadataCommandspublishers-importThis extension adds a new CLI command for importing publishers linked to CKAN
organizations. Thelist of
publishersshould be exported in a CSV data and it should have the following structure
(note the headers):organization,publisher,publisher_1,publisher_2,publisher_3,publisher_4,publisher_5
agricultural-marketing-service-department-of-agriculture,Department of Agriculture,Agricultural Marketing Service,,,,
ars-usda-gov,Department of Agriculture,Agricultural Research Service,,,,
aphis-usda-gov,Department of Agriculture,Animal and Plant Health Inspection Service,,,,
risk-management-agency-department-of-agriculture,Department of Agriculture,Departmental Management,,,,
usda-gov,Department of Agriculture,Office of Chief Information Officer,,,,
usda-gov,Department of Agriculture,Economic Research Service,,,,
usda-gov,Department of Agriculture,Farm Service Agency,,,,
usda-gov,Department of Agriculture,Food and Nutrition Service,,,,
usda-gov,Department of Agriculture,Food Safety and Inspection Service,,,,
usda-gov,Department of Agriculture,Foreign Agricultural Service,,,,
usda-gov,Department of Agriculture,National Agricultural Statistics Service,,,,
usda-gov,Department of Agriculture,National Institute of Food and Agriculture,,,,
usda-gov,Department of Agriculture,Natural Resources Conservation Service,Colorado State University,,,
usda-gov,Department of Agriculture,Rural Development,,,,
usda-gov,Department of Agriculture,GIPSA,Federal Grain Inspection Service,,,
usda-gov,Department of Agriculture,Natural Resources Conservation Service,,,,
usda-gov,Department of Agriculture,US Forest Service,,,,Each CKAN organization must have its own list of publishers.Example of running the command:$ ckan dcat-usmetadata import-publishers /path/to/publishers.csvDevelopmentPrerequisitesThese tools are required for development.Node.js12.xYarn1.22.xCypress6.0.0+SetupInstall Node.js dependencies.yarninstallBuild the JS application. The new build files can be found inckanext/dcat_usmetadata/publicfolder.Pass the--emptyOutDirflag to replace an existing build.yarnbuildBuild and start the docker containers.yarnbuild:docker
yarnupTestingThere are several levels of testing:SuiteDescriptionCommandUnit tests for the JS appTests for the React app.yarn test:metadata-appCKAN extension testsPython tests using Nosetestsyarn testEnd to end testsCypress testsyarn e2eLintingLint the python code.yarnlint:pythonLint the JavaScript code.yarnlint:jsMetadata appThe Metadata app was aCreate React App-bootstrapped project converted to useViteas the build tool.To run the app useyarn && yarn startcommand.TODO briefly describe how the metadata application relates to the CKAN
extension.DevelopmentThis project usescosmosfor development.Run CKAN locally (yarn up), get the Admin user's API Key and add it in/metadata-app/public/index.htmlasdata-apiKeyattribute of thedivelement. Add a test org for development purposes.Runyarn && yarn cosmosto start the cosmos server, which will watch themetadata-app/srcdirectory for changes.Run the unit tests:yarntest:metadata-app# To run it in watch mode:yarntest:metadata-app:watchUpdate Jest snapshotsSome tests render a fixture component withJestand then
match against a known good snapshot (HTML rendering) of the component. When you
edit a component, you'll usually have to update the snapshot and inspect the
diff to make sure all changes are as intended.yarntest:metadata-app--updateSnapshotLocal development and end-to-end testingTo build the latest JS code and update assets in the CKAN extension, you can run the following command from the root directory of this project:yarn buildFor convenience, we have prepared a single script that you can run to perform end-to-end tests locally. Don't forget toyarn buildprior to running e2e tests, otherwise, the tests could run against older builds:yarne2eNote, it may be necessary to remove cached images when rebuilding the docker container, in order to ensure that the new usmetadata-app template is included in the build. If you want to make sure that you aren't using cached builds, you can try:docker-composebuild--no-cache--pullckanext-dcat_usmetadata_appTo run e2e tests interactively use:yarne2e:interactiveLocally installing this extentionYou can install this extension locally (toinventory app, for example), but you mustyarn installand thenyarn buildthe JS/CSS prior to launching the app locally.Publishing a new version of the extensionWe publish this extension to PyPI -https://pypi.org/project/ckanext-dcat-usmetadata/. This is done by CI job that is triggered on tagged commit on master branch. When you need to release a new version of the extension, you need to:Create a new branch for releasing a new version of the extension. You can name your branch with the following convention:release/x.y.z;Update version insetup.py;Get your PR merged to master branch;Tag the merged commit with the new version (git tag $version).In the CI job, the following is done for tagged commits:It builds the JS bundles and puts them into the relevant directory so the extension can use them;It runs integration tests to make sure everything is working as expected;It packages the extension and publishes it to PyPI.Below is a sequence diagram demonstrating the flow (you need to havegithub + mermaidchrome extension to view it):sequenceDiagram
    Developer->>Git: Push tagged commit to master branch
    Git-->>CI/CD: Trigger deployment
    CI/CD-->>CI/CD: Build assets (JS bundles)
    CI/CD-->>CI/CD: Build python package
    CI/CD-->>CI/CD: Run tests
    CI/CD-->>PyPI: Publish the package
    Inventory-->>PyPI: InstallWays to ContributeThe Data.gov team manages all Data.gov updates, bugs, and feature additions via GitHub's publicissue trackerin this repository.If you do not already have a GitHub account, you cansign up for GitHub here. In the spirit of open source software, everyone is encouraged to help improve this project. Here are some ways you can contribute:by reporting bugsby suggesting new featuresby translating content to a new languageby writing or editing documentationby writing specificationsby writing code and documentation (no pull request is too small: fix typos, add code comments, clean up inconsistent whitespace)by reviewing pull requests.by closing issuesSubmit Great IssuesBefore submitting a new issue, check to make sure a similar issue isn't already open. If one is, contribute to that issue thread with your feedback.When submitting a bug report, please try to provide as much detail as possible, i.e. a screenshot or gist that demonstrates the problem, the technology you are using, and any relevant links.Ready for your HelpIssues labeledhelp wantedmake it easy for you to find ways you can contribute today.Public DomainThis project constitutes a work of the United States Government and is not subject to domestic copyright protection under 17 USC § 105. Additionally, we waive copyright and related rights in the work worldwide through the CC0 1.0Universal public domain dedication.All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.