Package: d2s
A Command Line Interface to help orchestrate the integration of heterogenous data sources under a commonRDF Knowledge Graphusing Python, RML mappings, Bash, and GitHub Actions workflows (YAML).You can find more informations about the Data2Services project on thed2s documentation website 📖InstallationRequirements:Python 3.7+GitOptional: Java 11+ to used2s sparql uploadOptional:occommand line toolfor deploying to theDSRI OpenShift cluster(for Maastricht University academics and students)Install from pypipipinstalld2sUsepip, pip3 orpipxdepending on your preferences.Updatepipinstall--upgraded2sInstall from GitHub branchYou can also install it from themasterbranch, if you want the latest updates:pipinstallgit+https://github.com/MaastrichtU-IDS/d2s-cli.git@masterUninstallpipuninstalld2sUse d2sDisplay the default help commandd2sGenerate metadataAnalyze a SPARQL endpoint metadata to generateHCLS descriptive metadatafor each graph:d2smetadataanalyzehttps://graphdb.dumontierlab.com/repositories/umids-kg-ometadata.ttlAnalyze a SPARQL endpoint metadata to generate metadata specific to Bio2RDF for each graph:d2smetadataanalyzehttps://bio2rdf.137.120.31.102.nip.io/sparql-ometadata.ttl-mbio2rdfYou can also generate detailed HCLS metadata for the dataset version and distribution by answering the questions after running this command:d2smetadatacreate-ometadata.ttlBootstrap a datasets conversion projectd2scan be used to help you converting datasets to RDF.You will need to initialize the current folder, it is highly recommended to do this at the root of a Git repository where the conversion will be stored:d2sinitThis command will create adatasetsfolder to store the datasets conversions and a.github/workflowsfolder for the workflows, if it does not exist already.Alld2scommands are designed to be run from the project folderYou can create a new dataset conversion:d2snewdatasetYou will be asked a few questions about the dataset via the terminal, then a folder will be generated with:Your dataset metadataExample YARRRML and RML mappingsExample python preprocessing scriptExample bash script to download the data to convertA GitHub Action workflow to run the different steps of the processingYou can now edit the file generated in thedatasetsfolder to implement your data conversion.Run the RML mapperRequirements: Java installedThis feature is still experimentald2scan be used to easily run the RML mapper:d2srmlmy-datasetEnable autocompletionEnable commandline autocompletion in the terminalRecommended, it makesd2smuch more user-friendlyZSH: add the import autocomplete line to the~/.zshrcfile.echo'eval "$(_D2S_COMPLETE=source_zsh d2s)"'>>~/.zshrcSet your terminal to useZSHby default:chsh-s/bin/zshAoh-my-zshtheme can be easily chosen for a personalized experience. Seethe zsh-theme-biradateto easily install a simple theme and configure your terminal in a few minutes.Bash: add the import autocomplete line to the~/.bashrcfile. Something like this probably:echo'eval "$(_D2S_COMPLETE=source d2s)"'>>~/.bashrcBuild and publishInstall d2s for developmentInstalld2sas executable to run it from the terminalClone the repository:gitclonehttps://github.com/MaastrichtU-IDS/d2s-cli.gitcdd2s-cliInstalld2s:pipinstall-e.d2swill be updated directly on change in the code.Optional: isolate with a Virtual EnvironmentIf you face conflicts with already installed packages, then you might want to use aVirtual Environmentto isolate the installation in the current folder before installingd2s:# Create the virtual environment folder in your workspacepython3-mvenv.venv# Activate it using a script in the created foldersource.venv/bin/activatePublish using DockerTo publish a new version onpypi:upgrade the version insetup.py(e.g. from0.2.1to0.2.2)use the following script to build and publish automatically usingDocker:./publish_pip.shA test will be run using Docker before publishing to make sured2s initworks.Build locallyBuilding and publishing can be done locally:# Build packages in dist/ folderpython3setup.pysdistbdist_wheel# Publish packages previously built in the dist/ foldertwineupload--repository-urlhttps://upload.pypi.org/legacy/dist/*Additional instructions to install twine locally (not needed)pipinstalltwineIf you experience issues with Bash or ZSH becaused2sis not defined when installing for dev. Then addpip install --editable develop/d2s-clito.zshrcYou might need to install Python3.7 for dev (dev with python3.7 should work though)sudoapt-getinstallpython3.7python3.7-venvpython3.7-dev# Set python3 to use 3.7sudoupdate-alternatives--install/usr/bin/python3python3/usr/bin/python3.71sudoupdate-alternatives--configpython3vim/usr/bin/gnome-terminal#!/usr/bin/python3.7If you face issue uploading the package on pypi:twinecheckdist/d2s-*-py3-none-any.whl