Package: kudaf_datasource_tools
KUDAF Datasource CLI ToolsThis is a set of Command Line Interface (CLI) tools to facilitate the technical tasks requirered from Data Providers that want to make their data available on the KUDAF data-sharing platform.The CLI can create the following Kudaf Data Source components:Variables Metadata, and /orREST API Datasource back-end (including variables metadata and possibly even the data files themselves)About KUDAF(Summary info and links for the Kudaf initiative)High-level workflow for Data Source administrators(Summary of the process for Datasource admins to make their data available in KUDAF, and where these CLI Tools fit in that journey)Local installation instructions (Linux/Mac)Make sure Python3 is installed on your computer (versions from 3.8 up to 3.11 should work fine)$python3 --versionNavigate to the folder chosen to contain this project$cd path/to/desired/folderCreate a Python virtual environment and activate it$python3 -m venv .venvThis created the virtualenv under the hidden folder.venvActivate it with:$source .venv/bin/activateInstall Kudaf Datasource Tools and other required Python packages$pip install kudaf_datasource_toolsCreating a YAML configuration fileClick here for abasic YAML syntax tutorialExample YAML configuration fileThe following file is included in the package and can be found in thedatasource_tools/configfolder:config_example.yaml---projectInfo:organization:"my-short-organizations-name"author:Author NamedatasourceName:"my-FeideKundeportal-Datasource-name"datasourceId:"my-FeideKundeportal-Datasource-UUID"dataFiles:-dataFile:&mydatafilefileNameExt:mydatafile.csvcsvParseDelimiter:";"# Valgfritt (som standard ","). Angir tegnet som brukes i CSV-filen for å skille verdier innenfor hver radfileDirectory:/path/to/my/datafiles/directory# Bare nødvendig hvis forskjellig fra gjeldende katalogunitTypes:-MIN_ENHETSTYPE1:&min_enhetstype1# Bare nødvendig hvis forskjellig fra de globale enhetstypene: PERSON/VIRKSOMHET/KOMMUNE/FYLKEshortName:MIN_ENHETSTYPE1# This shows as the Key indicator in the Front-endname:Kort identifikasjonsetikett# This will label the Identifier blue boxdescription:Detaljert beskrivelse av denne enhetstypendataType:LONG# En av STRING/DATE/LONG/DOUBLE-MIN_ENHETSTYPE2:&min_enhetstype2# Bare nødvendig hvis forskjellig fra de globale enhetstypene: PERSON/VIRKSOMHET/KOMMUNE/FYLKEshortName:MIN_ENHETSTYPE2# This shows as the Key indicator in the Front-endname:Kort identifikasjonsetikett# This will label the Identifier blue boxdescription:Detaljert beskrivelse av denne enhetstypendataType:LONG# En av STRING/DATE/LONG/DOUBLEvariables:-name:VARIABELENS_NAVNtemporalityType:FIXED# En av FIXED/EVENT/STATUS/ACCUMULATEDdataRetrievalUrl:https://my-datasource-api.no/api/v1/variables/VARIABELENS_NAVNsensitivityLevel:NONPUBLIC# En av PUBLIC/NONPUBLICpopulationDescription:-Beskrivelse av populasjonen som denne variabelen målerspatialCoverageDescription:-Norge-Annen geografisk beskrivelse som gjelder disse dataenesubjectFields:-Temaer/konsepter/begreper som disse dataene handler omidentifierVariables:-unitType:*min_enhetstype1# Kan også være en av de globale enhetstypene: PERSON/VIRKSOMHET/KOMMUNE/FYLKEmeasureVariables:-label:Kort etikett på hva denne variabelen måler/viserdescription:Detaljert beskrivelse av hva denne variabelen måler/viserdataType:STRING# En av STRING/LONG/DATE/DOUBLEdataMapping:dataFile:*mydatafileidentifierColumns:-Min_Identificatorkolonne# CSV-filkolonne for identifikatorenmeasureColumns:-Min_Målkolonnen# CSV-filkolonne for det som måles-name:VARIABELENS_NAVN_ACCUMtemporalityType:ACCUMULATED# En av FIXED/EVENT/STATUS/ACCUMULATEDdataRetrievalUrl:https://my-datasource-api.no/api/v1/variables/VARIABELENS_NAVNsensitivityLevel:NONPUBLIC# En av PUBLIC/NONPUBLICpopulationDescription:-Beskrivelse av populasjonen som denne variabelen målerspatialCoverageDescription:-Norge-Annen geografisk beskrivelse som gjelder disse dataenesubjectFields:-Temaer/konsepter/begreper som disse dataene handler omidentifierVariables:-unitType:*min_enhetstype2# Kan også være en av de globale enhetstypene: PERSON/VIRKSOMHET/KOMMUNE/FYLKEmeasureVariables:-label:Kort etikett på hva denne variabelen måler/viserdescription:Detaljert beskrivelse av hva denne variabelen måler/viserdataType:LONG# Hvis akkumulerte data må summeres, bør dette enten være en av LONG/DOUBLEdataMapping:dataFile:*mydatafileidentifierColumns:-Min_Identificatorkolonne# CSV-filkolonne for identifikatorenmeasureColumns:-Min_Målkolonnen# CSV-filkolonne for det som målesmeasureColumnsAccumulated:False# Valgfritt (True/False, som standard False). Angir om dataene i filen allerede er akkumulertattributeColumns:# Kun nødvendig for EVENT/STATUS/ACCUMULATED temporalityType(s)-Start_Time# CSV-filkolonnen for starttidspunktet-End_Time# CSV-filkolonne for sluttidspunktet-name:NØKKELVAR_ID-NØKKEL_MÅLE-NOKKELtemporalityType:FIXED# En av FIXED/EVENT/STATUS/ACCUMULATEDdataRetrievalUrl:https://my-datasource-api.no/api/v1/variables/NØKKELVAR_ID-NØKKEL_MÅLE-NOKKELsensitivityLevel:PUBLIC# En av PUBLIC/NONPUBLICpopulationDescription:-Beskrivelse av populasjonen som denne variabelen målerspatialCoverageDescription:-Norge-Annen geografisk beskrivelse som gjelder disse dataenesubjectFields:-Temaer/konsepter/begreper som disse dataene handler omidentifierVariables:-unitType:*min_enhetstype1# ID-NØKKEL - Kan også være en av de globale enhetstypene: PERSON/VIRKSOMHET/KOMMUNE/FYLKEmeasureVariables:-label:Kort etikett på hva denne variabelen måler/viserdescription:Detaljert beskrivelse av hva denne variabelen måler/viserunitType:*min_enhetstype2# MÅLE-NØKKEL - Kan også være en av de globale enhetstypene: PERSON/VIRKSOMHET/KOMMUNE/FYLKEdataType:LONG# En av STRING/LONG/DATE/DOUBLEdataMapping:dataFile:*mydatafileidentifierColumns:-Min_Identificatorkolonne# CSV-filkolonne for identifikatorenmeasureColumns:-Min_Målkolonnen# CSV-filkolonne for det som måles...Kudaf Datasource Tools CLI operationNavigate to the project directory and activate the virtual environment (only if not already activated):$source .venv/bin/activateThekudaf-generatecommandshould be now activated. This is the main entry point to the CLI's functionalities.Displaying the help menus$kudaf-generate --helpUsage: kudaf-generate [OPTIONS] COMMAND [ARGS]...

Kudaf Datasource Tools                                                                                             
                                                                                                                    
╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --install-completion          Install completion for the current shell.                                          │
│ --show-completion             Show completion for the current shell, to copy it or customize the installation.   │
│ --help                        Show this message and exit.                                                        │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ api                 Generate a Kudaf Datasource REST API back-end                                                │
│ metadata            Generate Variables/UnitTypes Metadata                                                        │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯As we can see, there aretwo sub-commandsavailable:apiandmetadata.We can obtainhelpon them as well:$kudaf-generate api --helpUsage: kudaf-generate api [OPTIONS]                                                                                
                                                                                                                    
Generate a Kudaf Datasource REST API back-end                                                                      
                                                                                                                    
╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --config-yaml-path            PATH  Absolute path to the YAML configuration file                                 │
│                                     [default: /home/me/current/directory/config.yaml]                            │
│ --input-data-files-dir        PATH  Absolute path to the data files directory                                    │
│                                     [default: /home/me/current/directory]                                        │
│ --output-api-dir              PATH  Absolute path to directory where the Datasource API folder is to be written  │
│                                     to                                                                           │
│                                     [default: /current/directory]                                                │
│ --help                              Show this message and exit.                                                  │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯$kudaf-generate metadata --helpUsage: kudaf-generate metadata [OPTIONS]                                                                           
                                                                                                                    
Generate Variables/UnitTypes Metadata                                                                              
JSON metadata files ('variables.json' and maybe 'unit_types.json') will be written to the                          
(optionally) given output directory.                                                                               
If any of the optional directories is not specified, the current directory is used as default.                     
                                                                                                                    
╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --config-yaml-path            PATH  Absolute path to the YAML configuration file                                 │
│                                     [default: /home/me/current/directory/config.yaml]                            │
│ --output-metadata-dir         PATH  Absolute path to directory where the Metadata files are to be written to     │
│                                     [default: /home/me/current/directory]                                        │
│ --help                              Show this message and exit.                                                  │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯Generating metadata only from a YAML configuration file$kudaf-generate metadata --config-yaml-path /home/me/path/to/config.yaml --output-metadata-dir /home/me/path/to/metadata/folderGenerating an API$kudaf-generate api --config-yaml-path /home/me/path/to/config.yaml --output-api-dir /home/me/path/to/api/folderLocal API launchNavigate to the folder containing the generated API:$cd /home/me/path/to/api/folderCreate a Python virtual environment and activate it$python3 -m venv .venvThis created the virtualenv under the hidden folder.venvActivate it with:$source .venv/bin/activateInstall needed Python packages$pip install -r requirements.txtLaunch the Kudaf Datasource API$uvicorn app.main:appBrowser: Navigate to the the API's interactive documentation at:http://localhost:8000/docsDocker API launch (Linux)Install Docker for your desktopFollow instructions for your desktop model athttps://docs.docker.com/desktop/Navigate to the folder containing the generated API:$cd /home/me/path/to/api/folderCreate a Docker image$sudo docker build -t my-kudaf-datasource-image .(Note: Thelast.in the above command is important! Make sure it's entered like that )Create and launch a Docker container for the generated image$sudo docker run -d --name my-kudaf-datasource-container -p 9000:8000 my-kudaf-datasource-imageBrowser: Navigate to the the API's interactive documentation at:http://localhost:9000/docsDevelopersDownload the package to your computerOption A: Installation from repository:Open up a Terminal window and clone the repo locally:$git clone https://gitlab.sikt.no/kudaf/kudaf-datasource-tools.gitOption B: Installation from source:Open up yourbrowserand navigate to the project's GitLab page:https://gitlab.sikt.no/kudaf/kudaf-datasource-toolsOnce there,download a ZIP file with the source codeMove the zipped file to whichever directory you want to use for this installationOpen aTerminal window and navigateto the directory where the zipped file isUnzip the downloaded file, it will create a folder calledkudaf-datasource-tools-mainSwitch to the newly created folder$cd path/to/kudaf-datasource-tools-mainMake sure Python3 is installed on your computer (versions from 3.8 up to 3.11 should work fine)$python3 --versionInstall Poetry (Python package and dependency manager) on your computerFull Poetry documentation can be found here:https://python-poetry.org/docs/Theofficial installershould work fine on the command line for Linux, macOS and Windows:$curl -sSL https://install.python-poetry.org | python3 -If the installation was successful, configure this option:$poetry config virtualenvs.in-project trueMac users: TroubleshootingIn case of errors installing Poetry on your Mac, you may have to try installing it withpipx. But to install that, we need to haveHomebrewinstalled first.$/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"(Homebrew documentation:https://brew.sh/)OnceHomebrewis installed, proceed to installpipx:$brew install pipx$pipx ensurepathFinally, installPoetry:$pipx install poetryCreate a Python virtual environment and activate it$python3 -m venv .venvThis created the virtualenv under the hidden folder.venvActivate it with:$source .venv/bin/activateInstall Kudaf Datasource Tools and other required Python packages$poetry install