Package: qalita
QALITA Command Line Interface (CLI)QALITA Command Line Interface (CLI) is a tool intended to be used by Data Engineers who setup's QALITA Platform's agents, sources and assets.It gives easy to use command to help them make an up & running qalita platform's environment in no time.QALITA Command Line Interface (CLI)Quick StartInstallationUsageSetupMinimal ConfigConnected ConfigMake an .env file and export ENV Values:qalita agentqalita agent loginqalita agent runJobWorkerqalita agent joblistqalita agent infoqalita packqalita pack initqalita pack listqalita pack runqalita pack validateqalita pack pushqalita sourceqalita source addqalita source validateqalita source listqalita source pushHow to 'Pack' ?InitAt runtimePost runtimeQuick StartInstallationAs simple as :pipinstallqalitaor there is aDocker Imageavailable at :UsageIf you want to have more detailed and contextual help, typeqalita COMMAND -hUsage:qalita[OPTIONS]COMMAND[ARGS]...QALITACommandLineInterfaceSetupThis CLI command communicates with the QALITA Platform API backend.There are several layers of configuration depending of your needs :Minimal ConfigQALITA_AGENT_NAME=<agent_name>The agent will help you identify it in the frontend interface, there are no restrictions on the name.QALITA_AGENT_MODE=<job/worker>The mode of the agent :Job: In job mode, when you use the commandqalita agent run, it will immediately try to run a job on the local current context.Worker: In worker mode, when you use the commandqalita agent runit will wait for the backend to gives him jobs to run. It is simmilar to a scheduler.Note that the commandqalita agent runneeds more configuration to run correctly, it will displays error otherwise.Connected ConfigQALITA_AGENT_ENDPOINT=<backend_api_url>Example :http://localhost:3080/api/v1The agent url endpoint gives the ability for the agent to communicate with the qalita's platform endpoints, it enables :* Listing packs
* Running Jobs
* Publishing sources
* Publishing packsQALITA_AGENT_TOKEN=<api_token>The token is provided while doing the quickstart steps in the frontend app. It is associated with your user and your role.Note that you need to have at least the[Data Engineer]role to use the QALITA CLIMake an .env file and export ENV Values:You can alternatively make an .env file and export the values to your environment..env-localQALITA_AGENT_NAME=<agent_name>QALITA_AGENT_MODE=<job/worker>QALITA_AGENT_ENDPOINT=https://api.company.com/api/v1QALITA_AGENT_TOKEN=<api_token>QALITA_PACK_NAME=<pack_name>Then export the values of the file to ENV values with :export$(xargs<.env-local)qalita agentTheqalita agentcommand allow you to :Register an agent to the platformGet information about your local agentRun a pack on a sourceList agent jobs (past & future)qalita agent loginParameters :name: the name of the agentmode: the mode of the agent <job/worker>token: the api token you get from the platformurl: the backend api url of the platformqalita agent loginregisters your local agent to the platform, it enables you to run jobs, or create routines (schedules) to run pack programmaticaly.You need to have configured your agent with :

* QALITA_AGENT_ENDPOINT=<backend_api_url>
* QALITA_AGENT_TOKEN=<api_token>You can get your token from the frontend or with an OAUTH2 API call to the /users/signin backend's endpointMore info on your frontend documentation, and on theConnected configof the docqalita agent runParameters :--name: the name of the agent--mode: the mode of the agent <job/worker>--token: the api token you get from the platform--url: the backend api url of the platformSpecific parameters injobmode :--source: the source id you want to run your job against--source-version(optional) : the source version, by default it will run to the latest soruce version--pack: the pack id you want to run your job against--pack-version(optional) : the pack version, by default it will run the latest version of the packqalitaagentrunruns in different mode :JobThe agent will run given configuration-p: a pack_id given with theqalita pack list, note that your pack needs to be pushed to the platform in order to have an id.-s: a source_id given with theqalita source list, note that your source needs to be pushed to the platform in order to have an id.WorkerThe agent will wait until it receives an order from the frontend, it will then worke as same as in job mode.Note that this mode will run indefinitelyqalita agent joblistParameters :--name: the name of the agent--mode: the mode of the agent <job/worker>--token: the api token you get from the platform--url: the backend api url of the platformList jobs from the platform backend.qalita agent infoParameters :--name: the name of the agent--mode: the mode of the agent <job/worker>--token: the api token you get from the platform--url: the backend api url of the platformGet infos about your local agent configuration.qalita packTheqalita packcommand allow you to :Initialize a new packList all available packsValidate itRun a local packPush your pack version to the platformqalita pack initParameters :--name: the name of the packInitialize a new pack, you need to have set aname, it will create a newfolderwith the name of the pack.You can set your name by passing a new parameters to the commandline or setting a new environment variable :QALITA_PACK_NAME=<my-super-pack>.Here is the arborescence created :./<pack-name>_pack/
        /run.sh             # Entrypoint file that will be run with qalita agent run
        /README.md          # Documentation file
        /properties.yaml    # Properties file that contains properties about the pack
        /main.py            # (pack specific) The main script (you can run your pack with whatever langage you choose)
        /config.json        # (pack specific) The config file of your pack, you can use it to set any configurations you like.
        /requirements.txt   # (pack specific) The requirements file that is run inside the run.shqalita pack listParameters :You need to have logged in withqalita agent loginList all the packs that are accessible to you with the Qalita Platform.qalita pack runParameters :--name: Pack nameRun your locally configured packqalita pack validateParameters :--name: Pack nameValidate your locally configured packqalita pack pushParameters :--name: Pack namePush your locally configured packqalita sourceTheqalita sourcecommand allow you to :Add a new source to your local configurationList your local sources from yourqalita-conf.ymlfilePush your local sources from yourqalita-conf.ymlfileValidate your conf fileqalita-conf.ymlNote , by default theqalita-conf.ymlfile is stored to~/.qalita/qalita-conf.yml, setQALITA_HOMEenv to customize the default path.qalita source addThis function will help you add a new source to your configuration fileqalita-conf.yamlThis command doesn't have parameters, you need to follow command prompts.Prompt 1: Source namePrompt 2: Source typePrompt 3: Is conditionnal, depends on the source type.Case : Source Type =file: Source pathCase : Source Type =database: host / port / username / password / databasePrompt 4: Source descriptionPrompt 5: Is the source a reference ? [bool] (default : false)Prompt 6: Is the source sensistive ? [bool] (default : false)Prompt 7: Visibility of the source (private, internal, public) (default : private)At the end of the prompt, the cli will check reachability of the source depending of theconfigurationandtype, this step is calledvalidate_sourceto complete the process of registering a new source to the platform, you need to push your source with the command :qalita source pushqalita source validateHelper function to help you add a new source to your configuration fileqalita-conf.yamlqalita source listParameters :You need to have aqalita-conf.yamlfile that contains your sources configuration.Exemple :version:1sources:-config:path:/home/user/data_dirdescription:Folder containing csv filesname:my_csv_filesowner:userreference:falsevisibility:privatesensitive:falsetype:fileIn this exemple we have :General keysKeyTypeDescriptionversionintThe version of the configurationsourceslistThe list of sourcesSource keysKeyTypeDescriptionnamestringThe name of the sourcedescriptionstringThe description of the sourceownerstringThe owner of the sourcetypestringThe type of the sourceconfigdictThe configuration of the sourcevisibilitystringThe visibility of the source <private/internal/public>referenceboolIs the source a reference sourcesensitiveboolIs the source containing sensitive dataqalita source pushRegisters your sources to the platformNote: If you want to run a pack on your source, you will first need to push your source to the platform. It will give you a source_id with which you can run your pack.How to 'Pack' ?A pack is an entity run by the agent, it can be created by anyone.It's purpose is to process the source and retrieve usefull informations about it to feed back into the platform.InitTo create the base pack, see :qalita pack init.At runtimeThe entrypoint of the pack is therun.shfile that is located at theroot pathof the temp local folder created by the agent.run.shExample :#/bin/bashpython-mpipinstall--quiet-rrequirements.txtpythonmain.pyThe pack is feed by asource_conf.jsonfile containing the source'sconfig:data. This file is located alongside therun.shentrypoint.source_conf.jsonExample :{"config":{"path":"/home/lucas/desktop"},"description":"Desktop files","id":1,"name":"local_data","owner":"lucas","type":"file","reference":false,"sensitive":false,"visibility":"private","validate":"valid"}Note : The pack is responsible for managing itself it'ssource typecompatibility by checking the sourcetypein thesource_conf.jsonfile.Post runtimeA the end of the pack run, the agent searchs for :logs.txt: File uploaded to give feedback logs to the platform in the frontend.logs.txtExample :2023-07-21 11:51:12,688 - qalita.commands.pack - INFO - ------------- Pack Run -------------
2023-07-21 11:51:15,087 - qalita.commands.pack - INFO - CSV files found:
2023-07-21 11:51:15,222 - qalita.commands.pack - ERROR - Summarize dataset:   0%|          | 0/5 [00:00<?, ?it/s]
...recommendations.jsonRecommendations file contains the recommendations given by the pack about the source.recommendations.jsonExample :{[{"type":"<type of recommendation>","scope":"<scope : is a json>","content":"<any content>"},{...}...]}metrics.jsonMetrics file contains the metrics given by the pack about the source.metrics.jsonExample :{[{"scope":"<scope : is a json>","key":"<metric key>","value":"<metric value>"},{...}...]}Metrics & recommendations are pushed to the platform and are then available to the source's pack run view.