Package: airflow-provider-kinetica
1. Kinetica Provider for Apache AirflowTheairflow-provider-kineticapackage provides a SQL operator and hook for Kinetica.1. Overview2. Installation3. Testing3.1. Configure Conda environment3.2. Install Airflow3.3. Start Airflow in Standalone mode3.4. Install the package in editable mode.4. Building5. See Also1. OverviewFeatures included in this package are:Airflow hookKineticaSqlHookAirflow operatorKineticaSqlOperatorCustom connection type with customized connection UI.Relevant files are:FileDescriptionkinetica_provider/get_provider_info.pyProvider infoexample_dags/kinetica_sql_example.pyExample DAG with operator and hook.kinetica_provider/operator/sql.pyContains KineticaSqlHookkinetica_provider/hooks/sql.pyContains KineticaSqlOperator2. InstallationThis step assumes that you have an existing.whldistribution of the package. You can eitherbuild the distributionor download it from the assets section of theGithub release.$pipinstall./dist/airflow_provider_kinetica-1.0.0-py3-none-any.whl[...]Successfullyinstalledairflow-provider-kinetica-1.0.0You will need to create a default connection namedkinetica_default. You can do this in the web UI or with the following syntax:$airflowconnectionsadd'kinetica_default'\--conn-type'kinetica'\--conn-login'admin'\--conn-password'???'\--conn-host'http://hostname:9191/'3. TestingThis section explains how to setup an environment used for build and test.3.1. Configure Conda environmentTo run Airflow we need a specific version of python with its dependencies and so we will use miniconda.The following steps show how to install miniconda on Linux. You should check theMiniconda documentationfor the most recent install instructions.[~]$wgethttps://repo.anaconda.com/miniconda/Miniconda3-py38_23.3.1-0-Linux-x86_64.sh[~]$bashMiniconda3-py38_23.3.1-0-Linux-x86_64.shAfter installing make sure you are in thebaseconda environment. Next we crate anairflowconda environment.(base)[~]$condacreate--nameairflowpython=3.8(base)[~]$condaactivateairflow(airflow)[~]$3.2. Install AirflowThese steps will show how to configure astandalone Airflow environment.Note: Before starting make sure you have activated theairflowconda envionmnet.Determine the download URL of the airflow installer.(airflow)[~]$AIRFLOW_VERSION=2.6.1(airflow)[~]$PYTHON_VERSION="$(python--version|cut-d" "-f2|cut-d"."-f1-2)"(airflow)[~]$CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"(airflow)[~]$echo$CONSTRAINT_URLhttps://raw.githubusercontent.com/apache/airflow/constraints-2.6.1/constraints-3.8.txtInstall the Airflow package.(airflow)[~]$pipinstall--upgradepip(airflow)[~]$pipinstall"apache-airflow==${AIRFLOW_VERSION}"--constraint"${CONSTRAINT_URL}"3.3. Start Airflow in Standalone modeYou must provide a location that will be used for the$AIRFLOW_HOME. We set this in the conda environment.(airflow)[~]$condaenvconfigvarssetAIRFLOW_HOME=~/fsq-airflow/airflow/standalone(airflow)[~]$condaenvconfigvarslist-nairflowAIRFLOW_HOME=~/fsq-airflow/airflow/standaloneYou must re-activate the environment for the variable to get loaded.(airflow)[~]$condaactivateairflow(airflow)[~]$echo$AIRFLOW_HOME~/fsq-airflow/airflow/standaloneWhen you startup airflow in standalone mode it will copy files into$AIRFLOW_HOMEif they do not already exist. When startup is complete it will show the admin and user password for the webserver.(airflow)[~]$cd$AIRFLOW_HOME(airflow)[standalone]$airflowstandalone[...]standalone|Airflowisready
standalone|Loginwithusername:adminpassword:39FrRzqzRYTK3pc9
standalone|AirflowStandaloneisfordevelopmentpurposesonly.Donotusethisinproduction!You can edit theairflow.cfgfile if you need to change any ports.3.4. Install the package in editable mode.When a package is installed for edit the contents of the specified directory get registered with the python environment. This allows for changes to be made without the need for reinstalling.Change to the location of the package and install it as editable.(airflow)[~]$cd~/fsq-airflow/airflow/airflow-provider-kinetica(airflow)[airflow-provider-kinetica]$pipinstall--editable.Now you can restart airflow to see the installed provider. Uninstall the package when you are done.(airflow)[airflow-provider-kinetica]$pythonsetup.pydevelop--uninstall4. BuildingThe conda environment created for testing can also be used for building. You will need thebuildpackage.(airflow)[~]$pipinstallbuildFrom the location of the provider execute the build process.(airflow)[~]$cd~/fsq-airflow/airflow/airflow-provider-kinetica(airflow)[airflow-provider-kinetica]$python-mbuild[...]Successfullybuiltairflow-provider-kinetica-1.0.0.tar.gzandairflow_provider_kinetica-1.0.0-py3-none-any.whlIt will create a "wheel" distribution package and you can use this to install the provider. If you have an editable version of the provider from the above section you should uninstall it first.(airflow)[airflow-provider-kinetica]$ls-1./dist
airflow_provider_kinetica-1.0.0-py3-none-any.whl
airflow-provider-kinetica-1.0.0.tar.gz(airflow)[airflow-provider-kinetica]$pipinstall./dist/airflow_provider_kinetica-1.0.0-py3-none-any.whl5. See AlsoKinetica DocsKinetica Python APIKinetica SQLAirflow DocsAirflow QuickstartManaging ConnectionsSQL OperatorsBuilding a ProviderAirflow Provider SamplePython Build ModuleSetuptools Wheels