Package: aircan
AirCanLoad data intoCKAN DataStoreusing Airflow as the runner. This is a replacement for DataPusher and Xloader.Clean separation of components so you can reuse what you want (e.g., you don't use Airflow but your own runner).AirCanGet StartedExamplesExample 1: CSV to JSONUsing Aircan DAGsExample 2: Local file to CKAN DataStore using the Datastore APIPreliminaries: Setup your CKAN instanceIgnore Example 3: Local file to CKAN DataStore using PostgresPreliminaries: Setup your CKAN instanceDoing the loadExample 2a: Remote file to DataStoreExample 3: Auto Load file uploaded to CKAN into CKAN DataStoreRun itTutorialsUsing Google Cloud ComposerGet StartedInstallPython>= 3.5 <= 3.7.x (and make a virtual environment).Cloneaircanso you have examples available:gitclonehttps://github.com/datopian/aircanInstall and setupAirflow(https://airflow.apache.org/docs/stable/installation.html):exportAIRFLOW_HOME=~/airflow
pipinstallapache-airflow
airflowinitdbNote: On recent versions of Python (3.7+), you may face the following error when executingairflow initdb:ModuleNotFoundError:Nomodulenamed'typing_extensions'This can be solved withpip install typing_extensions.Then, start the server and visit your Airflow admin UI:airflowwebserver-p8080By default, the server will be accessible athttp://localhost:8080/as shown in the output of the terminal where you ran the previous command.ExamplesExample 1: CSV to JSONIn this example we'll run an AirCan example to convert a CSV to JSON.Add the DAG to the default directory for Airflow to recognize it:mkdir~/airflow/dags/
cpexamples/aircan-example-1.csv~/airflow/
cpexamples/csv_to_json.py~/airflow/dags/To see this DAG appear in the Airflow admin UI, you may need to restart the server or launch the scheduler to update the list of DAGs (this may take about a minute or two to update, then refresh the page on the Airflow admin UI):airflowschedulerRun this DAG:Enable the dag in theadmin UIwith this toggle to make it run with the scheduler:"Trigger" the DAG with this button:After a moment, check the output. You should see a successful run for this DAG:Locate the output on disk at~/airflow/aircan-example-1.jsonUsing Aircan DAGs in a local Airflow instanceExample 2: Local file to CKAN DataStore using the Datastore APIWe'll assume you have:a local CKAN setup and running athttp://localhost:5000;a dataset (for example,my-dataset) with a resource (for example,my-resourcewith themy-res-id-123as itsresource_id);We also need to set up two environment variables for Airflow. Access theAirflow Variable paneland set upCKAN_SITE_URLand yourCKAN_SYSADMIN_API_KEY:Single-node DAGapi_ckan_load_single_nodeis a single-node DAG which deletes,creates and loads a resource to a local or remote CKAN instance. You can run theapi_ckan_load_single_nodeby following these steps:Open yourairflow.cfgfile (usually located at~/airflow/airflow.cfg) and point your DAG folder to AirCan:dags_folder=/your/path/to/aircan

...otherconfigsdag_run_conf_overrides_params=TrueNote: do not pointdags_folderto/your/path/to/aircan/aircan/dags. It must be pointing to the outeraircanfolder.Verify that Airflow finds the DAGs of Aircan by runningairflow list_dags. The output should list:-------------------------------------------------------------------
DAGS
-------------------------------------------------------------------
ckan_api_load_single_step
...otherDAGs...Make sure you have these environment variables properly set up:exportLC_ALL=en_US.UTF-8exportLANG=en_US.UTF-8exportOBJC_DISABLE_INITIALIZE_FORK_SAFETY=YESRun the Airflow webserver (in case you have skipped the previous example):airflow webserverRun the Airflow scheduler:airflow scheduler. Make sure the environment variables from (3) are set up.Access the Airflow UI (http://localhost:8080/). You should see the DAGckan_api_load_single_steplisted.Activate the DAG by hitting on theOffbutton on the interface.Now we can test the DAG. On your terminal, run:airflowtest\-tp"{ \"resource_id\": \"my-res-id-123\", \\"schema_fields_array\": \"[ 'field1', 'field2']\", \\"csv_input\": \"/path/to/my.csv\", \\"json_output\": \"/path/to/my.json\" }"\ckan_api_load_single_stepfull_load_via_apinowMake sure to replace the parameters accordingly.resource_idis the id of your resource on CKAN.schema_fields_arrayis the header of your CSV file. Everything is being treated as plain text at this time.csv_inputis the path to the CSV file you want to upload.The DAG will convert your CSV file to a JSON file and then upload it.json_outputspecifies the path where you want to dump your JSON file.Check your CKAN instance and verify that the data has been loaded.Trigger the DAG with the following:airflowtrigger_dagckan_api_load_single_step\--conf='{ "resource_id": "my-res-id-123", "schema_fields_array": [ "field1", "field2" ], "csv_input": "/path/to.csv", "json_output": "/path/to.json" }'Do not forget to properly replace the parameters with your data and properly escape the special characters.
Alternatively, you can just run the DAG with theairflow runcommand.api_ckan_load_single_nodealso works for remote CKAN instances. Just set up your AirflowCKAN_SITE_URLvariable accordingly.Multiple-node DAGckan_api_load_multiple_stepsdoes the same steps ofapi_ckan_load_single_node, but it uses multiple nodes (tasks). You can repeat the steps of the previous section and runckan_api_load_multiple_steps.[Ignore] Example 3: Local file to CKAN DataStore using PostgresWe'll load a local csv into CKAN DataStore instance.Preliminaries: Setup your CKAN instanceWe'll assume you have:a local CKAN setup and running athttp://localhost:5000datastore enabled. If you are using Docker, you might need to expose your Postgres Instance port. For example, add the following in yourdocker-composer.ymlfile:db:
  ports:
      - "5432:5432"(Useful to know: it is possible to access the Postgres on your Docker container. Rundocker psand you should see a container nameddocker-ckan_db, which corresponds to the CKAN database. Rundocker exec -it CONTAINER_ID bashand thenpsql -U ckanto access the corresponding Postgres instance).Now you need to set up some information on Airflow. Access your local Airflow Connections panel athttp://localhost:8080/admin/connection/. Create a new connection namedckan_postgreswith your datastore information. For example, assuming yourCKAN_DATASTORE_WRITE_URL=postgresql://ckan:ckan@db/datastore, use the following schema:We also need to set up two environment variables for Airflow. Access the Airflow Variable panel and set upCKAN_SITE_URLand yourCKAN_SYSADMIN_API_KEY:[TODO PARAMETERIZE VARS]
[TODO PARAMETERIZE PATHS]Then, create a dataset calledaircan-exampleusing this script:cdaircan
pipinstall-rrequirements-example.txt
pythonexamples/setup-ckan.py--api-keyDoing the loadWe assume you now have a dataset namedmy-first-dataset.Create the DAG for loadingcpaircan/lib/api_ckan_load.py~/airflow/dags/Check if Airflow recognize your DAG withairflow list_dags. You should see a DAG namedckan_load.Now you can test each task individually:To delete a datastore, runairflow test ckan_load delete_datastore_table nowTo create a datastore, runairflow test ckan_load create_datastore_table now. You can see the correspondingresource_idfor the datastore on your logs. [TODO JSON is hardcoded now; parameterize on kwargs or some other operator structure]To load a CSV to Postgres, runairflow test ckan_load load_csv_to_postgres_via_copy now. [TODO JSON is hardcoded now; insert resource_id on it. File path is also Hardcode, change it]Finally, set your datastore to active:airflow test ckan_load restore_indexes_and_set_datastore_active now.To run the entire DAG:Select the DAG [screenshot]Configure it with a path to ../your/aircan/examples/example1.csvRun it ... [screenshot]Check the outputVisithttp://localhost:5000/dataset/aircan-example/and see the resource named XXX. It will have data in its datastore now!Example 2a: Remote file to DataStoreSame as example 2 but use this DAG instead:cpaircan/examples/ckan-datastore-from-remote.py~/airflow/dags/Plus set a remote URL for loading.Examples 3: Auto Load file uploaded to CKAN into CKAN DataStoreConfigure CKAN to automatically load.Setup CKAN - see the previous sections.Also install this extension in your ckan instance:ckanext-aircan-connector.TODO: add instructions.Configure ckan with location of your airflow instance and the DAG id (aircan-load-csv).Run itRun this script which uploads a CSV file to your ckan instance and will trigger a load to the datastore.cdaircan
pipinstall-rrequirements-example.txt
pythonexamples/ckan-upload-csv.pyUsing Google Cloud ComposerSign up for an account athttps://cloud.google.com/composer. Create or select an existing project at Google Cloud Platform. For this example, we use one calledaircan-test-project.Create an environment at Google Cloud Composer, either by command line or by UI. Make sure you selectPython 3when creating the project. Here, we create an environment namedaircan-airflow.After creating your environment, it should appear in your environment list:Override the configuration fordag_run_conf_overrides_params:Access the designated DAGs folder (which will be a bucket). Upload the contents oflocal/path/to/aircan/aircanto the bucket:The contents of the subfolderaircanmust be:Enter the subdirectorydagsand delete the__init__.pyfile on this folder. It conflicts with Google Cloud Composer configurations.Similarly to what we did on Example 2, access your Airflow instance (created by Google Cloud Composer) and addCKAN_SITE_URLandCKAN_SYSADMIN_API_KEYas Variables. Now the DAGs must appear on the UI interface.Let's assume you have a resource onhttps://demo.ckan.org/withmy-res-id-123as its resource_id. We also assume you have, in the root of your DAG bucket on Google Cloud platform, two files: One CSV file with the resource you want to upload, namedr3.csv, with two columns,field1andfield2. The other file you must have in the root of your your bucket isr4.json, an empty JSON file.Since our DAGs expect parameters, you'll have to trigger them via CLI:
For example, to triggerapi_ckan_load_single_node, run (from your terminal):gcloudcomposerenvironmentsrunaircan-airflow\--locationus-east1\trigger_dag--ckan_api_load_single_step\--conf='{ "resource_id": "my-res-id-123", "schema_fields_array": [ "field1", "field2" ], "csv_input": "/home/airflow/gcs/dags/r3.csv", "json_output": "/home/airflow/gcs/dags/r4.json" }'Check the logs (tip: filter them by your DAG ID, for example,ckan_api_load_single_step). It should updload the data of your.csvfile todemo.ckansuccessfully.