Package: datapusher-plus
DataPusher+DataPusher+ is a fork ofDatapusherthat combines the speed and robustness ofckanext-xloaderwith the data type guessing of Datapusher.Datapusher+ is built usingCKAN Service Provider, withMessytablesreplaced byqsv.TNRIS/TWDBprovided the use cases that informed and supported the development
of Datapusher+, specifically, to support aResource-first upload workflow.For a more detailed overview, see theCKAN Monthly Live Jan 2023 presentation.It features:"Bullet-proof", ultra-fast data type inferencing with qsvUnlikeMessytableswhich scans only the the first few rows to guess the type of
a column,qsvscans the entire table so its data type inferences are guaranteed[^1].Despite this, qsv is still exponentially faster even if it scans the whole file, not
only inferring data types, it also calculatessummary statisticsas well. For example,scanning a 2.7 million row, 124MB CSV file for types and stats took 0.16 seconds[^2].It is very fast as qsv is written inRust, is multithreaded,
and uses all kinds ofperformance techniquesespecially designed for data-wrangling.Exponentially faster loading speedSimilar to xloader, we use PostgreSQL COPY to directly pipe the data into the datastore,
short-circuiting the additional processing/transformation/API calls used by Datapusher.But unlike xloader, we load everything using the proper data types and not as text, so there's
no need to reload the data again after adjusting the Data Dictionary, as you would with xloader.Far more Storage Efficient AND Performant Datastore with easier to compose SQL queriesAs we create the Datastore tables using the most efficient PostgreSQL data type for each column
using qsv's guaranteed type inferences - the Datastore is not only more storage efficient, it is
also far more more performant for loading AND querying.With its "smartint" data type (with qsv inferring the most efficient integer data type for the range of
values in the column); comprehensive date format inferencing (supporting19 date formats, with each
format having several variants & with configurable DMY/MDY preference parsing) & auto-formatting dates to
RFC3339 format so they are stored as Postgres timestamps; cardinality-aware, configurable auto-indexing;
automatic sanitization of column names to valid PostgreSQL column identifiers; auto PostgreSQL vacuuming &
analysis of resources after loading; and more - DP+ enables the Datastore to tap into PostgreSQL's full power.Configurable auto-aliasing of resources also makes it easier to compose SQL queries, as you can
use more intuitive resource aliases instead of cryptic resource IDs.Production-ready RobustnessIn production, the number one source of support issues is Datapusher - primarily, because of
data quality issues and Datapusher's inability to correctly infer data types, gracefully handle
errors[^3], and provide the Data Publisher actionable information to correct the data.Datapusher+'s design directly addresses all these issues.More informative datastore loading messagesDatapusher+ messages are designed to be more verbose and actionable, so the data publisher's
user experience is far better and makes it possible to have a resource-first upload workflow.Extended preprocessing with qsvqsv is leveraged by Datapusher+ to:create "Smarter" Data Dictionaries, with:guaranteed data type inferencesoptional ability to automatically choose the best integer PostgreSQL data type ("smartint") based on the range of the numeric column (PostgreSQL's int, bigint and numeric types) for optimal storage/indexing efficiency and SQL query performance.sanitized column names (guaranteeing valid PostgreSQL column identifiers) while preserving the original column name as a label, which is used to label columns in DataTables_view.an optional "summary stats" resource as an extension of the Data Dictionary, with comprehensive summary statistics for each column - sum, min/max/range, min/max length, mean, stddev, variance, nullcount, sparsity, quartiles, IQR, lower/upper fences, skewness, median, mode/s, antimode/s & cardinality.convert Excel & OpenOffice/LibreOffice Calc (ODS) files to CSV, with the ability to choose which sheet to use by default (e.g. 0 is the first sheet, -1 is the last sheet, -2 the second to last sheet, etc.)convert various date formats (19 date formats are recognizedwith each format having several variants; ~80 date format permutations in total) to a standardRFC 3339formatenable random access of a CSV by creating a CSV index - which also enables parallel processing of different parts of a CSV simultaneously (a major reason type inferencing and stats calculation is so fast)instantaneously count the number of rows with a CSV indexvalidate if an uploaded CSV conforms to theRFC-4180standardnormalizes and transcodes CSV/TSV dialects into a standard UTF-8 encoded RFC-4180 CSV formatoptionally create a preview subset, with the ability to only download the firstnpreview rows of a file, and not the entire file (e.g. only download first 1,000 rows of 3 gb CSV file - especially good for harvesting/cataloging external sites where you only want to harvest the metadata and a small sample of each file).optionally create a preview subset from the end of a file (e.g. last 1,000 rows, good for time-series/sensor data)auto-index columns based on its cardinality/format (unique indices created for columns with all unique values, auto-index columns whose cardinality is below a given threshold; auto-index date columns)check for duplicates, and optionally deduplicate rowsoptionally screen for Personally Identifiable Information (PII), with an option to "quarantine" the PII-candidate rows in a separate resource, while still creating the screened resource.optional ability to specify a custom PII screening regex set, instead of thedefault PII screening regex set.Even with all these pre-processing tasks, qsv typically takes less than 5 seconds to finish all its analysis tasks, even for a 100mb CSV file.Future versions of Datapusher+ will further leverage qsv's 80+ commands to do additional
preprocessing, data-wrangling and validation. The Roadmap is availablehere.
Ideas, suggestions and your feedback are most welcome![^1]:Why use qsv instead of a "proper" python data analysis library like pandas?[^2]: It takes 0.16 seconds with an index to runqsv statsagainst theqsv whirlwind tour sample fileon a Ryzen 4800H (8 physical/16 logical cores) with 32 gb memory and a 1 TB SSD.
Without an index, it takes 1.3 seconds.
[^3]: Imagine you have a 1M row CSV, and the last row has an invalid value for a numeric column (e.g. "N/A" instead of a number).
After spending hours pushing the data very slowly, legacy datapusher will abort on the last row and the ENTIRE job is invalid.
Ok, that's bad, but what makes it worse is that the old table has been deleted already, and Datapusher doesn't tell you what
caused the job to fail! YIKES!!!!Development InstallationDatapusher+ is a drop-in replacement for Datapusher, so it's installed the same way.Install the required packages.sudoaptinstallpython3-virtualenvpython3-devpython3-pippython3-wheelbuild-essentiallibxslt1-devlibxml2-devzlib1g-devgitlibffi-devlibpq-devfileCreate a virtual environment for Datapusher+ using at least python 3.8.cd/usr/lib/ckan
sudopython3.8-mvenvdpplus_venv
sudochown-R$(whoami)dpplus_venv
.dpplus_venv/bin/activatecddpplus_venvℹ️NOTE:DP+ requires at least python 3.8 as it makes extensive use of new capabilities introduced in 3.7/3.8
to thesubprocess module.
If you're using Ubuntu 18.04 or earlier, follow the procedure below to install python 3.8:sudoadd-apt-repositoryppa:deadsnakes/ppa# we use 3.8 here, but you can get a higher version by changing the version suffix of the packages belowsudoaptinstallpython3.8python3.8-venvpython3.8-dev# install additional dependenciessudoaptinstallbuild-essentiallibxslt1-devlibxml2-devzlib1g-devgitlibffi-devNote that DP+ still works with CKAN<=2.8, which uses older versions of python.Get the code.mkdirsrccdsrc
gitclone--branch0.11.0https://github.com/datHere/datapusher-pluscddatapusher-plusInstall the dependencies.pipinstallwheel
pipinstall-rrequirements-dev.txt
pipinstall-e.Installqsv.Download the appropriate precompiled binariesfor your platform and copy
it to the appropriate directory, e.g. for Linux:wgethttps://github.com/jqnatividad/qsv/releases/download/0.108.0/qsv-0.108.0-x86_64-unknown-linux-gnu.zip
unzipqsv-0.108.0-x86_64-unknown-linux-gnu.zip
rmqsv-0.108.0-x86_64-unknown-linux-gnu.zip
sudomvqsv*/usr/local/binAlternatively, if you want to install qsv from source, follow
the instructionshere. Note that when compiling from source,
you may want to look into thePerformance Tuningsection to squeeze even more performance from qsv.Also, if you get glibc errors when starting qsv, your Linux distro may not have the required version of the GNU C Library
(This will be the case when running Ubuntu 18.04 or older).
If so, use theqsvdp_glibc-2.31binary as its linked to an older version of glibc. If that still fails, the use theunknown-linux-musl.ziparchive as it is statically linked with the MUSL C Library.If you already have qsv, update it to the latest release by using the --update option.qsvdp --updateℹ️NOTE:qsv is a general purpose CSV data-wrangling toolkit that gets regular updates. To update to the latest version, just run
qsv with the--updateoption and it will check for the latest version and update as required.Configure the Datapusher+ database.Make sure to create thedatapusherPostgreSQL user and thedatapusher_jobsdatabase
(seeDataPusher+ Database Setup).Copy thedatapusher/dot-env.templatetodatapusher/.envandmodify your configuration.cd/usr/lib/ckan/dpplus_env/src/datapusher-plus/datapusher
cpdot-env.template.env# configure your installation as requirednano.envRun Datapusher+ in thedpplus_venvvirtual environment.pythonmain.pyconfig.pyBy default, DP+ should be running at the following port:http://localhost:8800/Production DeploymentThere are two ways to deploy Datapusher+:Manual DeploymentThese instructions set up the DataPusher web service onuWSGIrunning on port 8800, but
can be easily adapted to other WSGI servers like Gunicorn. You'll probably need
to set up Nginx as a reverse proxy in front of it and something like Supervisor
to keep the process up.# Install requirements for DataPusher+. Be sure to have at least Python 3.8sudoaptinstallpython3-virtualenvpython3-devpython3-pippython3-wheelbuild-essentiallibxslt1-devlibxml2-devzlib1g-devgitlibffi-devlibpq-devfile# Install qsv, if requiredwgethttps://github.com/jqnatividad/qsv/releases/download/0.108.0/qsv-0.108.0-x86_64-unknown-linux-gnu.zip-P/tmp
unzip/tmp/qsv-0.108.0-x86_64-unknown-linux-gnu.zip-d/tmp
rm/tmp/qsv-0.108.0-x86_64-unknown-linux-gnu.zip
sudomv/tmp/qsv*/usr/local/bin# if qsv is already installed, be sure to update it to the latest releasesudoqsvdp--update# if you get a glibc error when running `qsvdp --update`# you're on an old distro (e.g. Ubuntu 18.04) without the required version of the glibc libraries.# If so, try running the qsvdp_glibc-2.31 binary instead. If it runs, you can use it instead of the default qsvdp binary.# If that still doesnt work, use the statically linked MUSL version instead# https://github.com/jqnatividad/qsv/releases/download/0.108.0/qsv-0.108.0-x86_64-unknown-linux-musl.zip# find out the locale settingslocale# ONLY IF LANG is not "en_US.UTF-8", set localeexportLC_ALL="en_US.UTF-8"exportLC_CTYPE="en_US.UTF-8"sudodpkg-reconfigurelocales# Create a virtualenv for DataPusher+. DP+ requires at least python 3.8.sudopython3.8-mvenv/usr/lib/ckan/dpplus_venv
sudochown-R$(whoami)dpplus_venv# install datapusher-plus in the virtual environment./usr/lib/ckan/dpplus_venv/bin/activate
pipinstallwheel
pipinstalldatapusher-plus# create an .env file and tune DP+ settings. Tune the uwsgi.ini file as wellsudomkdir-p/etc/ckan/datapusher-plus
sudocurlhttps://raw.githubusercontent.com/dathere/datapusher-plus/master/datapusher/dot-env.template-o/etc/ckan/datapusher-plus/.env
sudocurlhttps://raw.githubusercontent.com/dathere/datapusher-plus/master/deployment/datapusher-uwsgi.ini-o/etc/ckan/datapusher-plus/uwsgi.ini# Be sure to initialize the database if required. (See Database Setup section below)# Be sure to edit the .env file and set the right database connect strings!# Create a user to run the web service (if necessary)sudoaddgroupwww-data
sudoadduser-Gwww-datawww-dataAt this point you can run DataPusher+ with the following command:/usr/lib/ckan/dpplus_venv/bin/uwsgi--enable-threads-i/etc/ckan/datapusher-plus/uwsgi.iniYou might need to change theuidandguidin theuwsgi.inifile when using a different user.To deploy it using supervisor:sudocurlhttps://raw.githubusercontent.com/dathere/datapusher-plus/master/deployment/datapusher-uwsgi.conf-o/etc/supervisor/conf.d/datapusher-uwsgi.conf
sudoservicesupervisorrestartDockerized DeploymentAs Datapusher+ is quite involved as evinced by the above procedure, a containerized installation
will make it far easier not only to deploy DP+ to production, but also to experiment with.Instructions to set up the DP+ Docker instance can be foundhere.The DP+ Docker will also expose additional features and administrative interface to manage
not only Datapusher+ jobs, but also to manage the CKAN Datastore.ConfiguringCKAN ConfigurationAdddatapusherto the plugins in your CKAN configuration file
(generally located at/etc/ckan/default/ckan.ini):ckan.plugins=<other plugins> datapusherIn order to tell CKAN where this webservice is located, the following must be
added to the[app:main]section of your CKAN configuration file :ckan.datapusher.url=http://127.0.0.1:8800/There are other CKAN configuration options that allow to customize the CKAN - DataPusher
integration. Please refer to theDataPusher Settingssection in the CKAN documentation for more details.ℹ️NOTE:DP+ recognizes some additional TSV and spreadsheet subformats -xlsmandxlsbfor Excel Spreadsheets,
andtabfor TSV files. To process these subformats, setckan.datapusher.formatsas follows in your CKAN.INI file:ckan.datapusher.formats=csv xls xlsx xlsm xlsb tsv tab application/csv application/vnd.ms-excel application/vnd.openxmlformats-officedocument.spreadsheetml.sheet ods application/vnd.oasis.opendocument.spreadsheetand add this entry to your CKAN'sresource_formats.jsonfile.["TAB","Tab Separated Values File","text/tab-separated-values",[]],DataPusher+ ConfigurationThe DataPusher+ instance is configured in the.envfile located in the working directory of DP+
(/etc/ckan/datapusher-pluswhen running a production deployment. Thedatapusher-plus/datapushersource directory when running a development installation.)Seedot-env.templatefor a summary of configuration options available.DataPusher+ Database SetupDP+ requires a dedicated PostgreSQL account nameddatapusherto connect to the CKAN Datastore.To create thedatapusheruser and give it the required privileges to thedatastore_defaultdatabase:su-postgres
psql-ddatastore_default
CREATEROLEdatapusherLOGINPASSWORD'YOURPASSWORD';GRANTCREATE,CONNECT,TEMPORARY,SUPERUSERONDATABASEdatastore_defaultTOdatapusher;GRANTSELECT,INSERT,UPDATE,DELETE,TRUNCATEONALLTABLESINSCHEMApublicTOdatapusher;\qDP+ also requires its own job_store database to keep track of all the DP+ jobs. In the original Datapusher,
this was a sqlite database by default. Though DP+ can still use a sqlite database, we are discouraging its use.To setup thedatapusher_jobsdatabase and its user:sudo-upostgrescreateuser-S-D-R-Pdatapusher_jobs
sudo-upostgrescreatedb-Odatapusher_jobsdatapusher_jobs-Eutf-8UsageAny file that has one of the supported formats (defined inckan.datapusher.formats) will be attempted to be loaded
into the DataStore.You can also manually trigger resources to be resubmitted. When editing a resource in CKAN (clicking the "Manage" button on a resource page), a new tab named "DataStore" will appear. This will contain a log of the last attempted upload and a button to retry the upload. Once a resource has been "pushed" into the Datastore, a "Data Dictionary" tab will also be available where the data pusblisher can fine-tune the inferred data dictionary.Command lineRun the following command to submit all resources to datapusher, although it will skip files whose hash of the data file has not changed:ckan -c /etc/ckan/default/ckan.ini datapusher resubmitOn CKAN<=2.8:paster --plugin=ckan datapusher resubmit -c /etc/ckan/default/ckan.iniTo Resubmit a specific resource, whether or not the hash of the data file has changed::ckan -c /etc/ckan/default/ckan.ini datapusher submit {dataset_id}On CKAN<=2.8:paster --plugin=ckan datapusher submit <pkgname> -c /etc/ckan/default/ckan.iniTestingTo test Datapusher-plus, you can use the following test script available on GitHub:test script.Uninstalling Datapusher+Should you need to remove Datapusher+, and you followed either the Development or Production Installation procedures above:# if you're running inside the dpplus_venv virtual environment, deactivate it firstdeactivate# remove the DP+ python virtual environmentsudorm-rf/usr/lib/ckan/dpplus_venv# remove the supervisor DP+ configurationsudorm-rf/etc/supervisor/conf.d/datapusher-uwsgi.conf# remove the DP+ production deployment directorysudorm-rf/etc/ckan/datapusher-plus# remove qsv binary variantssudorm/usr/local/bin/qsv/usr/local/bin/qsvdp/usr/local/bin/qsvlite/usr/local/bin/qsv_nightly/usr/local/bin/qsvdp_nightly/usr/local/bin/qsvlite_nightly# restart the supervisor, without the Datapusher+ servicesudoservicesupervisorreload# ========= DATABASE objects ============# OPTIONAL: backup the datapusher_jobs database first if# you want to retain the DP+ job historysudo-upostgrespg_dump--format=custom-ddatapusher_jobs>datapusher_jobs.dump# to remove the Datapusher+ job database and the datapusher_jobs user/rolesudo-upostgresdropdbdatapusher_jobs
sudo-upostgresdropuserdatapusher_jobs# to drop the datapusher user which DP+ uses to write to the CKAN Datastoresudo-upostgresdropuserdatapusherTo ensure the Datapusher+ service is not automatically invoked when tabular resources are uploaded, removedatapusherfromckan.pluginsin yourckan.inifile.Also remove/comment out the followingckan.datapusherentries in yourckan.ini:ckan.datapusher.formatsckan.datapusher.urlckan.datapusher.callback_url_baseckan.datapusher.assume_task_stale_afterNote that resources which has been pushed previously will still be available on the CKAN Datastore.
You will have to delete these resources separately using the UI or the CKANresource_deleteAPI.If you're no longer using the CKAN Datastore:Edit yourckan.iniand remove/commentdatastorefromckan.plugins.Remove/comment out theckan.datastore.write_urlandckan.datastore.read_urlentries.To confirm the uninstallation is successful, upload a new tabular resource and check if:tabular Resource Views (e.g. datatables_view, recline_view, etc.) are no longer availabletheDatastoreandData Dictionarytabs are no longer availabletheDownloadbutton on the resource page will no longer offer alternate download formats (CSV, TSV, JSON, XML)theDatastore APIbutton will no longer display on tabular resourcesLicenseThis material is copyright (c) 2020 Open Knowledge Foundation and other contributorsIt is open and licensed under the GNU Affero General Public License (AGPL) v3.0
whose full text may be found at:http://www.fsf.org/licensing/licenses/agpl-3.0.html