Package: datapunt-processing
Data-processingAt the City of Amsterdam we deal with many different types of structured and unstructered data. Much of the data is not of high quality and are missing proper semantics to do proper analytics with.This repository combines generic command line functions to create extract, transform and load steps we can then use for creating a reproducable data for analytics and usage in dashboards and maps.For more information about the how we use these functions in our workflow, read thedata-pipeline guide.How to useFull documentation can be found here:amsterdam.github.io/data-processingQuickstart:
To use a function in python you can use:from datapunt_processing.extract import download_from_catalog

or

from datapunt_processing.helpers.connections import objectstore_connectionTo use the functions directly from the command line in your virtual environment or docker shell you can use it like this:download_from_data_amsterdam -hTo see the list of command line functions see the modules below or directly insetup.pyGetting StartedTo get the functions up and running:pipinstalldatapunt-processingTo develop the functions locally use these steps:Clone the repository:gitclonehttps://github.com/Amsterdam/data-processing.gitcddata-processingCreate Virtual environment in Windows# Create and activate a virtual environment, for example with:python-mvenv--copies--promptdata-processing.venv.venv\Scripts\activateCreate Virtual environment in OSXvirtualenv--python=$(whichpython3)venvsourcevenv/bin/activateInstall the data-processing modules in editable modepipinstall-e.4. A database is required for the transform and load functions.
You can setup your postgres database credentials in the config.ini file to apply to the functions.If want to useDocker, you can start a database server for your project in a new terminal. The name, port and login of the database can be changed in the docker-compose.yml. Also change them in the config.ini file which will be used by the functions to connect to that database.docker-composeup-ddatabaseNotebooksSome of the examples are in the form of runnable Jupyter notebooks. Copies of these with all the images and output included are hosted at Anaconda Cloud. To run these notebooks on your own system, start up a Jupyter notebook server:To install jupyter:pipinstall-e.\[dev\]jupyternotebook--NotebookApp.iopub_data_rate_limit=100000000How to ContributeIf you want to contribute please follow thecontribute guidelines0. PrequisitesFork this repository to your local github account to add add and test new functions.gitclonehttps://github.com/Amsterdam/data-processing.gitInstall the docs,test,dev packages using this command:pipinstall-e.[docs,test,dev]orwhenusingzshpipinstall-e.\[docs,test,dev\]This package is build by usingsetuptoolsto be able to release stable versions on PyPi. It follows some oftheseguidelines of setting up a python package.1. Add functionWe try to use command line functions as much as possible to ensure we create functions to work easily with different environments and to force yourself creating more generic functions with input variables.If possible, convert your function into apython-package command line scriptusing theboilerplate_function.pyAdd your function to the appropriatefolder:extracttransformloadhelpersside note: not all functions are suitable for CL. Machine learning preprocessing steps or general API calls for instance, (that often require parameters in the form of dicts or lists) as input are not suitable and can be used as stand-alone scripts.2. Add testsAdd test to thetest folderand run:pythonsetup.pytestto test if no other functions are breaking. Correct those issues as well if needed.3. Add documentationCreate a awesome_module.rst file withSphinx Argparse extensionfields to generate the description and argument fields by reusing anexisting rst file. The helpers docs will generate automatically, so you can skip this step if it is placed in the helper function.Add the rst filename to the list inmodules.rstto be found on the main page.Regenerate the documentation to test the docs output using this command line function:sphinx/makedocsandtestifthereadmeisnotbroken:opendocs/index.html4. Add a Pull RequestMake a PR to add the add your awesome function to our processing code to be reused by many other developpers and data analists.