Package: dataflows
DataFlowsDataFlows is a simple and intuitive way of building data processing flows.It's built for small-to-medium-data processing - data that fits on your hard drive, but is too big to load in Excel or as-is into Python, and not big enough to require spinning up a Hadoop cluster...It's built upon the foundation of the Frictionless Data project - which means that all data produced by these flows is easily reusable by others.It's a pattern not a heavy-weight framework: if you already have a bunch of download and extract scripts this will be a natural fitRead more in theFeatures section below.QuickStartInstalldataflowsviapip install.(If you are using minimal UNIX OS, run firstsudo apt install build-essential)Then use the command-line interface to bootstrap a basic processing script for any remote data file:# Install from PyPi$pipinstalldataflows# Inspect a remote CSV file$dataflowsinithttps://raw.githubusercontent.com/datahq/dataflows/master/data/academy.csv
Writingprocessingcodeintoacademy_csv.py
Runningacademy_csv.py
academy:#     Year           Ceremony  Award                                 Winner  Name                            Film(string)(integer)(string)(string)(string)(string)--------------------------------------------------------------------------------------------------------------------11927/19281ActorRichardBarthelmessTheNoose21927/19281Actor1EmilJanningsTheLastCommand31927/19281ActressLouiseDresserAShipComesIn41927/19281Actress1JanetGaynor7thHeaven51927/19281ActressGloriaSwansonSadieThompson61927/19281ArtDirectionRochusGlieseSunrise71927/19281ArtDirection1WilliamCameronMenziesTheDove;Tempest
...# dataflows create a local package of the data and a reusable processing script which you can tinker with$tree
.
├──academy_csv
│├──academy.csv
│└──datapackage.json
└──academy_csv.py1directory,3files# Resulting 'Data Package' is super easy to use in Python[adam]~/code/budgetkey-apps/budgetkey-app-main-page/tmp(master=)$python
Python3.6.1(default,Mar272017,00:25:54)[GCC4.2.1CompatibleAppleLLVM8.0.0(clang-800.0.42.1)]ondarwin
Type"help","copyright","credits"or"license"formoreinformation.
>>>fromdatapackageimportPackage
>>>pkg=Package('academy_csv/datapackage.json')>>>it=pkg.resources[0].iter(keyed=True)>>>next(it){'Year':'1927/1928','Ceremony':1,'Award':'Actor','Winner':None,'Name':'Richard Barthelmess','Film':'The Noose'}>>>next(it){'Year':'1927/1928','Ceremony':1,'Award':'Actor','Winner':'1','Name':'Emil Jannings','Film':'The Last Command'}# You now run `academy_csv.py` to repeat the process# And obviously modify it to add data modification stepsFeaturesTrivial to get started and easy to scale upSet up and run from command line in seconds ...dataflows init=>flow.pypython flow.pyValidate input (and esp source) quickly (non-zero length, right structure, etc.)Supports caching data from source and even between stepsso that we can run and test quickly (retrieving is slow)Immediate test is run: and look at output ...Log, debug, rerunDegrades to simple pythonConventions over configurationLog exceptions and / or terminateThe input to each stage is a Data Package or Data Resource (not a previous task)Data package based and compatibleProcessors can be a function (or a class) processing row-by-row, resource-by-resource or a full packageA pre-existing decent contrib library of Readers (Collectors) and Processors and WritersLearn moreDive into theTutorialto get a deeper glimpse into everything thatdataflowscan do.
Also review this list ofBuilt-in Processors, which also includes an API reference for each one of them.