Package: runpandarun
Run Panda Run:panda_face: :panda_face: :panda_face: :panda_face: :panda_face: :panda_face: :panda_face:A simple interface written in python for reproducible i/o workflows around tabular data viapandasDataFramespecified viayaml"playbooks".Apart from anypandasfunction possible that can alter data, alsodatapatchis included for an additional and easier way to patch data.NOTICEAs of july 2023, this package only handles pandas transform logic, no data warehousing anymore.  Seearchived versionQuickstartInstall via pipSpecify your operations viayamlsyntax:read:uri:./data.csvoptions:skiprows:3operations:-handler:DataFrame.renameoptions:columns:value:amount-handler:Series.mapcolumn:slugoptions:func:"lambdax:normality.slugify(x)ifisinstance(x,str)else'NODATA'"store this as a filepandas.yml, and apply a data source:cat data.csv | runpandarun pandas.yml > data_transformed.csvOr, use within your python scripts:fromrunpandarunimportPlaybookplay=Playbook.from_yaml("./pandas.yml")df=play.run()# get the transformed dataframe# change playbook parameters on run time:play.read.uri="s3://my-bucket/data.csv"df=play.run()df.to_excel("./output.xlsx")# the play can be applied directly to a data frame,# this allows more granular controldf=get_my_data_from_somewhere_else()df=play.run(df)InstallationRequires at least python3.10 Virtualenv use recommended.Additional dependencies (pandaset. al.) will be installed automatically:pip install runpandarunAfter this, you should be able to execute in your terminal:runpandarun --helpReferenceThe playbook can be programmatically obtained in different ways:fromrunpandarunimportPlaybook# via yaml fileplay=Playbook.from_yaml('./path/to/config.yml')# via yaml stringplay=Playbook.from_string("""operations:- handler: DataFrame.sort_valuesoptions:by: my_sort_column""")# directly via the Playbook object (which is a pydantic object)play=Playbook(operations=[{"handler":"DataFrane.sort_values","options":{"by":"my_sort_column"}}])All options within the Playbook are optional, if you apply an empty play to a DataFrame, it will just remain untouched (butrunpandarunwon't break)The playbook has three sections:read: instructions for reading in a source dataframeoperations: a list of functions with their options (kwargs) executed in the given orderwrite: instructions for saving a transformed dataframe to a targetRead and writepandascan read and write from many local and remote sources and targets.More information about handlers and their options:Pandas IO toolsFor example, you could transform a source froms3to asftpendpoint:runpandarun pandas.yml -i s3://my_bucket/data.csv -o sftp://user@host/data.csvyou can overwrite theuriarguments in the command line with-i / --in-uriand-o / --out-uriread:uri:s3://my-bucket/data.xls# input uri, anything that pandas can readhandler:read_excel# default: guess by file extension, fallback: read_csvoptions:# options for the handlerskiprows:2write:uri:./data.xlsx# output uri, anything that pandas can write tohandler:write_excel# default: guess by file extension, fallback: write_csvoptions:# options for the handlerindex:falseOperationsTheoperationskey of the yaml spec holds the transformations that should be applied to the data in order.An operation can be any function frompd.DataFrameorpd.Series. Refer to these documentations to see their possible options (as in**kwargs).For the handler, specify the module path without apdorpandasprefix, justDataFrame.<func>orSeries.<func>. When using a function that applies to aSeries, tell :panda_face: which one to use via thecolumnprop.operations:-handler:DataFrame.renameoptions:columns:value:amountThis exactly represents this python call to the processed dataframe:df.rename(columns={"value":"amount"})env varsFor api keys or other secrets, you can put environment variables anywhere into the config. They will simply resolved viaos.path.expandvarsread:options:storage_options:header:"api-key":${MY_API_KEY}ExampleA full playbook example that covers a few of the possible cases.See the yaml files in./tests/fixtures/for more.read:uri:https://api.example.org/data?format=csvoptions:storage_options:header:"api-key":${API_KEY}skipfooter:1operations:-handler:DataFrame.renameoptions:columns:value:amount-handler:Series.str.lowercolumn:state-handler:DataFrame.assignoptions:city_id:"lambdax:x['state']+'-'+x['city'].map(normality.slugify)"-handler:DataFrame.set_indexoptions:keys:-city_id-handler:DataFrame.sort_valuesoptions:by:-state-citypatch:city:options:-match:Zarizrivalue:Zar1zr1write:uri:ftp://user:${FTP_PASSWORD}@host/data.csvoptions:index:falseHow to...Rename columnsDataFrame.renameoperations:-handler:DataFrame.renameoptions:columns:value:amount"Firstname":first_nameApply modification to a columnSeries.mapoperations:-handler:Series.mapcolumn:my_columnoptions:func:"lambdax:x.lower()"Set an indexDataFrame.set_indexoperations:-handler:DataFrame.set_indexoptions:keys:-city_idSort valuesDataFrame.sort_valuesoperations:-sort_values:by:-column1-column2ascending:falseDe-duplicateDataFrame.drop_duplicateswhen using a subset of columns, use in conjunction withsort_valuesto make sure to keep the right recordsoperations:-drop_duplicates:subset:-column1-column2keep:lastCompute a new column based on existing dataDataFrame.assignoperations:-handler:DataFrame.assignoptions:city_id:"lambdax:x['state']+'-'+x['city'].map(normality.slugify)"SQLPandas SQL ioread:uri:postgresql://user:password@host/databaseoptions:sql:"SELECT*FROMmy_tableWHEREcategory='A'"Patch dataApart from anypandasfunction possible that can alter data, alsodatapatchis included for an additional and easier way to patch data.Simply add apatchconfig to the yaml. Refer to the datapatch readme for details.The patching is appliedafterall theoperationsare applied.patch:countries:normalize:truelowercase:trueoptions:-match:Frankreichvalue:France-match:-Northkorea-Nordkorea-Northern Korea-NKorea-DPRKvalue:North Korea-contains:Britainvalue:Great Britainsave evalOk wait, you are executing arbitrary python code in the yaml specs?Not really, there is a strict allow list of possible modules that can be used. Seerunpandarun.util.safe_evalThis includes:any pandas or numpy modulesnormalityfingerprintsSo, this would, of course,NOT WORK(as tested here)operations:-handler:DataFrame.applyfunc:"__import__('os').system('rm-rf/')"developmentPackage is managed viaPoetrygit clone https://github.com/investigativedata/runpandarunInstall requirements:poetry install --with devTest:make testFundingSince July 2023, this project is part ofinvestigraphand development of this project is funded byMedia Tech Lab Bayern batch #3