Package: pybatchintory
pybatchintorypybatchintoryrepresents a middleware for batch oriented data pipelines. It enables incremental processing and provides first class support for reprocessing historical data with predictable workloads.Only meta information of data items is consumed and processed. The actual data is not read. Aninventoryof already processed and unseen data items is managed while providing an API to interact with it.ReasoningThis package may greatly improve data pipelines by enabling the following four features:Incremental processing: Only new, unseen data items can be processed, avoiding recomputation of all data items.Backfill scenarios: Historical data items can be reprocessed in a configurable and automated way.Predictable workloads: The amount of data to be processed can be defined and is known upfront to adjust compute resources accordingly for best efficiency and cost/performance ratio for both incremental processing and backfill scenarios.Transparency and observability: Each data item can be enriched with information about when it was processed by what job.Preconditionspybatchintoryassumes the existence of a metadata table that contains information about the data items, such as their file location and registration timestamps. Importantly, to properly generate ranges of data items constituting a single batch to be processed,pybatchintoryrequires the metadata table to provide aunique auto-increment ID column.AssumptionsData items are always processed chronologically according to monotonic increasing unique idOnly continuous ranges of ids can be processedExample meta data source tableSchemaFieldTypeKeyExtraidint(11)PRIauto_incrementfile_locationvarchar(255)size_in_bytesint(11)importedtimestampContentidfile_locationsize_in_bytesimported6/path/to/sixth/file327682021-01-06 00:00:007/path/to/seventh/file655362021-01-07 00:00:008/path/to/eighth/file1310722021-01-08 00:00:009/path/to/ninth/file2621442021-01-09 00:00:0010/path/to/tenth/file5242882021-01-10 00:00:00UsageConfigurationSqlAlchemy connection strings for inventory and meta table need to be
provided. These can be provided via environment variables as follows:Environment variablesexportPYBATCHINTORY_INVENTORY_CONN="postgresql+psycopg2://user:password@host:port/dbname"exportPYBATCHINTORY_META_CONN="postgresql+psycopg2://user:password@host:port/dbname"Dot-env fileYou may also provide a path to a dot-env file via an environment variable:exportPYBATCHINTORY_ENV_FILE="PATH_TO_DOT_ENV_FILE"ProgrammaticallyIn addition, you may set a dot-env file and explicit settings programmatically:frompybatchintoryimportconfigureconfigure(dot_env="PATH_TO_DOT_ENV_FILE",settings=dict(INVENTORY_CONN="CONN_STRING"))InvocationIncremental with predictable workloadfrompybatchintoryimportacquire_batchbatch=acquire_batch(meta_table_name="meta_table",meta_table_cols={"uid":"id","item":"file","weight":"size_in_mib"},job="incremental_job",batch_weight=100)process_func(batch.items)batch.succeeded()Backfill with predictable workloadfrompybatchintoryimportacquire_batchbatch=acquire_batch(meta_table_name="meta_table",meta_table_cols={"uid":"id","item":"file","weight":"size_in_mib"},job="backfill_job",batch_id_min=20,batch_id_max=80,batch_weight=100)process_func(batch.items)batch.succeeded()Multiple batchesNot yet implementedfrompybatchintoryimportacquire_batchesbatches=acquire_batches(meta_table_name="meta_table",job="incremental_job",batch_weight=10,iterations=5)forbatchinbatches:process_func(batch.items)batch.succeeded()Error handlingfrompybatchintoryimportacquire_batch# version 1 - manual error handlingbatch=acquire_batch(meta_table_name="meta_table",job="incremental_job",batch_weight=10)try:process_func(batch.items)batch.succeeded()exceptExceptionase:batch.failed(e)raise# version 2 - automatic error handling - not yet implementedbatch=acquire_batch(meta_table_name="meta_table",job="incremental_job",batch_weight=10)batch.process(func,args,kwargs)Requirements (non ordered)Allow concurrent batch generation/processing for the same job identifierread/update transactionsfor secure and reliable functioning, isolation level "serializable" is required however "read committed" should be sufficent in most casesSeparate database connections for meta data (external) and inventory tables (backend)use sqlalchemy for engine creation and query building (guard against sql injections)Credentials to be read via environment variables or dotenv files (pydantic)Allow recursive processing of inventory table itself to allow incremental/backfill on already incremental/backfilled jobsProvide filter condition for meta data source table to select only relevant subset of data itemsInventory tableColumn NameTypeConstraintsidIntegerprimary_key=True, autoincrement=Truemeta_tableStringnullable=FalsejobStringnullable=Falsejob_identifierStringjob_result_itemJSONprocessing_startDateTimenullable=False, default=TSprocessing_endDateTimebatch_id_startIntegernullable=Falsebatch_id_endIntegernullable=Falsebatch_weightIntegerbatch_countIntegernullable=FalseattemptIntegernullable=False, default=1statusEnum(*cfg.settings.INVENTORY_STATUS_ENUMS)nullable=False, default="running"loggingStringconfigJSONnullable=FalsePoem (thanks to chatGPT)This Python package is a wondrous tool,For data processing, it's oh so cool,With middleware for batches of data,Incremental processing, a much-needed beta.No more recomputation of all data,Only the unseen items, a true data saver,Backfill scenarios are now a breeze,With historical data, we can reprocess with ease.Predictable workloads, we can define,Adjusting resources, a task that's benign,For cost and performance, it's the way to go,Efficient processing, with no overflow.Transparency and observability in tow,Each data item enriched, as we know,When and how it was processed, we see,A Python package that's just meant to be.