Package: arbalest
Arbalestis a Python data pipeline orchestration library forAmazon S3andAmazon Redshift.
It takes care of the heavy lifting of making data queryable at scale in AWS.It takes care of:Ingesting data into Amazon RedshiftSchema creation and validationCreating highly available and scalable data import strategiesGenerating and uploading prerequisite artifacts for importRunning data import jobsOrchestrating idempotent and fault tolerant multi-step ETL pipelines with SQLWhy Arbalest?Lightweight library over heavyweight frameworks that can be composed with existing data toolsPython is ade factolinguafrancafor data scienceConfiguration as codeBatteries included, for example, strategies for ingesting time series or sparse data (arbalest.pipeline), or integration with an existing pipeline topology (arbalest.contrib)Use CasesArbalest is not a MapReduce framework, but rather designed to make Amazon Redshift (and all its strengths) easy to use
with typical data workflows and tools. Here are a few examples:You are already using aMapReduceframeworkto process data in S3. Arbalest could make the results of anElastic MapReducejob queryable with SQL in Redshift. You can then hand off to Arbalest to define additional ETL in plain old SQL.You treat S3 as a catch all data sink, perhaps persisting JSON messages or events from a message system likeKafkaor RabbitMQ. Arbalest can expose some or all of this data into a data warehouse using Redshift. The ecosystem of SQL is now available for dashboards, reports, ad-hoc analysis.You have complex pipelines that could benefit from a fast, SQL queryable data sink. Arbalest has support out of the box (arbalest.contrib) to integrate with tools likeLuigito be part of a multi-dependency, multi-step pipeline topology.Getting StartedGetting started is easy withpip:pip install arbalestExamples of Arbalest pipeline are inexamples/. An overview of concepts and classes are below.NoteArbalest depends on psycopg2. However, installing psycopg2 on Windows may not be straight forward.To install psycopg2 on Windows:64 bit Python installation:pip install-egit+https://github.com/nwcell/psycopg2-windows.git@win64-py27#egg=psycopg232 bit Python installation:pip install-egit+https://github.com/nwcell/psycopg2-windows.git@win32-py27#egg=psycopg2PipelinesArbalest orchestrates data loading using pipelines. EachPipelinecan have one or many steps that are made up of three parts:metadata: Path in an S3 bucket to store information needed for the copy process.source: Path in an S3 bucket where data to be copied from is located consisting of JSON object files:{ "id":"66bc8153-d6d9-4351-bada-803330f22db7","someNumber": 1 }schema: Definition of JSON objects to map into Redshift rows.SchemasA schema is defined using aJsonObjectmapper which consists of one or manyPropertydeclarations.
By default the name of the JSON property is used as the column, but can be set
to a custom column name. Column names have amaximum length of 127 characters. Column names
longer than 127 characters will be truncated.
Nested properties will create a default column name delimited by an underscore.Example JSON Object (whitespace for clarity):{
  "id": "66bc8153-d6d9-4351-bada-803330f22db7",
  "someNumber": 1,
  "child" : {
    "someBoolean": true
  }
}Example Schema:JsonObject('destination_table_name',Property('id','VARCHAR(36)'),Property('someNumber','INTEGER','custom_column_name'),Property('child',Property('someBoolean','BOOLEAN')))Copy StrategiesTheS3CopyPipelinesupports different strategies for copying data from S3 to Redshift.Bulk copyBulk copy imports all keys in an S3 path into a Redshift table using a staging table.
By dropping and reimporting all data, duplication is eliminated.
This type of copy is useful for data that does not change very often or will
only be ingested once (e.g. immutable time series).Manifest copyA manifest copy imports all keys in an S3 path into a Redshift table using amanifest.
In addition, a journal of successfully imported objects is persisted to themetadatapath.
Subsequent runs of this copy step will only copy S3 keys that do not exist in the journal.
This type of copy is useful for data in a path that changes often.Example data copies:#!/usr/bin/env pythonimportpsycopg2fromarbalest.configurationimportenvfromarbalest.redshiftimportS3CopyPipelinefromarbalest.redshift.schemaimportJsonObject,Propertyif__name__=='__main__':pipeline=S3CopyPipeline(aws_access_key_id=env('AWS_ACCESS_KEY_ID'),aws_secret_access_key=env('AWS_SECRET_ACCESS_KEY'),bucket=env('BUCKET_NAME'),db_connection=psycopg2.connect(env('REDSHIFT_CONNECTION')))pipeline.bulk_copy(metadata='path_to_save_pipeline_metadata',source='path_of_source_data',schema=JsonObject('destination_table_name',Property('id','VARCHAR(36)'),Property('someNumber','INTEGER','custom_column_name')))pipeline.manifest_copy(metadata='path_to_save_pipeline_metadata',source='path_of_incremental_source_data',schema=JsonObject('incremental_destination_table_name',Property('id','VARCHAR(36)'),Property('someNumber','INTEGER','custom_column_name')))pipeline.run()SQLPipelines can also have arbitrary SQL steps.
Each SQL step can have one or many statements which are executed in a transaction, for example, orchestrating additional ETL (extract, transform, and load).
Expanding on the previous example:#!/usr/bin/env pythonimportpsycopg2fromarbalest.configurationimportenvfromarbalest.redshiftimportS3CopyPipelinefromarbalest.redshift.schemaimportJsonObject,Propertyif__name__=='__main__':pipeline=S3CopyPipeline(aws_access_key_id=env('AWS_ACCESS_KEY_ID'),aws_secret_access_key=env('AWS_SECRET_ACCESS_KEY'),bucket=env('BUCKET_NAME'),db_connection=psycopg2.connect(env('REDSHIFT_CONNECTION')))pipeline.bulk_copy(metadata='path_to_save_pipeline_metadata',source='path_of_source_data',schema=JsonObject('destination_table_name',Property('id','VARCHAR(36)'),Property('someNumber','INTEGER','custom_column_name')))pipeline.manifest_copy(metadata='path_to_save_pipeline_metadata',source='path_of_incremental_source_data',schema=JsonObject('incremental_destination_table_name',Property('id','VARCHAR(36)'),Property('someNumber','INTEGER','custom_column_name')))pipeline.sql(('SELECT someNumber +%s''INTO some_olap_table FROM destination_table_name',1),('SELECT * INTO destination_table_name_copy ''FROM destination_table_name'))pipeline.run()Orchestration HelpersIncluded in this project are a variety of orchestration helpers to assist with
the creation of pipelines.
These classes are defined in thearbalest.pipelineandarbalest.contribmodules.Sorted Data SourcesAssuming source data is stored in a sortable series of directories,S3SortedDataSourcesfacilitates the retrieval of S3 paths in a sequence for import, given a start
and/or end. In addition, it has methods to mark a cursor in an S3 persisted journal.Examples of data stored as a sorted seriesSequential integers:s3://bucket/child/1/*
s3://bucket/child/2/*
s3://bucket/child/3/*Time series:s3://bucket/child/2015-01-01/*
s3://bucket/child/2015-01-02/*
s3://bucket/child/2015-01-03/*
s3://bucket/child/2015-01-04/00/*Example of sorted data source classS3SortedDataSources(metadata='',source='child',bucket=bucket,start=env('START'),end=env('END'))Time SeriesSqlTimeSeriesImportimplements a bulk copy and update strategy of data from
a list of time series sources fromS3SortedDataSourcesinto an existing
target table.Example time series import from an S3 time series topology, ingesting a day of objectsTime series path topology:s3://bucket/child/2015-01-01/*
s3://bucket/child/2015-01-02/*ExamplePipeline(S3CopyPipeline):def__init__(self,aws_access_key_id,aws_secret_access_key,bucket,db_connection):super(ExamplePipeline,self).__init__(aws_access_key_id,aws_secret_access_key,bucket,db_connection)# Create table to ingest data into if it does not existself.sql('CREATE target_table IF NOT EXISTS target_table(id VARCHAR(36), someNumber INTEGER, timestamp TIMESTAMP);')time_series=SqlTimeSeriesImport(destination_table='target_table',update_date='2015-01-01',# Replace existing events, if any, after this timestampsources=S3SortedDataSources(metadata='',source='child',bucket=bucket,start='2015-01-01',end='2015-01-02'),Property('id','VARCHAR(36)'),Property('someNumber','INTEGER'),Property('timestamp','TIMESTAMP'))# Populate target_table using a bulk copy per daytime_series.bulk_copy(pipeline=self,metadata='',max_error=1000,# Maximum errors tolerated by Redshift COPYorder_by_column='timestamp')# Use column named timestamp to sort by and replace existing events, if anyLuigiPipelineTaskwraps anyarbalest.core.Pipelineinto aLuigi Task.
This allows for the composition of workflows with dependency graphs, for example,
data pipelines that are dependent on multiple steps or other pipelines. Luigi then takes care of
the heavy lifting ofscheduling and executingmultistep pipelines.LicenseArbalest is licensed under theMIT License.Authors and ContributorsArbalest was built at Dwolla, primarily byFredrick Galoso.
Initial support for Luigi and contributions to orchestration helpers byHayden Goldstien.
We gladly welcome contributions and feedback. If you are using Arbalest we would love to know.