Package: beam-nuggets
AboutA collection of random transforms for theApache beampython SDK . Many are
simple transforms. The most useful ones are those for
reading/writing from/to relational databases.InstallationUsing pippipinstallbeam-nuggetsFrom sourcegitclonegit@github.com:mohaseeb/beam-nuggets.gitcdbeam-nuggets
pipinstall.Supported transformsIOrelational_db.ReadFromDBfor reading from relational database tables.relational_db.Writefor writing to relational database tables.Above transforms usesSqlAlchemyto communicate with the database,
and hence they can read from and write to all relational databases supported
by SqlAlchemy.
The transformsare testedagainst PostgreSQL, MySQL and SQLite.kafkaio.KafkaProducefor writing to Kafka topics.kafkaio.KafkaConsumefor consuming from kafka topics.csvio.Readfor reading CSV files.OthersSelectFromNestedDictSelects a subset from records formed of nested dictionaries.ParseJsonAssignUniqueIdDocumentationSeehere.UsageWrite data to an SQLite table using beam-nugget'srelational_db.Writetransform.# write_sqlite.py contentsimportapache_beamasbeamfromapache_beam.options.pipeline_optionsimportPipelineOptionsfrombeam_nuggets.ioimportrelational_dbrecords=[{'name':'Jan','num':1},{'name':'Feb','num':2}]source_config=relational_db.SourceConfiguration(drivername='sqlite',database='/tmp/months_db.sqlite',create_if_missing=True# create the database if not there)table_config=relational_db.TableConfiguration(name='months',create_if_missing=True,# automatically create the table if not thereprimary_key_columns=['num']# and use 'num' column as primary key)withbeam.Pipeline(options=PipelineOptions())asp:# Will use local runnermonths=p|"Reading month records">>beam.Create(records)months|'Writing to DB'>>relational_db.Write(source_config=source_config,table_config=table_config)Execute the pipelinepythonwrite_sqlite.pyExamine the contentssqlite3/tmp/months_db.sqlite'select * from months'# output:# 1.0|Jan# 2.0|FebTo write the same data to a PostgreSQL table instead, just create a suitablerelational_db.SourceConfigurationas follows.source_config=relational_db.SourceConfiguration(drivername='postgresql+pg8000',host='localhost',port=5432,username='postgres',password='password',database='calendar',create_if_missing=True# create the database if not there)Clickherefor more examples, including writing to PostgreSQL in Google Cloud Platform
using the DataFlowRunner.An example showing how you can use beam-nugget'srelational_db.ReadFromDBtransform to read from a PostgreSQL database table.from__future__importprint_functionimportapache_beamasbeamfromapache_beam.options.pipeline_optionsimportPipelineOptionsfrombeam_nuggets.ioimportrelational_dbwithbeam.Pipeline(options=PipelineOptions())asp:source_config=relational_db.SourceConfiguration(drivername='postgresql+pg8000',host='localhost',port=5432,username='postgres',password='password',database='calendar',)records=p|"Reading records from db">>relational_db.ReadFromDB(source_config=source_config,table_name='months',query='select num, name from months'# optional. When omitted, all table records are returned.)records|'Writing to stdout'>>beam.Map(print)Seeherefor more examples.DevelopmentInstallgitclonegit@github.com:mohaseeb/beam-nuggets.gitcdbeam-nuggetsexportBEAM_NUGGETS_ROOT=`pwd`pipinstall-e.[dev]Make changes on dedicated dev branchesRun testscd$BEAM_NUGGETS_ROOTpython-munittestdiscover-vGenerate docscd$BEAM_NUGGETS_ROOTdocs/generate_docs.shCreate a PR against master.After merging the accepted PR and updating the local master, upload a new
build to pypi.cd$BEAM_NUGGETS_ROOTscripts/build_test_deploy.shBacklogversioned docs?Summarize the investigation of using Source/Sink Vs ParDo(and GroupBy) for IOmore nuggets: WriteToCsvInvestigate readiness of SDF ParDo, and possibility to use for relational_db.ReadFromDBintegration testsDB transforms failures handling on IO transformsmore nuggets: Elasticsearch, MongoWriteToRelationalDB, loggingContributions bymohaseeb,astrocox,2514millerj,alfredo,shivangkumarLicenceMIT