Package: dbsink
dbsinkRead from a kafka topic and sink to a database table, one row per message.This is not unlike the Kafka Connect JdbcConnector. This project has a much lower bar of entry and doesn't require diving into the Kafka Connect ecosystem. I wrote the equivilent to this project using a custom JdbcConnector and it was getting out of control and was basically un-testable. So here we are.You can choose to unpack the data asavro,msgpackor the defaultjson.avrorequires an additionalregistryparameter.Docker images:https://hub.docker.com/r/axiom/dbsink/buildsWHY?I needed to read from well-defined kafka topics and store the results in a database table so collaborators could interact with the data in a more familiar way.It is also a very convienent and easy to setup PostgREST on top of the resulting tables to get a quick read-only REST API on top of the tabled messages.Mapping messages to tablesYou can define custom mappings between messages and tables using a python class. You may register your custom mappings with thedbsink.mapsentrypoint to have them available todbsinkat run-time.entry_points={'dbsink.maps':['YourCustomMap    = you.custom.map.module:CustomMapClass',# ...]}Custom mapping classes should inherit from theBaseMapclass indbsinkand override the following functions as needed:upsert_constraint_name- Name of the constraint to use for upserting data. Set to toNoneto disable upserting. Use this class property when creating the upsert constraint on your table (see example below).unique_index_name- Unique index name based on the table name. Use this if defining a single unique index on your table.sequence_name- Unique sequence name based on the table name. Use this if defining a single sequence column on your table._check_key- Checks for validity of a message'skeybefore trying to sink. ReturnTrueif valid and raise an error if not._check_value- Checks for validity of a message'svaluebefore trying to sink. ReturnTrueif valid and raise an error if not.schema- A list of SQLAlchmyColumn,Index, andConstraintschema definitions to use in table creation and updating. This fully describes your table's schema.message_to_values- A function acceptingkeyandvaluearguments and returning a tuplekey, dictwhere the dict is thevaluesto pass to SQLAlchemy'sinsert().valuesmethod. Thevalueargument to this function will already be unpacked ifavroormsgpackpacking was specified.insert(table).values(# dict_returned_ends_up_here)ExampleA simple example is theStringMapmapping included withdbsinkfromdatetimeimportdatetimeimportpytzimportsqlalchemyassqlimportsimplejsonasjsonfromdbsink.mapsimportBaseMapclassStringMap(BaseMap):@propertydefupsert_constraint_name(self):returnNone# Ignore upsertsdef_check_key(self,key):returnTrue# All keys are validdef_check_value(self,value):# Make sure values are JSON parsable_=json.loads(json.dumps(value,ignore_nan=True))returnTrue@propertydefschema(self):return[sql.Column('id',sql.Integer,sql.Sequence(self.sequence_name),primary_key=True),sql.Column('sinked',sql.DateTime(timezone=True),index=True),sql.Column('key',sql.String,default='',index=True),sql.Column('payload',sql.String)]defmessage_to_values(self,key,value):# Raises if invalid. This calls `._check_key` and `._check_value`self.check(key,value)values={'sinked':datetime.utcnow().replace(tzinfo=pytz.utc).isoformat(),'key':key,'payload':json.dumps(value),}returnkey,valuesAdvanced ExampleThere are no restrictions on table schemas or how you map the message data into the schema. Take this example below that uses aPostGIScolumn.fromdatetimeimportdatetimeimportpytzimportsqlalchemyassqlimportsimplejsonasjsonfromshapely.geometryimportshapefromgeoalchemy2.typesimportGeographyfromdbsink.mapsimportBaseMapclassNamedGenericGeography(BaseMap):def_check_key(self,key):returnTrue# All keys are validdef_check_value(self,value):# Make sure values are JSON parsable_=json.loads(json.dumps(value,ignore_nan=True))returnTrue@propertydefschema(self):return[sql.Column('id',sql.Integer,sql.Sequence(self.sequence_name),primary_key=True),sql.Column('name',sql.String,default='',index=True),sql.Column('time',sql.DateTime(timezone=True),index=True),sql.Column('geom',Geography(srid=4326)),sql.Index(self.unique_index_name,'name','time',unique=True,),sql.UniqueConstraint('name','time',name=self.upsert_constraint_name)]defmessage_to_values(self,key,value):""" Assumes a message format of{"time": 1000000000, # unix epoch"name": "my cool thing","geojson": {"geometry": {"type": "Polygon","coordinates": [ [ [ -118.532116484818843, 32.107425500492766 ], [ -118.457544847012443, 32.107425500492702 ], [ -118.457544847012443, 32.054517056541435 ], [ -118.532116484818872, 32.054517056541464 ], [ -118.532116484818843, 32.107425500492766 ] ] ]}}}"""# Raises if invalidself.check(key,value)# GeoJSON `geometry` attribute to WKTgeometry=shape(value['geojson']['geometry']).wktvalues={'name':value['name']'time':datetime.fromtimestamp(value['time'],pytz.utc).isoformat()'geom':geometry}returnkey,valuesConfigurationThis program usesClickfor the CLI interface. For all options please use thehelp:$dbsink--helpEnvironmental VariablesAll configuration options can be specified with environmental variables using the patternDBSINK_[argument_name]=[value]. For more information seethe click documentation.DBSINK_TOPIC="topic-to-listen-to"\DBSINK_LOOKUP="StringMap"\DBSINK_TABLE="MyCoolTable"\DBSINK_CONSUMER="myconsumer"\DBSINK_PACKING="msgpack"\DBSINK_OFFSET="earlist"\DBSINK_DROP="true"\DBSINK_VERBOSE="1"\dbsinkTestingYou can run the tests usingpytest. To run the integration tests, start a database withdocker run -p 30300:5432 --name dbsink-int-testing-db -e POSTGRES_USER=sink -e POSTGRES_PASSWORD=sink -e POSTGRES_DB=sink -d mdillon/postgis:11and runpytest -m integration