Package: async-kinesis-client
async-kinesis-clientPython Kinesis Client library utilising asyncioBased on Kinesis-Python project by Evan Borgstromeborgstrom@nerdwallet.comhttps://github.com/NerdWalletOSS/kinesis-pythonbut with asyncio magicThe problem with Kinesis-Python is that all the data ends up in a single thread
and being checkpointed from there - so despite having many processes, the client
is clogged by checkpointing. Besides, it checkpoints every single record and this is
not configurable.This client is based on aioboto3 library and uses Python 3.6+ async methods.Usage:importasynciofromasync_kinesis_client.kinesis_consumerimportAsyncKinesisConsumerasyncdefread_stream():# This is a coroutine that reads all the records from a shardasyncdefread_records(shard_reader):asyncforrecordsinshard_reader.get_records():forrinrecords:print('Shard:{}; Record:{}'.format(shard_reader.shard_id,r))consumer=AsyncKinesisConsumer(stream_name='my-stream',checkpoint_table='my-checkpoint-table')# consumer will yield existing shards and will continue yielding# new shards if re-sharding happensasyncforshard_readerinconsumer.get_shard_readers():print('Got shard reader for shard id:{}'.format(shard_reader.shard_id))asyncio.ensure_future(read_records(shard_reader))asyncio.get_event_loop().run_until_complete(read_stream())AsyncShardReaderandAsyncKinesisConsumercan be stopped from parallel coroutine by callingstop()method,
consumer will stop all shard readers in that case.
If you want to be notified of shard closing, catchShardClosedExceptionwhile reading recordsAsyncShardReaderexposes property millis_behind_latest which could be useful for determining application performance.AsyncKinesisConsumerhas following configuration methods:set_checkpoint_interval(records)- how many records to skip before checkpointingset_lock_duration(time)- how many seconds to hold the lock. Consumer would attempt to refresh the lock before that timeset_reader_sleep_time(time)- how long should shard reader wait (in seconds, fractions possible) if it did not receive any records from Kinesis streamset_checkpoint_callback(coro)- set callback coroutine to be called before checkpointing next batch of records. Coroutine arguments:ShardId,SequenceNumberProducer is rather trivial:fromasync_kinesis_client.kinesis_producerimportAsyncKinesisProducer# ...asyncdefwrite_stream():producer=AsyncKinesisProducer(stream_name='my-stream',ordered=True)awaitproducer.put_record(record=b'bytes',partition_key='string',# optional, if none, default time-based key is usedexplicit_hash_key='string'# optional)Sending multiple records at once:fromasync_kinesis_client.kinesis_producerimportAsyncKinesisProducer# ...asyncdefwrite_stream():producer=AsyncKinesisProducer(stream_name='my-stream',ordered=True)records=[{'Data':b'bytes','PartitionKey':'string',# optional, if none, default time-based key is used'ExplicitHashKey':'string'# optional},...]response=awaitproducer.put_records(records=records)# See boto3 docs for response structure:# https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/kinesis.html#Kinesis.Client.put_recordsAWS authentication. For testing outside AWS cloud, especially when Mutil-Factor Authentication is in use I find following snippet extremely useful:importosimportaioboto3frombotocoreimportcredentialsfromaiobotocoreimportAioSessionworking_dir=os.path.join(os.path.expanduser('~'),'.aws/cli/cache')session=AioSession(profile=os.environ.get('AWS_PROFILE'))provider=session.get_component('credential_provider').get_provider('assume-role')provider.cache=credentials.JSONFileCache(working_dir)aioboto3.setup_default_session(botocore_session=session)This allows re-using cached session token after completing any aws command underawsudo, all you need is to set AWS_PROFILE environment variable.Currently library still not tested enough for different network events.
Use it on your own risk, you've been warned.