Package: async-kinesis
async-kinesispip install async-kinesisFeaturesuses queues for both producer and consumerproducer flushes with put_records() if has enough to flush or after "buffer_time" reachedconsumer iterates over msg queue independent of shard readersConfigurable to handle Sharding limits but will throttle/retry if requiredie multiple independent clients are saturating the ShardsCheckpointing with heartbeatsdeadlock + reallocation of shards if checkpoint fails to heartbeat within "session_timeout"processors (aggregator + serializer)json line delimited, msgpackSeedocs/designfor more details.
Seedocs/yetanotheras to why reinvent the wheel.Environment VariablesAs required by boto3AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEYProducerfrom kinesis import Producer

async with Producer(stream_name="test") as producer:
    # Put item onto queue to be flushed via put_records()
    await producer.put({'my': 'data'})Options:(comments in quotes are Kinesis Limits as per AWS Docs)ArgDefaultDescriptionsessionNoneAioSession (to use non default profile etc)region_nameNoneAWS Regionbuffer_time0.5Buffer time in seconds before auto flushing recordsput_rate_limit_per_shard1000"A single shard can ingest up to 1 MiB of data per second (including partition keys) or 1,000 records per second for writes"put_bandwidth_limit_per_shard1024Kb per sec. max is 1024 per shard (ie 1 MiB). Keep below to minimize ProvisionedThroughputExceeded" errors *batch_size500"Each PutRecords request can support up to 500 records"max_queue_size10000put() method will block when queue is at maxafter_flush_funNoneasync function to call after doing a flush (err put_records()) callprocessorJsonProcessor()Record aggregator/serializer. Default is JSON without aggregation. Note this is highly inefficient as each record can be up to 1Mibretry_limitNoneHow many connection attempts should be made before raising a exceptionexpo_backoffNoneExponential Backoff when connection attempt failsexpo_backoff_limit120Max amount of seconds Exponential Backoff can growcreate_streamFalseCreates a Kinesis Stream based on thestream_namekeyword argument. Note if stream already existing it will ignorecreate_stream_shards1Sets the amount of shard you want for your new stream. Note if stream already existing it will ignoreThroughput exceeded. The docs (for Java/KPL see:https://docs.aws.amazon.com/streams/latest/dev/kinesis-producer-adv-retries-rate-limiting.html) state:You can lower this limit to reduce spamming due to excessive retries. However, the best practice is for each producer is to retry for maximum throughput aggressively and to handle any resulting throttling determined as excessive by expanding the capacity of the stream and implementing an appropriate partition key strategy.Even though our default here is to limit at this threshold (1024kb) in reality the threshold seems lower (~80%).
If you wish to avoid excessive throttling or have multiple producers on a stream you will want to set this quite a bit lower.Consumerfrom kinesis import Consumer

async with Consumer(stream_name="test") as consumer:
    while True:
        async for item in consumer:
            print(item)
        # caught up.. take a breather~Options:(comments in quotes are Kinesis Limits as per AWS Docs)ArgDefaultDescriptionsessionNoneAioSession (to use non default profile etc)region_nameNoneAWS Regionmax_queue_size10000the fetch() task shard will block when queue is at maxmax_shard_consumersNoneMax number of shards to use. None = allrecord_limit10000Number of records to fetch with get_records()sleep_time_no_records2No of seconds to sleep when caught upiterator_typeTRIM_HORIZONDefault shard iterator type for new/unknown shards (ie start from start of stream). Alternatives are "LATEST" (ie end of stream), "AT_TIMESTAMP" (ie particular point in time, requires definingtimestamparg)shard_fetch_rate1No of fetches per second (max = 5). 1 is recommended as allows having multiple consumers without hitting the max limit.checkpointerMemoryCheckPointer()Checkpointer to useprocessorJsonProcessor()Record aggregator/serializer. Must Match processor used by Producer()retry_limitNoneHow many connection attempts should be made before raising a exceptionexpo_backoffNoneExponential Backoff when connection attempt failsexpo_backoff_limit120Max amount of seconds Exponential Backoff can growcreate_streamFalseCreates a Kinesis Stream based on thestream_namekeyword argument. Note if stream already existing it will ignorecreate_stream_shards1Sets the amount of shard you want for your new stream. Note if stream already existing it will ignoretimestampNoneTimestamp to start reading stream from. Used with iterator type "AT_TIMESTAMP"Checkpointersmemory (the default but kinda pointless)MemoryCheckPointer()redisRedisCheckPointer(name, session_timeout=60, heartbeat_frequency=15, is_cluster=False)Requires ENV:REDIS_HOSTRequirespip install aredisProcessors (Aggregator + Serializer)Aggregation enable batching up multiple records to more efficiently use the stream.
Referhttps://aws.amazon.com/blogs/big-data/implementing-efficient-and-reliable-producers-with-the-amazon-kinesis-producer-library/ClassAggregatorSerializerDescriptionStringProcessorSimpleAggregatorStringSerializerSingle String recordJsonProcessorSimpleAggregatorJsonSerializerSingle JSON recordJsonLineProcessorNewlineAggregatorJsonSerializerMultiple JSON record separated by new line charJsonListProcessorListAggregatorJsonSerializerMultiple JSON record returned by listMsgpackProcessorNetstringAggregatorMsgpackSerializerMultiple Msgpack record framed with Netstring Protocol (https://en.wikipedia.org/wiki/Netstring)KPLJsonProcessorKPLAggregatorJsonSerializerMultiple JSON record in a KPL Aggregated Record (https://github.com/awslabs/amazon-kinesis-producer/blob/master/aggregation-format.md)KPLStringProcessorKPLAggregatorStringSerializerMultiple String record in a KPL Aggregated Record (https://github.com/awslabs/amazon-kinesis-producer/blob/master/aggregation-format.md)Note you can define your own processor easily as it's simply a class inheriting the Aggregator + Serializer.class MsgpackProcessor(Processor, NetstringAggregator, MsgpackSerializer):
    passJust define a new Serializer class with serialize() and deserialize() methods.Note:Json will usepip install ujsonif installedMsgpack requirespip install msgpackto be installedKPL requirespip install aws-kinesis-aggto be installedBenchmark/ExampleSeebenchmark.pyfor code50k items of approx 1k (python) in size, using single shard.Unit TestingUseshttps://github.com/mhart/kinesalitefor local testing.Run tests via dockerdocker-compose up --abort-on-container-exit --exit-code-from testFor local testing usedocker-compose up kinesis redisthen within your virtualenvnosetests

# or run individual test
nosetests tests.py:KinesisTests.test_create_stream_shard_limit_exceededNote there are a few test cases using theactualAWS Kinesis (AWSKinesisTests)
These require setting an env in order to runCreate an ".env" file withTESTING_USE_AWS_KINESIS=1Note you can ignore these tests if submitting PR unless core batching/processing behaviour is being changed.