Package: py-kinesis
async-kinesisFeaturesuses queues for both producer and consumerproducer flushes with put_records() if has enough to flush or after "buffer_time" reachedconsumer iterates over msg queue independent of shard readersConfigurable to handle Sharding limits but will throttle/retry if requiredie multiple independent clients are saturating the ShardsCheckpointing with heartbeatsdeadlock + reallocation of shards if checkpoint fails to heartbeat within "session_timeout"Consumer Design(Bears some explanation, kinda complex~)fetch() gets called periodically (0.2 sec (ie max 5x per second as is the limit on shard get_records()))iterate over the list of shards (set on startup, does not currently detect resharding)assign shard if not in use and not at "max_shard_consumers" limit otherwise ignore/continueignore/continue if this shard is still fetchingprocess records if shard is done fetchingput records on queueadd checkpoint record to queueassign NextShardIteratorcreate (get_records()) task againNote that get_records() is throttled via "shard_fetch_rate=5" (ie the same 0.2 sec/ 5x limit)This pattern seemed like the easiest way to maintain a pool of consumers without needing to think too hard about starting it's next job or handling new shards etc.Not Implementedreshardingclient rebalancing (ie share the shards between consumers)See alsohttps://aws.amazon.com/blogs/big-data/implementing-efficient-and-reliable-producers-with-the-amazon-kinesis-producer-library/Producerasync with Producer(stream_name="test") as producer:
    # Put item onto queue to be flushed via put_records()
    await producer.put({'my': 'data'})Options:(comments in quotes are Kinesis Limits as per AWS Docs)region_nameAWS Regionbuffer_time=0.5Buffer time in seconds before auto flushing recordsput_rate_limit_per_shard=1000"A single shard can ingest up to 1 MiB of data per second (including partition keys) or 1,000 records per second for writesbatch_size=500"Each PutRecords request can support up to 500 records"max_queue_size=10000put() method will block when queue is at maxafter_flush_funasync function to call after doing a flush (err put_records()) callConsumerasync with Consumer(stream_name="test") as consumer:
    while True:
        async for item in consumer:
            print(item)
        # caught up.. take a breather~Options:(comments in quotes are Kinesis Limits as per AWS Docs)region_nameAWS Regionmax_queue_size=1000the fetch() task shardmax_shard_consumers=NoneMax number of shards to use. None = allrecord_limit=10000Number of records to fetch with get_records()sleep_time_no_records=2No of seconds to sleep when caught upiterator_type="TRIM_HORIZON"Default shard iterator type for new/unknown shards (ie start from start of stream)
Alternative is "LATEST" (ie end of stream)shard_fetch_rate=5No of fetches per second (max = 5)checkpointer=NoneCheckpointer to useCheckpointersmemoryredisYet another Python Kinesis Library?Sadly I had issues with every other library I could find :(https://github.com/NerdWalletOSS/kinesis-pythonpro:kinda worksconthreadedOutstanding PR to fix some issuescheckpoints on every record on main threadhttps://github.com/ungikim/kinsumerpro:handles shard changesno producerno redis checkpointer/heartbeatthreaded/seems kinda complicated~conconsumer onlyhttps://github.com/bufferapp/kinerpro:BatchingconProducer onlyhttps://github.com/niklio/aiokinesispro:asynciono checkpointingconlimited to 1 shard / too simplistichttps://github.com/ticketea/pynesispro:checkpointsconhasn't been updated for 1 yeardoesnt use put_records()single threaded / round robin reads shardshttps://github.com/whale2/async-kinesis-clientpro:checkpointsasynciocon?(Actually I only found this one recently, might be ok alternative?)