Package: aws-dynamodb-parallel-scan
aws-dynamodb-parallel-scanAmazon DynamoDB parallel scan paginator for boto3.InstallationInstall from PyPI with pippip install aws-dynamodb-parallel-scanor with the package manager of choice.UsageThe library is a drop-in replacement forboto3 DynamoDB Scan Paginator. Example:importaws_dynamodb_parallel_scanimportboto3# Create DynamoDB client to use for scan operationsclient=boto3.resource("dynamodb").meta.client# Create the parallel scan paginator with the clientpaginator=aws_dynamodb_parallel_scan.get_paginator(client)# Scan "mytable" in five segments. Each segment is scanned in parallel.forpageinpaginator.paginate(TableName="mytable",TotalSegments=5):items=page.get("Items",[])Notes:paginate()accepts the same arguments as boto3DynamoDB.Client.scan()method. Arguments
are passed toDynamoDB.Client.scan()as-is.paginate()uses the value ofTotalSegmentsargument as parallelism level. Each segment
is scanned in parallel in a separate thread.paginate()yields DynamoDB Scan API responses in the same format as boto3DynamoDB.Client.scan()method.See boto3DynamoDB.Client.scan() documentationfor details on supported arguments and the response format.CLIThis package also provides a CLI tool (aws-dynamodb-parallel-scan) to scan a DynamoDB table
with parallel scan. The tool supports all non-deprecated arguments of DynamoDB Scan API. Executeaws-dynamodb-parallel-scan -hfor detailsHere's some examples:# Scan "mytable" sequentially$aws-dynamodb-parallel-scan--table-namemytable{"Items":[...],"Count":10256,"ScannedCount":10256,"ResponseMetadata":{}}{"Items":[...],"Count":12,"ScannedCount":12,"ResponseMetadata":{}}# Scan "mytable" in parallel (5 parallel segments)$aws-dynamodb-parallel-scan--table-namemytable--total-segments5{"Items":[...],"Count":32,"ScannedCount":32,"ResponseMetadata":{}}{"Items":[...],"Count":47,"ScannedCount":47,"ResponseMetadata":{}}{"Items":[...],"Count":52,"ScannedCount":52,"ResponseMetadata":{}}{"Items":[...],"Count":34,"ScannedCount":34,"ResponseMetadata":{}}{"Items":[...],"Count":40,"ScannedCount":40,"ResponseMetadata":{}}# Scan "mytable" in parallel and return items, not Scan API responses (--output-items flag)$aws-dynamodb-parallel-scan--table-namemytable--total-segments5\--output-items{"pk":{"S":"item1"},"quantity":{"N":"99"}}{"pk":{"S":"item24"},"quantity":{"N":"25"}}...# Scan "mytable" in parallel, return items with native types, not DynamoDB types (--use-document-client flag)$aws-dynamodb-parallel-scan--table-namemytable--total-segments5\--output-items--use-document-client{"pk":"item1","quantity":99}{"pk":"item24","quantity":25}...# Scan "mytable" with a filter expression, return items$aws-dynamodb-parallel-scan--table-namemytable--total-segments5\--filter-expression"quantity < :value"\--expression-attribute-values'{":value": {"N": "5"}}'\--output-items{"pk":{"S":"item142"},"quantity":{"N":"4"}}{"pk":{"S":"item874"},"quantity":{"N":"1"}}# Scan "mytable" with a filter expression using native types, return items$aws-dynamodb-parallel-scan--table-namemytable--total-segments5\--filter-expression"quantity < :value"\--expression-attribute-values'{":value": 5}'\--use-document-client--output-items{"pk":"item142","quantity":4}{"pk":"item874","quantity":1}DevelopmentRequires Python 3 and Poetry. Useful commands:# Setup environmentpoetryinstall# Run tests (integration test requires rights to create, delete and use DynamoDB tables)maketest# Run lintersmake-klint# Format codemakeformatLicenseMITCreditsAlex Chan,Getting every item from a DynamoDB table with Python