Package: aws-kinesis-consumer
aws-kinesis-consumerConsume anAWS Kinesis Data Streamto look over the records from a terminal.Demo$aws-kinesis-consumer--stream-nameMyStream
>preparingshards2/2
>shard_id=shardId-000000000000,records=1Record-001
>shard_id=shardId-000000000001,records=2Record-002
Record-003UsagePre-requirementConnect to AWS and set
thedefault AWS environment variables.exportAWS_DEFAULT_REGION=eu-central-1exportAWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLEexportAWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEYAlternatively,aws configurecan help to set the environment variables.Use with Python# installpipinstallaws-kinesis-consumer# consume a streamaws-kinesis-consumer--stream-nameMyStreamUse with Dockerdockerrun\-eAWS_DEFAULT_REGION\-eAWS_ACCESS_KEY_ID\-eAWS_SECRET_ACCESS_KEY\-eAWS_SESSION_TOKEN\thinow/aws-kinesis-consumer--stream-nameMyStreamArgumentsDefaultDescription-s--stream-nameREQUIREDName of the AWS Kinesis Stream.-e--endpointCustom AWS endpoint url to communicate with the AWS API. Could be used in order to specify a region (e.g.https://kinesis.us-east-1.amazonaws.com/).-i--iterator-typelatestDefines how to start consuming records from the stream. Uselatestto consume the new records only. Or usetrim-horizonto consume all the records already existing in the stream.-m--max-records-per-request10Defines the maximum number of records per request.-r--regionAWS region of the stream (e.g.us-east-1). By default, the AWS region available in your terminal (e.g. environment variableAWS_DEFAULT_REGION).-h--helpPrints the help message.-V--verbosePrints all the details when an error stops the application.-v--versionPrints the version of the binary.FAQWhat is the motivation ? What is the issue with AWS CLI ?TheAWS CLIis able to fetch
records from Kinesis, but the users need to list the shards, to generate iterator tokens, use subsequent tokens, delay
operations, and so on.aws-kinesis-consumerin contrary is able to get records by using the stream name, and only the stream name. Therefore
there is no need for an extra script.How to filter the records ?aws-kinesis-consumercan be piped with other command such asgrep,
or evenjqto filter json records.# all the records$aws-kinesis-consumer--stream-nameMyStream{"name":"foo","status":"ok"}{"name":"bar","status":"pending"}{"name":"baz","status":"error"}# records containing the text "ba" (e.g. "bar" and "baz", but not "foo")$aws-kinesis-consumer--stream-nameMyStream|grep"ba"{"name":"bar","status":"pending"}{"name":"baz","status":"error"}# records where the json property "status" has the value "error"$aws-kinesis-consumer--stream-nameMyStream|jq'select(.status == "error")'{"name":"baz","status":"error"}What are the required AWS permissions ?aws-kinesis-consumerrequires the following AWS permissions :kinesis:ListShardskinesis:GetShardIteratorkinesis:GetRecordsThe following policy is an example which can be applied to an AWS user or an AWS role :{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["kinesis:ListShards","kinesis:GetShardIterator","kinesis:GetRecords"],"Resource":["arn:aws:kinesis:REGION:ACCOUNT-ID:stream/STREAM-NAME"]}]}How to contribute to the project ?See the guidelines in the CONTRIBUTING.md file.ChangelogSee all the changes per release.Special thanksThanks to the contributors of thekinesaliteproject which make test and
development of this project extremely easy and reliable!