Amazon Kinesis Data Streams Connector 1.0 Reference | MuleSoft Documentation
# Amazon Kinesis Data Streams Connector 1.0 Reference
Amazon Kinesis Data Streams Connector doesn’t support Windows because it uses the Kinesis Producer Library (KPL), which deprecated Windows support starting in version 0.14.0.   
---  
## Configurations
### Config
The configuration for the connector. You can use this configuration for both the producer and consumer.
#### Parameters
Name | Type | Description | Default Value | Required  
---|---|---|---|---  
The name for this configuration. Connectors reference the configuration with this name.  
Connection |  Kinesis Connection | The connection types to provide to this configuration.  
Aggregation Enabled | When set to `true`, this option packs multiple user data records into a single Amazon Kinesis data record.  
Response Timeout | The maximum total time that can elapse between the start of an HTTP request and when the responses are received. If this time is exceeded, the request times out.  
Request Timeout Unit |  Enumeration, one of:
  * NANOSECONDS
  * MICROSECONDS
  * MILLISECONDS
  * SECONDS
  * MINUTES
  * HOURS

| A time unit that qualifies the **Response Timeout** value  
Record Max Buffered Time | Maximum amount of time that an Amazon Kinesis data record spends buffering before it is sent. Records can be sent earlier than this value, depending on other buffering limits.  
Record Max Buffered Time Unit |  Enumeration, one of:
  * NANOSECONDS
  * MICROSECONDS
  * MILLISECONDS
  * SECONDS
  * MINUTES
  * HOURS

| A time unit that qualifies the **Record Max Buffered Time** value  
Aggregation Max Count | Maximum number of items to pack into an aggregated Amazon Kinesis data record. | 4294967295  
Aggregation Max Size | Maximum data size to pack into an aggregated Amazon Kinesis data record.  
Aggregation Max Size Unit |  Enumeration, one of: | A data unit that qualifies the **Aggregation Max Size** value  
Collection Max Count | Maximum number of items to pack into a batch request.  
Collection Max Size | Maximum amount of data to send with a **Put Record** operation.  
Collection Max Size Unit |  Enumeration, one of: | A data unit that qualifies the **Collection Max Size** value  
Connection Timeout | The amount of time to wait when initially establishing the TCP connection between the connector and Amazon Kinesis server before throwing an exception if the connection fails  
Connection Timeout Unit |  Enumeration, one of:
  * NANOSECONDS
  * MICROSECONDS
  * MILLISECONDS
  * SECONDS
  * MINUTES
  * HOURS

| The time unit for the **Connection Timeout** value  
Max Connections | Maximum number of connections to open to the backend. HTTP requests are sent in parallel over multiple connections. Setting this value too high can impact latency and consume additional resources without increasing throughput.  
Min Connections | Minimum number of connections to keep open to the backend. In most cases, there is no need to increase value.  
Rate Limit | Limits the number of data records per second and number of bytes per second sent to a shard. This limit can reduce bandwidth and CPU cycle wastage from sending requests that will fail from throttling. The default value, 150%, enables a single producer instance to completely saturate the allowance for a shard. To reduce throttling errors rather than completely saturate the shard, consider reducing this setting.  
Record TTL | Time limit for data records to be put. Records that are not successfully put within this limit fail. When setting this value, take into consideration the fact that data records still incur network latency after they leave the Amazon Kinesis Producer Library (KPL).  
Record TTL Unit |  Enumeration, one of:
  * NANOSECONDS
  * MICROSECONDS
  * MILLISECONDS
  * SECONDS
  * MINUTES
  * HOURS

| The time unit for the **Record TTL** value.  
Thread Pool Size | Maximum number of threads with which to configure the native process thread pool. The default value, 0, enables the KPL process to choose the size of the thread pool.  
#### Connection Types
##### Kinesis Connection
Connects to AWS Kinesis through the use of a user-specified access and secret key and, optionally, a role. These credentials securely sign requests sent to AWS services.
###### Parameters
Name | Type | Description | Default Value | Required  
---|---|---|---|---  
Proxy Configuration |  Proxy Configuration | Proxy connection settings for outbound connections. This setting applies to the **Listener** source and **Checkpoint** operation. It does not affect the **Put Record** operation.  
TLS Configuration | Protocol to use for communication. When using the HTTPS protocol, the HTTP communication is secured using TLS or SSL. If HTTPS is configured as the protocol, then, at a minimum, you must configure the keystore in the `tls:context` child element of the `listener-config`. This setting applies to the **Listener** source and **Checkpoint** operation. It does not affect the **Put Record** operation.  
Access Key | Access key provided by Amazon.  
Secret Key | Secret key provided by Amazon.  
Region Endpoint | Topic region endpoint | us-east-1  
Role configuration  
Connection Timeout | The amount of time to wait when initially establishing the TCP connection between the connector and Amazon Kinesis server before throwing an exception if the connection fails  
Connection Timeout Time Unit |  Enumeration, one of:
  * NANOSECONDS
  * MICROSECONDS
  * MILLISECONDS
  * SECONDS
  * MINUTES
  * HOURS

| The time unit for the **Connection Timeout** field.  
Response Timeout | The maximum time to wait between the time an HTTP request is sent and when a response is received  
Response Timeout Unit |  Enumeration, one of:
  * NANOSECONDS
  * MICROSECONDS
  * MILLISECONDS
  * SECONDS
  * MINUTES
  * HOURS

| Time unit for the **Response Timeout** value.  
Reconnection |  | When the application is deployed, a connectivity test is performed on all connectors. If set to `true`, deployment fails if the test doesn’t pass after exhausting the associated reconnection strategy.  
## Supported Operations
  *   * 

## Associated Sources
## Checkpoint
`<kinesis:checkpoint>`
The operation checkpoint position of a listener into the Amazon DynamoDB table. Upon failover, the connector starts fetching records that are located after the check pointed position. The **Checkpoint** operation does not modify the current position of the listener.
### Parameters
Name | Type | Description | Default Value | Required  
---|---|---|---|---  
Configuration | The name of the configuration to use.  
Stream Name | The stream name.  
Application Name | Name of the DynamoDB table that holds data about the current stream positions.  
Shard ID | The shard identifier.  
Sequence Number | A sequence number at which to checkpoint in the shard.  
Subsequence Number | A subsequence number at which to checkpoint within the shard.  
Config Ref |  ConfigurationProvider | The name of the configuration to use to execute this component  
### For Configurations
### Throws
  * KINESIS:ILLEGAL_STATE
  * KINESIS:INVALID_ARGUMENT
  * KINESIS:PROVISIONED_THROUGHPUT_EXCEEDED
  * KINESIS:TIMEOUT
  * KINESIS:UNAUTHORIZED


## Put Record
`<kinesis:put-record>`
Puts a data record into an Amazon Kinesis data stream.
### Parameters
Name | Type | Description | Default Value | Required  
---|---|---|---|---  
Configuration | The name of the configuration to use.  
Stream Name | The stream name  
Partition Key | The partition key. If not provided, the connector generates a UUID for this value.  
Explicit Hash Key | Overrides the Amazon Kinesis Data Streams explicitHashKey value, which is normally computed with using MD5 function of the data record partition key.  
The content to put in the Amazon Kinesis data stream. This content can be up to 1 MB in size. | #[payload]  
Config Ref |  ConfigurationProvider | The name of the configuration to use to execute this component  
Streaming Strategy | 
  * Repeatable In Memory Stream
  * Repeatable File Store Stream
  * non-repeatable-stream

| Disables the repeatable stream functionality and uses non-repeatable streams to have lower performance overhead, memory use, and costs  
Target Variable | Name of the variable in which to store the operation’s output  
Target Value | Expression that evaluates the operation’s output. The expression outcome is stored in the target variable. | #[payload]  
Reconnection Strategy | 
  *   * Reconnect Forever

| A retry strategy in case of connectivity errors  
### For Configurations
### Throws
  * KINESIS:CONNECTIVITY
  * KINESIS:INVALID_ARGUMENT
  * KINESIS:PROCESSING
  * KINESIS:RETRY_EXHAUSTED
  * KINESIS:TIMEOUT
  * KINESIS:UNAUTHORIZED
| Disables the repeatable stream functionality and uses non-repeatable streams to have lower performance overhead, memory use, and costs  
Target Variable | Name of the variable in which to store the operation’s output  
Target Value | Expression that evaluates the operation’s output. The expression outcome is stored in the target variable. | #[payload]  
Reconnection Strategy | 
  *   * Reconnect Forever

| A retry strategy in case of connectivity errors  
### For Configurations
### Throws
  * KINESIS:CONNECTIVITY
  * KINESIS:INVALID_ARGUMENT
  * KINESIS:PROCESSING
  * KINESIS:RETRY_EXHAUSTED
  * KINESIS:TIMEOUT
  * KINESIS:UNAUTHORIZED


## Listener
`<kinesis:listener>`
Uses the Amazon Kinesis Client Library (KCL) to listen from a specified point in a stream.
Due to varying batch sizes, checkpoints can occur in the incorrect sequence, potentially leading to transaction loss in certain cases. Set `maxConcurrency=1` to ensure transactions are processed sequentially.   
---  
### Parameters
Name | Type | Description | Default Value | Required  
---|---|---|---|---  
Configuration | The name of the configuration to use.  
Output Mime Type | The MIME type of the payload that this operation outputs.  
Output Encoding | The encoding of the payload that this operation outputs.  
Config Ref |  ConfigurationProvider | The name of the configuration to use to execute this component  
Primary Node Only | Whether this listener executes on the primary node only when running in a cluster  
On Capacity Overload |  Enumeration, one of: | Strategy that Mule applies when the flow receives more messages than it can handle  
Redelivery Policy |  Redelivery Policy | Policy for processing the redelivery of a message  
Stream Name | The stream name.  
Application Name | Name of the DynamoDB table that holds data about current stream positions. If the table doesn’t exist, then the connector creates a new one at the initial position.  
Shard Configs |  Array of Shard Config | The shard configuration. If specified, the listener listens only from selected shards.  
Max Batch Size | Maximum number of data records a batch can carry. A batch is the number of records that the listener receives from the stream.  
Max Leases For Worker | Number of shards to which the listener can subscribe in parallel. | 2137483647  
Shard Prioritization |  Enumeration, one of:
  * PARENTS_FIRST_SHARD_PRIORITIZATION
  * NO_OP_SHARD_PRIORITIZATION

| Logic used to prioritize or filter the shards before their execution. PARENTS_FIRST_SHARD_PRIORITIZATION Prioritizes parent shards first and limits the number of shards that are available for initialization based on their depth (**Max Depth** field). NO_OP_SHARD_PRIORITIZATION Returns the original list of shards without any modifications. | NO_OP_SHARD_PRIORITIZATION  
Max Depth | Used by the PARENTS_FIRST_SHARD_PRIORITIZATION shard prioritization option. Any shard that is deeper than this value is excluded from processing.  
Read Capacity Units | The maximum number of strongly consistent reads consumed per second before Amazon DynamoDB returns a ThrottlingException.  
Write Capacity Units | The maximum number of writes consumed per second before Amazon DynamoDB returns a ThrottlingException.  
Failover Time | Duration of a lease for a worker. This parameter also determines the frequency with which a worker looks for new leases to work on. The frequency is roughly twice the specified value.  
Failover Time Unit |  Enumeration, one of:
  * NANOSECONDS
  * MICROSECONDS
  * MILLISECONDS
  * SECONDS
  * MINUTES
  * HOURS

| Time unit for the **Failover Time** field.  
Checkpoint On Complete | If `true`, the listener checkpoints the current reading position to Amazon DynamoDB after data records are processed by the owning flow, without taking into account whether the processing finished successfully.  
Absolute Position |  Enumeration, one of:
  * LATEST
  * FIRST

| Whether the listener listens from the latest data record or the oldest data record  
Timestamp | Date and time from which the listener listens  
Reconnection Strategy | 
  *   * Reconnect Forever

| A retry strategy in case of connectivity errors  
### Output
Array of binary messages with [Record Attributes]  
---  
### For Configurations
## Types
### Proxy Configuration
Field | Type | Description | Default Value | Required  
---|---|---|---|---  
Host where the proxy requests are sent.  
Port where the proxy requests are sent.  
The username to authenticate against the proxy.  
The password to authenticate against the proxy.  
**Non Proxy Hosts** |  Array of String |  A list of hosts against which the proxy should not be used.  
**Ntlm Domain** |  The domain to authenticate against the proxy.  
Field | Type | Description | Default Value | Required  
---|---|---|---|---  
**Enabled Protocols** |  A comma-separated list of protocols enabled for this context.  
**Enabled Cipher Suites** |  A comma-separated list of cipher suites enabled for this context.  
**Trust Store**  
**Revocation Check** | 
  * Standard Revocation Check
  * Custom Ocsp Responder

  
### Trust Store
Field | Type | Description | Default Value | Required  
---|---|---|---|---  
The location (which is resolved relative to the current classpath and file system, if possible) of the truststore.  
The password used to protect the truststore.  
The type of store used.  
The algorithm used by the truststore.  
If set to `true`, no certificate validations are performed, which makes connections vulnerable to attacks. Use at your own risk.  
### Key Store
Field | Type | Description | Default Value | Required  
---|---|---|---|---  
The location (which is resolved relative to the current classpath and file system, if possible) of the keystore.  
The type of store used.  
When the keystore contains many private keys, this attribute indicates the alias of the key that should be used. If not defined, the first key in the file is used by default.  
**Key Password** |  The password used to protect the private key.  
The password used to protect the keystore.  
The algorithm used by the keystore.  
### Standard Revocation Check
Field | Type | Description | Default Value | Required  
---|---|---|---|---  
**Only End Entities** |  Verifies the last element of the certificate chain only.  
**Prefer Crls** |  Tries CRL instead of OCSP first.  
**No Fallback** |  Does not use the secondary checking method.  
Avoids a certification failure when the revocation server cannot be reached or is busy.  
### Custom Ocsp Responder
Field | Type | Description | Default Value | Required  
---|---|---|---|---  
The URL of the OCSP responder.  
**Cert Alias** |  Alias of the signing certificate for the OCSP response (must be in the truststore), if present.  
### Crl File
Field | Type | Description | Default Value | Required  
---|---|---|---|---  
The path to the CRL file.  
### Role
Field | Type | Description | Default Value | Required  
---|---|---|---|---  
The Amazon Resource Name (ARN) of the role to assume.  
**External Id** |  A unique identifier that might be required when you assume a role in another account. If the administrator of the account to which the role belongs provides an external ID, then provide that value in this field.  
The duration of the role session.  
**Duration Time Unit** |  Enumeration, one of:
  * NANOSECONDS
  * MICROSECONDS
  * MILLISECONDS
  * SECONDS
  * MINUTES
  * HOURS

|  Time unit for the **Duration** field  
**Referred Policy Arns** |  Array of String |  The Amazon Resource Names (ARNs) of the IAM-managed policies to use as managed session policies. The policies must exist in the same account as the role.  
A list of session tags that you want to pass. Each session tag consists of a key name and an associated value.  
### Reconnection
Field | Type | Description | Default Value | Required  
---|---|---|---|---  
**Fails Deployment** |  When the application is deployed, a connectivity test is performed on all connectors. If set to `true`, deployment fails if the test doesn’t pass after exhausting the associated reconnection strategy.  
**Reconnection Strategy** | 
  *   * Reconnect Forever
|  Time unit for the **Duration** field  
**Referred Policy Arns** |  Array of String |  The Amazon Resource Names (ARNs) of the IAM-managed policies to use as managed session policies. The policies must exist in the same account as the role.  
A list of session tags that you want to pass. Each session tag consists of a key name and an associated value.  
### Reconnection
Field | Type | Description | Default Value | Required  
---|---|---|---|---  
**Fails Deployment** |  When the application is deployed, a connectivity test is performed on all connectors. If set to `true`, deployment fails if the test doesn’t pass after exhausting the associated reconnection strategy.  
**Reconnection Strategy** | 
  *   * Reconnect Forever

|  The reconnection strategy to use.  
### Reconnect
Field | Type | Description | Default Value | Required  
---|---|---|---|---  
How often in milliseconds to reconnect  
How many reconnection attempts to make.  
### Reconnect Forever
Field | Type | Description | Default Value | Required  
---|---|---|---|---  
How often in milliseconds to reconnect  
### Record Attributes
Field | Type | Description | Default Value | Required  
---|---|---|---|---  
**Aggregated**  
**Approximate Arrival Timestamp**  
**Encryption Type**  
**Partition Key**  
**Partition Key**  
**Sequence Number**  
**Sub Sequence Number**  
### Redelivery Policy
Field | Type | Description | Default Value | Required  
---|---|---|---|---  
**Max Redelivery Count** |  The maximum number of times a message can be redelivered and processed unsuccessfully before triggering a process-failed message  
**Message Digest Algorithm** |  The secure hashing algorithm to use.  
**Message Identifier** |  Redelivery Policy Message Identifier |  The strategy used to identify the messages.  
**Object Store** |  ObjectStore |  The object store that stores the redelivery counter for each message.  
### Redelivery Policy Message Identifier
Field | Type | Description | Default Value | Required  
---|---|---|---|---  
**Use Secure Hash** |  Whether to use a secure hash algorithm to identify a redelivered message.  
**Id Expression** |  One or more expressions to use to determine when a message was redelivered. This property can be set only if the value of the **Use Secure Hash** field is `false`.  
### Shard Config
Field | Type | Description | Default Value | Required  
---|---|---|---|---  
The shard identifier.  
**Shard Initial Position** |  Shard Initial Position |  The shard initial position.  
### Shard Initial Position
Field | Type | Description | Default Value | Required  
---|---|---|---|---  
**At Sequence Number** |  The listener starts listening at this position in the data stream.  
**After Sequence Number** |  The listener starts listening after this position in the data stream.  
**Absolute Position** |  Enumeration, one of:
  * LATEST
  * FIRST

|  The listener starts listening from the latest or oldest record, depending on the value.  
### Repeatable In Memory Stream
Field | Type | Description | Default Value | Required  
---|---|---|---|---  
**Initial Buffer Size** |  The amount of memory to allocate to consume the stream and provide random access to it. If the stream contains more data than can fit into this buffer, then the buffer expands according to the **Buffer Size Increment** value, with an upper limit of the **Max In Memory Size** value.  
**Buffer Size Increment** |  This is by how much the buffer size expands if it exceeds its initial size. Setting a value of zero or lower means that the buffer should not expand, meaning that a `STREAM_MAXIMUM_SIZE_EXCEEDED` error is raised when the buffer is full.  
**Max Buffer Size** |  The maximum amount of memory to use. If more memory is used, then a `STREAM_MAXIMUM_SIZE_EXCEEDED` error is raised. A value less than or equal to zero means no limit.  
**Buffer Unit** |  Enumeration, one of: |  The unit in which the fields in this table are expressed  
### Repeatable File Store Stream
Field | Type | Description | Default Value | Required  
---|---|---|---|---  
**In Memory Size** |  Maximum memory that the stream should use to keep data in memory. If more than that is consumed, content on the disk is buffered.  
**Buffer Unit** |  Enumeration, one of: |  The unit in which the **In Memory Size** value is expressed  
## See Also
  * Introduction to Anypoint Connectors
  *   * MuleSoft Help Center