Connect Flink with Kinesis /Kinesalite using Scala
By: Avi Levi
Tags: Kinesis, Flink Streaming, Scala, Flink

Kinesalite provides local simulation of kinesis, this will enable to simulate a local instance of Kinesis on your local machine and you can execute all aws CLI commands locally simply by adding the — endpoint-url flag (e.g — endpoint-url=http://localhost:4567)

I want to share my journey building simple pipeline with Flink and Gradle as our build tool.

Lets start :

Dependencies (gradle.build):

compile group: 'com.lightbend.akka', name: 'akka-stream-alpakka-kinesis_2.12', version: '2.0.2'

compile group: 'com.amazonaws', name: 'aws-java-sdk-kinesis', version: '1.11.913'

compile group: 'org.apache.flink', name: 'flink-connector-kinesis_2.12', version: 1.11.2

Let’s write some code:

In general I used the following imports:

import org.apache.flink.api.common.serialization.SimpleStringSchema

import org.apache.flink.streaming.api.TimeCharacteristic

import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment

import org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants

import org.apache.flink.streaming.connectors.kinesis.{FlinkKinesisConsumer, FlinkKinesisProducer}

import org.apache.flink.streaming.api.scala._

If you want to use kinesalite you need to disable CBOR because it is not supported by kinesalite (no worries, you can use kinesis with CBOR disabled)

System.setProperty(com.amazonaws.SDKGlobalConfiguration.AWS_CBOR_DISABLE_SYSTEM_PROPERTY, "true")

System.setProperty(SDKGlobalConfiguration.AWS_CBOR_DISABLE_SYSTEM_PROPERTY, "true")

Add your credentials:

val systemProperties = System.getProperties

systemProperties.setProperty(AWS_ACCESS_KEY_ID, awsAccessKey)

systemProperties.setProperty(AWS_SECRET_KEY, awsSecretAccessKey)

systemProperties.setProperty(AWS_SESSION_TOKEN, awsSessionToken)//in case you have a session token

Now we can configure our consumer and producer (If you are using a security token you need to set credentials provider property as well as shown below):

def commonProperties(): Properties = {

val p: Properties = new Properties()

p.put(AWSConfigConstants.AWS_REGION, awsRegion)

p.put(com.amazonaws.auth.profile.internal.ProfileKeyConstants.AWS_SESSION_TOKEN, AwsConfig.awsSessionToken)

endPointOption.foreach { ep =>

p.put(AWSConfigConstants.AWS_ENDPOINT, ep)

}

p.put(AWSConfigConstants.AWS_CREDENTIALS_PROVIDER, "SYS_PROP")

p

}

The consumer (reads messages from the stream):

val consumerConfig: Properties = {

val p = commonProperties()

p.put(ConsumerConfigConstants.STREAM_INITIAL_POSITION, "LATEST")

p

}

val consumer = new FlinkKinesisConsumer[String](

"ExampleInputStream", new SimpleStringSchema, consumerConfig)

In this case we set the starting point of the consumer to consume only new messages “LATEST” other Valid Values: AT_SEQUENCE_NUMBER | AFTER_SEQUENCE_NUMBER | TRIM_HORIZON | LATEST | AT_TIMESTAMP you can read more about it here

The producer :

val producer = new FlinkKinesisProducer(new SimpleStringSchema(), producerConfig)

producer.setFailOnError(true)

producer.setDefaultStream("ExampleOutputStream")

producer.setDefaultPartition("0")

Now all we need to do is to set our Flink environment and assemble simple pipeline that reads from the input stream and omit that message to the output stream:

val env = StreamExecutionEnvironment.getExecutionEnvironment

env.enableCheckpointing(5000)

env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime) env.addSource(consumer)

.map { str =>

println("*" * 50) //nothing much just printing the input

println(str)

println("*" * 50)

str

}

.addSink(producer) env.execute("Sample Job")

If you want to use Kinesalite you can download the docker container from here

using AWS* (using the CLI):

We need to create the appropriate streams :

aws kinesis create-stream \

--endpoint-url=http://localhost:4567 \

--stream-name ExampleInputStream \

--shard-count 2

and now we can push some messages to stream

aws --endpoint-url=http://localhost:4567 kinesis put-record --stream-name ExampleInputStream --partition-key '0' --data 'Your message'

*The following examples are aimed for Kinesalite, if you want to work with kinesis, simply omit the endpoint line.