Twitter Streaming with Apache Kafka
By: Shefali Bisht
Tags: Engineering, Twitter, Kafka, Technology, Big Data

Writing Kafka Producer ..

Add following maven dependencies in pom.xml file. Kafka-client library for connecting to kafka and slf4j-simple for logging messages. We’ll be using twitter Hosebird client core which is a Java HTTP client for consuming Twitter’s standard Streaming API.

<dependencies>

<dependency>

<groupId>com.twitter</groupId>

<artifactId>hbc-core</artifactId>

<version>2.2.0</version>

</dependency>

<dependency>

<groupId>org.apache.kafka</group Id>

<artifactId>kafka-clients</artifactId>

<version>2.3.0</version>

</dependency>

<dependency>

<groupId>org.slf4j</groupId>

<artifactId>slf4j-simple</artifactId>

<version>1.7.26</version>

</dependency>

</dependencies>

Create KafkaConfig.java file to store credentials. It’s always better to store configurations in a config file rather than to directly expose or hard code it.

public static final String BOOTSTRAPSERVERS = "127.0.0.1:9092";

public static final String TOPIC = "Twitter-Kafka";

public static final String ACKS_CONFIG = "all";

public static final String MAX_IN_FLIGHT_CONN = "5"; public static final String COMPRESSION_TYPE = "snappy";

public static final String RETRIES_CONFIG = Integer.toString(Integer.MAX_VALUE);

public static final String LINGER_CONFIG = "20";

public static final String BATCH_SIZE = Integer.toString(32*1024);

Similarly create TwitterConfig.java to store Twitter API keys and access tokens

public static final String CONSUMER_KEYS= "H8***42";

public static final String CONSUMER_SECRETS= "xn***dI";

public static final String SECRET = "SPz***Xg";

public static final String TOKEN = "13***1";

Twitter Producer class

Create TwitterProducer.java which will look something like

Here we are searching for term ‘coronavirus’ to filter out tweets.

Now let’s write createTwitterClient() and createKafkaProducer() to connect to Twitter and Kafka respectively. Logger will be used to log messages.

Twitter Client

Method createTwitterClient() will use Hosebird client to sccs and return client.

Kafka Producer Settings

The createKafkaProducer() method uses KafkaConfig.java to get property settings to connect to our Kafka server (bootstrap-server) , serializes the keys and values to be sent to the Kafka broker. Few other settings have been configured like ACKS_CONFIG, BATCH_SIZE_CONFIG, ENABLE_IDEMPOTENCE_CONFIG, RETRIES_CONFIG etc. to optimize the producer.

Finishing up ..

Now we create the run() method which first calls createTwitterClient() method to connect to Twitter client and then createKafkaProducer() to create and get kafka producer instance.

We use BlockingQueue<String> msgQueue to fetch the data from client and producer.send(new ProducerRecord<String, String>(KafkaConfig.TOPIC, null, msg) to send data to the kafka topic.

Here is the while() loop which runs till the client in not finished.

while (!client.isDone()) {

String msg = null;

try {

msg = msgQueue.poll(5, TimeUnit.SECONDS);

} catch (InterruptedException e) {

// Stop client

client.stop();

}

if (msg != null) {

producer.send(new ProducerRecord<String, String>(KafkaConfig.TOPIC, null, msg), new Callback() {

// Override onCompletion() method

});

}

}

Here is the snippet of run() method.

Let’s start streaming ..

Finally we start the Kafka Producer by running TwitterProducer().run()

public static void main(String[] args) {

new TwitterProducer().run();

}

Output

Real time Twitter streaming

Lets verify that the data is published to the kafka topic.

bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic Twitter-Kafka — group my-first-app --from-beginning

If the consumer does not already have an established offset to consume from, using --from-beginning will start with the earliest message present in the log rather than the latest message.

Kafka consumer (console)

In the next part we’ll see how to create a Kafka Producer class in Java to subscribe to Kafka topic, read tweets and ingest the data into MongoDB.

For your reference, the full code is available on GitHub.

Conclusion

We saw how a Java Kafka producer can be created with a simple Java HTTP kafka-client library. We used Twitter API to stream user tweets in real time and publish them to a Kafka topic.