This ecosystem is a micro-services architecture, not a stream processing application.
By: Martin Swanson
Tags: Flink, Stream Processing, Web Crawler, Microservices, Kafka

This ecosystem is a micro-services architecture, not a stream processing application.

This is something I see quite frequently – people start with Kafka and then try to bolt different services together. This is the wrong way around.

Given the stated goal is to build a highly scalable web-crawler, you should think of Flink as the first order component in your architecture, and seek to avoid use of Kafka/external services wherever possible.

Note that many stream processing applications do not even need Kafka. You should really only use Kafka if you need CQRS, and even then you only need 1 topic per schema type.

Generally speaking, you should try to do as much processing as possible within Flink so you can leverage Flink’s semantics and optimisations. This will significantly reduce complexity, latency, and remove bottlenecks.

Some thoughts/suggestions:

1. Why do you need two Flink jobs? The application is pretty simple .... if you use a single job you can set the parallelism for each Flink operator independently and let Flink manage back pressure across the entire execution graph.

2. Why do you need four Kafka topics?

- The first two should be rolled into a single “URL” topic (assuming they both use the same schema).

- The third topic can be eliminated if you use a single Flink job. There is no point writing everything to disk, only to read it back in again.

- The last topic can be eliminated if you just write directly from Flink to the sink(s), bypassing Kafka. This can be done with exactly-once guarantees.

3. Interacting with an external service like the Page Scraper API is going to severely limit the throughput of your whole Flink job. Flink will spend a lot of time waiting for the external service to reply, even if you use Flink’s Async I/O.

- Flink is extremely efficient at executing business logic. Almost everything is done in memory and you can distribute the computation using FlatMap functions. This virtually eliminates network I/O.

- If your external service is stateful, you also complicate fault tolerance and recovery, as you need to synchronise Flink’s state with the external service on recovery.

Personally, I would have a single Kafka topic for reading URLs, and a single Flink job for all the processing (including page scraping) that connects directly to your sinks using Flink’s TwoPhaseCommitFunction to guarantee exactly once processing.

You could seed the startup URLs using Flink’s static configuration, or make a call to a non-Kafka persistence store. The only real reason I see to use a Kafka topic is for the cyclical flow of new URLs, as Flink’s iterative streams do not yet guarantee exactly once semantics. Once that is resolved, the cyclical flow can be managed entirely within Flink.

One final thought – Kafka has recently been rebranded as a stream processing platform. In reality, KStreams offers you a way to do “micro-services with Kafka topics”. It is not really a stream processing platform and does not scale or distribute the computation anywhere near as well as Flink.