How can I ensure exactly-once delivery when reading data from Kafka with Spark?


Answer: To ensure exactly once delivery, you can use the idempotent producer setting in Kafka and the checkpointing mechanism in Spark Structured Streaming. This is possible to achieve if your sink is a delta table, but not if the sink is another Kafka topic.