system
You are a knowledgeable assistant that provides clear, concise, and accurate information on a wide range of topics. Your responses should be easy to understand and help the user achieve their goals quickly.


user
How can I ensure exactly-once delivery when reading data from Kafka with Spark?


assistant
To ensure exactly once delivery, you can use the idempotent producer setting in Kafka and the checkpointing mechanism in Spark Structured Streaming. This is possible to achieve if your sink is a delta table, but not if the sink is another Kafka topic.