Repository: apache/pulsar
Pull request #24020: [improve][client] PIP-409: support producer configuration for retry/dead letter topic producer
Author: login: thetumbled
Labels: name: area/client
name: type/feature
description: The PR added a new feature or issue requested a new feature
name: doc-required
description: Your PR changes impact docs and you will update later.
name: PIP
name: cherry-picked/branch-4.0
name: release/4.0.4
Created: 2025-02-25T03:40:05Z | Merged: 2025-03-06T08:41:34Z

PIP: #24022 
Fixes #19463
Fixes #21954

### Motivation

Currently, we don't support configure the producer of retry/dead letter topic. But enable the chunk message feature
and disable the batch configuration in hard code, which can't handle many situations. For example, when the throughput 
of message of retry topic become considerable, the resource consumed by the un-batched messages is pretty large. 
There is no reason that we disable the batch message feature.

For better control for the retry/dead letter topic feature, we can support configuration for the producer of 
retry/dead letter topic.


### Modifications

Support configuration for the producer of retry/dead letter topic.

### Verifying this change

- [x] Make sure that the change passes the CI checks.

*(Please pick either of the following options)*

This change added tests and can be verified as follows:

*(example:)*
  - *Added integration tests for end-to-end deployment with large payloads (10MB)*
  - *Extended integration test for recovery after broker failure*

### Does this pull request potentially affect one of the following parts:

<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->

*If the box was checked, please highlight the changes*

- [ ] Dependencies (add or upgrade a dependency)
- [ ] The public API
- [ ] The schema
- [ ] The default values of configurations
- [ ] The threading model
- [ ] The binary protocol
- [ ] The REST endpoints
- [ ] The admin CLI options
- [ ] The metrics
- [ ] Anything that affects deployment

### Documentation

<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->

- [ ] `doc` <!-- Your PR contains doc changes. -->
- [x] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->

### Matching PR in forked repository

PR in forked repository: https://github.com/thetumbled/pulsar/pull/71


Linked issue #19463: Support configuring batch message behavior in Java client's DeadLetterPolicy / DLQ feature
### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.


### Motivation

Pulsar's DLQ feature is mainly implemented on the client side. 
It is not currently possible to disable batch messages that might get produced to the dead letter topic when 2 messages happen to get delivered to the dead letter topic about the same time.

For some applications and environments, it could be desired to be able to configure this behavior. The broker configuration doesn't set `acknowledgmentAtBatchIndexLevelEnabled=true` by default which would be necessary for using the [PIP-54 Support acknowledgment at the batch index level](https://pulsar.apache.org/blog/2020/06/18/Apache-Pulsar-2-6-0/#pip-54-support-acknowledgment-at-the-batch-index-level) feature needed for individual message acknowledgement within batchmessages. Re-trying messages within the DLQ topic might fail and without PIP-54, all messages in the batch message would get redelived when even one of the messages in the batch message isn't successfully acknowledged. 

To support retaining messages as non-batch messages, there should be a setting in DeadLetterPolicy to control this.




### Solution

Add a setting to DeadLetterPolicy for controlling batch message behavior. 

Setting `DeadLetterPolicy.BatchBehavior` (name could be adjusted later)
```java
enum BatchBehavior {
    ENABLED,
    DISABLED,
    ENABLE_FOR_BATCHED_MESSAGES
}
```

In addition to disabling/enabling a batch, there could be a third option `ENABLE_FOR_BATCHED_MESSAGES` which would enable batching only when the incoming message is already a batch message.






### Alternatives

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!
Linked issue comments: user: login: zzzming
created_at: 2023-02-08T13:52:14Z
body: Should this policy apply to the Retry queue too?
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2023-03-11T01:56:40Z
body: The issue had no activity for 30 days, mark with Stale label.
author_association: NONE
user: login: lhotari
created_at: 2025-02-28T09:03:39Z
body: Update: In #21048, the logic was changed so that batching is always disabled and chunking is always enabled.
author_association: MEMBER
user: login: lhotari
created_at: 2025-02-28T09:05:00Z
body: There's also #20959 as a proposal 
author_association: MEMBER
user: login: lhotari
created_at: 2025-02-28T14:27:36Z
body: This issue will be addressed with #24020
author_association: MEMBER

Diff:
diff --git a/pip/pip-409.md b/pip/pip-409.md
index bc91c27610b85..b9882360465cf 100644
--- a/pip/pip-409.md
+++ b/pip/pip-409.md
@@ -33,25 +33,90 @@ retry/dead letter topic.
 
 ## Design & Implementation Details
 
-- Add two new configurations in `DeadLetterPolicy`:
+- Add `DeadLetterProducerBuilderContext` and `DeadLetterProducerBuilderCustomizer` interface:
+```java
+public interface DeadLetterProducerBuilderContext {
+    /**
+     * Returns the default name of topic for the dead letter or retry letter producer. This topic name is used
+     * unless the DeadLetterProducerBuilderCustomizer overrides it.
+     *
+     * @return a {@code String} representing the input topic name
+     */
+    String getDefaultTopicName();
+
+    /**
+     * Returns the name of the input topic for which the dead letter or retry letter producer is being configured.
+     *
+     * @return a {@code String} representing the input topic name
+     */
+    String getInputTopicName();
+
+    /**
+     * Returns the name of the subscription for which the dead letter or retry letter producer is being configured.
+     *
+     * @return a {@code String} representing the subscription name
+     */
+    String getInputTopicSubscriptionName();
+
+    /**
+     * Returns the name of the consumer for which the dead letter or
+     * retry letter producer is being configured.
+     * @return a {@code String} representing the consumer name
+     */
+    String getInputTopicConsumerName();
+}
+
+public interface DeadLetterProducerBuilderCustomizer {
+    /**
+     * Customize the given producer builder with settings specific to the topic context provided.
+     *
+     * @param context         the context containing information about the input topic and the subscription
+     * @param producerBuilder the producer builder instance to be customized
+     */
+    void customize(DeadLetterProducerBuilderContext context, ProducerBuilder<byte[]> producerBuilder);
+}
+```
+
+- Add two fields in `DeadLetterPolicy`:
 ```java
 public class DeadLetterPolicy implements Serializable {
-  /**
-   * Function to build the producer for the retry letter topic.
-   * The input parameter is the topic name.
-   */
-  private Function<String, ProducerBuilder<byte[]>> retryLetterProducerBuilder;
-
-  /**
-   * Function to build the producer for the dead letter topic.
-   * The input parameter is the topic name.
-   */
-  private Function<String, ProducerBuilder<byte[]>> deadLetterProducerBuilder;
+    /**
+     * Customizer for configuring the producer builder for the retry letter topic.
+     *
+     * <p>This field holds a function that allows the caller to customize the producer builder
+     * settings for the retry letter topic before the producer is created. The customization logic
+     * can use the provided context (which includes input topic and subscription details) to adjust
+     * configurations such as timeouts, batching, or message routing.
+     */
+    private DeadLetterProducerBuilderCustomizer retryLetterProducerBuilderCustomizer;
+    /**
+     * Customizer for configuring the producer builder for the dead letter topic.
+     *
+     * <p>This field holds a function that allows the caller to customize the producer builder
+     * settings for the dead letter topic before the producer is created. Using the provided context,
+     * implementations can perform specific adjustments that ensure the dead letter queue operates
+     * with the appropriate configurations tailored for handling undeliverable messages.
+     */
+    private DeadLetterProducerBuilderCustomizer deadLetterProducerBuilderCustomizer;
 }
 ```
 
-- use the `retryLetterProducerBuilder` to build the producer for retry topic, and use the
-`deadLetterProducerBuilder` to build the producer for dead letter topic.
+- use the `DeadLetterProducerBuilderCustomizer` to customize the producer of retry/dead letter topic like this:
+```java
+// enable batch
+DeadLetterProducerBuilderCustomizer producerBuilderCustomizer = (context, producerBuilder) -> {
+        producerBuilder.enableBatching(true);
+};
+Consumer<byte[]> consumer = pulsarClient.newConsumer(Schema.BYTES)
+        .topic(topic)
+        .subscriptionName(subscriptionName)
+        .enableRetry(true)
+        .deadLetterPolicy(DeadLetterPolicy.builder()
+                .maxRedeliverCount(maxRedeliveryCount)
+                .retryLetterProducerBuilderCustomizer(producerBuilderCustomizer)
+                .build())
+        .subscribe();
+```
 
 
 # Backward & Forward Compatibility
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/DeadLetterTopicTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/DeadLetterTopicTest.java
index ab26949c04fc6..0fb70c74172c8 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/DeadLetterTopicTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/DeadLetterTopicTest.java
@@ -1052,6 +1052,94 @@ public void testDeadLetterTopicWithInitialSubscription() throws Exception {
         consumer.close();
     }
 
+    @Test()
+    public void testDeadLetterTopicWithProducerBuilder() throws Exception {
+        final String topic = "persistent://my-property/my-ns/dead-letter-topic-with-producer-builder";
+        final int maxRedeliveryCount = 2;
+        final int sendMessages = 100;
+
+        // enable batch
+        DeadLetterProducerBuilderCustomizer producerBuilderCustomizer = (context, producerBuilder) -> {
+            producerBuilder.enableBatching(true);
+        };
+        String subscriptionName = "my-subscription";
+        String subscriptionNameDLQ = "my-subscription-DLQ";
+        Consumer<byte[]> consumer = pulsarClient.newConsumer(Schema.BYTES)
+                .topic(topic)
+                .subscriptionName(subscriptionName)
+                .subscriptionType(SubscriptionType.Shared)
+                .enableRetry(true)
+                .deadLetterPolicy(DeadLetterPolicy.builder()
+                        .maxRedeliverCount(maxRedeliveryCount)
+                        .initialSubscriptionName(subscriptionNameDLQ)
+                        .deadLetterProducerBuilderCustomizer(producerBuilderCustomizer)
+                        .retryLetterProducerBuilderCustomizer(producerBuilderCustomizer)
+                        .build())
+                .receiverQueueSize(100)
+                .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
+                .subscribe();
+
+        @Cleanup
+        PulsarClient newPulsarClient = newPulsarClient(lookupUrl.toString(), 0);// Creates new client connection
+        Consumer<byte[]> deadLetterConsumer = newPulsarClient.newConsumer(Schema.BYTES)
+                .topic(topic + "-" + subscriptionName + "-DLQ")
+                .subscriptionName(subscriptionNameDLQ)
+                .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
+                .subscribe();
+
+        Producer<byte[]> producer = pulsarClient.newProducer(Schema.BYTES)
+                .topic(topic)
+                .enableBatching(true)
+                .create();
+
+        Map<Integer, String> messageContent = new HashMap<>();
+        for (int i = 0; i < sendMessages; i++) {
+            String data = String.format("Hello Pulsar [%d]", i);
+            producer.newMessage().key(String.valueOf(i)).value(data.getBytes()).send();
+            messageContent.put(i, data);
+        }
+        producer.close();
+
+        int totalReceived = 0;
+        do {
+            Message<byte[]> message = consumer.receive(5, TimeUnit.SECONDS);
+            assertNotNull(message, "The consumer should be able to receive messages.");
+            log.info("consumer received message : {}", message.getMessageId());
+            totalReceived++;
+            consumer.reconsumeLater(message, 1, TimeUnit.SECONDS);
+        } while (totalReceived < sendMessages * (maxRedeliveryCount + 1));
+
+        int totalInDeadLetter = 0;
+        do {
+            Message message = deadLetterConsumer.receive(5, TimeUnit.SECONDS);
+            assertNotNull(message, "the deadLetterConsumer should receive messages.");
+            assertEquals(new String(message.getData()), messageContent.get(Integer.parseInt(message.getKey())));
+            messageContent.remove(Integer.parseInt(message.getKey()));
+            log.info("dead letter consumer received message : {}", message.getMessageId());
+            deadLetterConsumer.acknowledge(message);
+            totalInDeadLetter++;
+        } while (totalInDeadLetter < sendMessages);
+        assertTrue(messageContent.isEmpty());
+
+        deadLetterConsumer.close();
+        consumer.close();
+
+        Consumer<byte[]> checkConsumer = this.pulsarClient.newConsumer(Schema.BYTES)
+                .topic(topic)
+                .subscriptionName("my-subscription")
+                .subscriptionType(SubscriptionType.Shared)
+                .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
+                .subscribe();
+
+        Message<byte[]> checkMessage = checkConsumer.receive(3, TimeUnit.SECONDS);
+        if (checkMessage != null) {
+            log.info("check consumer received message : {} {}", checkMessage.getMessageId(), new String(checkMessage.getData()));
+        }
+        assertNull(checkMessage);
+
+        checkConsumer.close();
+    }
+
     private CompletableFuture<Void> consumerReceiveForDLQ(Consumer<byte[]> consumer, AtomicInteger totalReceived,
                                                           int sendMessages, int maxRedeliveryCount) {
         return CompletableFuture.runAsync(() -> {
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/RetryTopicTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/RetryTopicTest.java
index 2b897760b6f00..b87bc27ed7270 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/RetryTopicTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/RetryTopicTest.java
@@ -136,6 +136,83 @@ public void testRetryTopic() throws Exception {
         checkConsumer.close();
     }
 
+    @Test
+    public void testRetryTopicWithProducerBuilder() throws Exception {
+        final String topic = "persistent://my-property/my-ns/retry-topic-with-producer-builder";
+        final int maxRedeliveryCount = 2;
+        final int sendMessages = 100;
+
+        // enable batch
+        DeadLetterProducerBuilderCustomizer producerBuilderCustomizer = (context, producerBuilder) -> {
+            producerBuilder.enableBatching(true);
+        };
+        String subscriptionName = "my-subscription";
+        String subscriptionNameDLQ = "my-subscription-DLQ";
+        Consumer<byte[]> consumer = pulsarClient.newConsumer(Schema.BYTES)
+                .topic(topic)
+                .subscriptionName(subscriptionName)
+                .subscriptionType(SubscriptionType.Shared)
+                .enableRetry(true)
+                .deadLetterPolicy(DeadLetterPolicy.builder()
+                        .maxRedeliverCount(maxRedeliveryCount)
+                        .retryLetterProducerBuilderCustomizer(producerBuilderCustomizer)
+                        .build())
+                .receiverQueueSize(100)
+                .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
+                .subscribe();
+
+        @Cleanup
+        PulsarClient newPulsarClient = newPulsarClient(lookupUrl.toString(), 0);// Creates new client connection
+        Consumer<byte[]> deadLetterConsumer = newPulsarClient.newConsumer(Schema.BYTES)
+                .topic(topic + "-" + subscriptionName + "-DLQ")
+                .subscriptionName(subscriptionNameDLQ)
+                .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
+                .subscribe();
+
+        Producer<byte[]> producer = pulsarClient.newProducer(Schema.BYTES)
+                .topic(topic)
+                .create();
+
+        for (int i = 0; i < sendMessages; i++) {
+            producer.send(String.format("Hello Pulsar [%d]", i).getBytes());
+        }
+        producer.close();
+
+        int totalReceived = 0;
+        do {
+            Message<byte[]> message = consumer.receive();
+            log.info("consumer received message : {} {}", message.getMessageId(), new String(message.getData()));
+            consumer.reconsumeLater(message, 1, TimeUnit.SECONDS);
+            totalReceived++;
+        } while (totalReceived < sendMessages * (maxRedeliveryCount + 1));
+
+        int totalInDeadLetter = 0;
+        do {
+            Message<byte[]> message = deadLetterConsumer.receive();
+            log.info("dead letter consumer received message : {} {}", message.getMessageId(), new String(message.getData()));
+            deadLetterConsumer.acknowledge(message);
+            totalInDeadLetter++;
+        } while (totalInDeadLetter < sendMessages);
+
+        deadLetterConsumer.close();
+        consumer.close();
+
+        Consumer<byte[]> checkConsumer = this.pulsarClient.newConsumer(Schema.BYTES)
+                .topic(topic)
+                .subscriptionName(subscriptionName)
+                .subscriptionType(SubscriptionType.Shared)
+                .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
+                .subscribe();
+
+        Message<byte[]> checkMessage = checkConsumer.receive(3, TimeUnit.SECONDS);
+        if (checkMessage != null) {
+            log.info("check consumer received message : {} {}", checkMessage.getMessageId(), new String(checkMessage.getData()));
+        }
+        assertNull(checkMessage);
+
+        checkConsumer.close();
+    }
+
     /**
      * Retry topic feature relies on the delay queue feature when consumer produce a delayed message
      * to the retry topic. The delay queue feature is only supported in shared and key-shared subscription type.
diff --git a/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/DeadLetterPolicy.java b/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/DeadLetterPolicy.java
index c2a172666b01d..f45a15437da6c 100644
--- a/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/DeadLetterPolicy.java
+++ b/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/DeadLetterPolicy.java
@@ -62,4 +62,24 @@ public class DeadLetterPolicy implements Serializable {
      * to be created.
      */
     private String initialSubscriptionName;
+
+    /**
+     * Customizer for configuring the producer builder for the retry letter topic.
+     *
+     * <p>This field holds a function that allows the caller to customize the producer builder
+     * settings for the retry letter topic before the producer is created. The customization logic
+     * can use the provided context (which includes input topic and subscription details) to adjust
+     * configurations such as timeouts, batching, or message routing.
+     */
+    private DeadLetterProducerBuilderCustomizer retryLetterProducerBuilderCustomizer;
+
+    /**
+     * Customizer for configuring the producer builder for the dead letter topic.
+     *
+     * <p>This field holds a function that allows the caller to customize the producer builder
+     * settings for the dead letter topic before the producer is created. Using the provided context,
+     * implementations can perform specific adjustments that ensure the dead letter queue operates
+     * with the appropriate configurations tailored for handling undeliverable messages.
+     */
+    private DeadLetterProducerBuilderCustomizer deadLetterProducerBuilderCustomizer;
 }
diff --git a/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/DeadLetterProducerBuilderContext.java b/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/DeadLetterProducerBuilderContext.java
new file mode 100644
index 0000000000000..6e74f8c5ca005
--- /dev/null
+++ b/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/DeadLetterProducerBuilderContext.java
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.api;
+
+/**
+ * Provides context information required for customizing a producer builder.
+ *
+ * <p>This interface supplies relevant details such as the name of the input topic and associated subscription name.
+ * This contextual information helps in correctly configuring the producer for the appropriate topic.
+ */
+public interface DeadLetterProducerBuilderContext {
+    /**
+     * Returns the default name of topic for the dead letter or retry letter producer. This topic name is used
+     * unless the ProducerBuilderCustomizer overrides it.
+     *
+     * @return a {@code String} representing the input topic name
+     */
+    String getDefaultTopicName();
+
+    /**
+     * Returns the name of the input topic for which the dead letter or retry letter producer is being configured.
+     *
+     * @return a {@code String} representing the input topic name
+     */
+    String getInputTopicName();
+
+    /**
+     * Returns the name of the subscription for which the dead letter or retry letter producer is being configured.
+     *
+     * @return a {@code String} representing the subscription name
+     */
+    String getInputTopicSubscriptionName();
+
+    /**
+     * Returns the name of the consumer for which the dead letter or
+     * retry letter producer is being configured.
+     * @return a {@code String} representing the consumer name
+     */
+    String getInputTopicConsumerName();
+}
+
+
diff --git a/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/DeadLetterProducerBuilderCustomizer.java b/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/DeadLetterProducerBuilderCustomizer.java
new file mode 100644
index 0000000000000..85670132a8940
--- /dev/null
+++ b/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/DeadLetterProducerBuilderCustomizer.java
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.api;
+
+/**
+ * Functional interface for customizing a producer builder for a specific topic.
+ *
+ * <p>This interface allows for customizing the producer builder configuration for either the retry letter topic
+ * or the dead letter topic. The customization might include setting producer properties such as batching, timeouts,
+ * or any other producer-specific configuration.
+ *
+ * @see DeadLetterProducerBuilderContext
+ */
+public interface DeadLetterProducerBuilderCustomizer {
+    /**
+     * Customize the given producer builder with settings specific to the topic context provided.
+     *
+     * @param context         the context containing information about the input topic and the subscription
+     * @param producerBuilder the producer builder instance to be customized
+     */
+    void customize(DeadLetterProducerBuilderContext context, ProducerBuilder<byte[]> producerBuilder);
+}
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
index 6f2ad9152d3f6..1e9917f8936a4 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
@@ -77,12 +77,15 @@
 import org.apache.pulsar.client.api.Consumer;
 import org.apache.pulsar.client.api.ConsumerCryptoFailureAction;
 import org.apache.pulsar.client.api.DeadLetterPolicy;
+import org.apache.pulsar.client.api.DeadLetterProducerBuilderContext;
+import org.apache.pulsar.client.api.DeadLetterProducerBuilderCustomizer;
 import org.apache.pulsar.client.api.Message;
 import org.apache.pulsar.client.api.MessageCrypto;
 import org.apache.pulsar.client.api.MessageId;
 import org.apache.pulsar.client.api.MessageIdAdv;
 import org.apache.pulsar.client.api.Messages;
 import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.ProducerBuilder;
 import org.apache.pulsar.client.api.PulsarClientException;
 import org.apache.pulsar.client.api.PulsarClientException.TopicDoesNotExistException;
 import org.apache.pulsar.client.api.Schema;
@@ -2286,6 +2289,16 @@ private CompletableFuture<Boolean> processPossibleToDLQ(MessageIdAdv messageId)
         return result;
     }
 
+    private void customizeDeadLetterProducerBuilder(DeadLetterProducerBuilderCustomizer customizer,
+                                                    String deadLetterOrRetryLetterTopic,
+                                                    ProducerBuilder<byte[]> builder) {
+        if (customizer != null) {
+            DeadLetterProducerBuilderContext context = new DeadLetterProducerBuilderContextImpl(
+                    deadLetterOrRetryLetterTopic, topic, subscription, consumerName);
+            customizer.customize(context, builder);
+        }
+    }
+
     private CompletableFuture<Producer<byte[]>> initDeadLetterProducerIfNeeded() {
         CompletableFuture<Producer<byte[]>> p = deadLetterProducer;
         if (p == null || p.isCompletedExceptionally()) {
@@ -2294,7 +2307,7 @@ private CompletableFuture<Producer<byte[]>> initDeadLetterProducerIfNeeded() {
                 p = deadLetterProducer;
                 if (p == null || p.isCompletedExceptionally()) {
                     p = createProducerWithBackOff(() -> {
-                        CompletableFuture<Producer<byte[]>> newProducer =
+                        ProducerBuilder<byte[]> builder =
                                 ((ProducerBuilderImpl<byte[]>) client.newProducer(Schema.AUTO_PRODUCE_BYTES(schema)))
                                         .initialSubscriptionName(this.deadLetterPolicy.getInitialSubscriptionName())
                                         .topic(this.deadLetterPolicy.getDeadLetterTopic())
@@ -2303,8 +2316,10 @@ private CompletableFuture<Producer<byte[]>> initDeadLetterProducerIfNeeded() {
                                                         this.consumerName, RandomStringUtils.randomAlphanumeric(5)))
                                         .blockIfQueueFull(false)
                                         .enableBatching(false)
-                                        .enableChunking(true)
-                                        .createAsync();
+                                        .enableChunking(true);
+                        customizeDeadLetterProducerBuilder(deadLetterPolicy.getDeadLetterProducerBuilderCustomizer(),
+                                deadLetterPolicy.getDeadLetterTopic(), builder);
+                        CompletableFuture<Producer<byte[]>> newProducer = builder.createAsync();
                         newProducer.whenComplete((producer, ex) -> {
                             if (ex != null) {
                                 log.error("[{}] [{}] [{}] Failed to create dead letter producer for topic {}",
@@ -2364,13 +2379,15 @@ private CompletableFuture<Producer<byte[]>> initRetryLetterProducerIfNeeded() {
                 p = retryLetterProducer;
                 if (p == null || p.isCompletedExceptionally()) {
                     p = createProducerWithBackOff(() -> {
-                        CompletableFuture<Producer<byte[]>> newProducer = client
+                        ProducerBuilder<byte[]> builder = client
                                 .newProducer(Schema.AUTO_PRODUCE_BYTES(schema))
                                 .topic(this.deadLetterPolicy.getRetryLetterTopic())
                                 .enableBatching(false)
                                 .enableChunking(true)
-                                .blockIfQueueFull(false)
-                                .createAsync();
+                                .blockIfQueueFull(false);
+                        customizeDeadLetterProducerBuilder(deadLetterPolicy.getRetryLetterProducerBuilderCustomizer(),
+                                deadLetterPolicy.getRetryLetterTopic(), builder);
+                        CompletableFuture<Producer<byte[]>> newProducer = builder.createAsync();
                         newProducer.whenComplete((producer, ex) -> {
                             if (ex != null) {
                                 log.error("[{}] [{}] [{}] Failed to create retry letter producer for topic {}",
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/DeadLetterProducerBuilderContextImpl.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/DeadLetterProducerBuilderContextImpl.java
new file mode 100644
index 0000000000000..0df67561c93e2
--- /dev/null
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/DeadLetterProducerBuilderContextImpl.java
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl;
+
+import lombok.AllArgsConstructor;
+import org.apache.pulsar.client.api.DeadLetterProducerBuilderContext;
+
+@AllArgsConstructor
+class DeadLetterProducerBuilderContextImpl implements DeadLetterProducerBuilderContext {
+    private final String defaultTopicName;
+    private final String inputTopicName;
+    private final String inputTopicSubscriptionName;
+    private final String inputTopicConsumerName;
+
+    @Override
+    public String getDefaultTopicName() {
+        return defaultTopicName;
+    }
+
+    @Override
+    public String getInputTopicName() {
+        return inputTopicName;
+    }
+
+    @Override
+    public String getInputTopicSubscriptionName() {
+        return inputTopicSubscriptionName;
+    }
+
+    @Override
+    public String getInputTopicConsumerName() {
+        return inputTopicConsumerName;
+    }
+}


Reviews:
user: login: lhotari
body: Instead of using java.util.function.Function, I'd suggest using a richer interface so that more context parameters can be added besides the topic name. This pattern would also allow adding more context parameters later without breaking the API in backwards incompatible ways.
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-02-28T14:22:36Z
user: login: thetumbled
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T02:42:01Z
user: login: BewareMyPower
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T03:41:41Z
user: login: nodece
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T03:47:13Z
user: login: lhotari
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T06:04:32Z
user: login: BewareMyPower
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T07:03:37Z
user: login: thetumbled
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T07:28:04Z
user: login: nodece
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T07:46:15Z
user: login: BewareMyPower
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T07:53:37Z
user: login: thetumbled
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T07:57:13Z
user: login: lhotari
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T08:09:09Z
user: login: thetumbled
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T08:19:08Z
user: login: nodece
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T09:03:35Z
user: login: BewareMyPower
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T10:11:01Z
user: login: BewareMyPower
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T10:17:17Z
user: login: thetumbled
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T06:53:02Z
user: login: lhotari
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T07:17:12Z
user: login: lhotari
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T07:18:26Z
user: login: lhotari
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T07:19:13Z
user: login: lhotari
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T07:21:12Z
user: login: thetumbled
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T07:35:37Z
user: login: lhotari
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T08:47:31Z
user: login: thetumbled
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T08:54:18Z
user: login: lhotari
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T09:36:40Z
user: login: lhotari
body: LGTM, good work @thetumbled . Great to see this finally solving many use cases.
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-05T09:38:04Z
user: login: lhotari
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T09:46:20Z
user: login: lhotari
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T09:50:32Z
user: login: lhotari
body: One remaining issue about naming. Commented added.
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-03-05T09:50:54Z
user: login: thetumbled
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T09:55:08Z
user: login: thetumbled
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T09:59:15Z
user: login: lhotari
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T10:17:25Z
user: login: lhotari
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T10:19:01Z
user: login: thetumbled
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-06T02:19:30Z
user: login: thetumbled
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-06T02:27:09Z
user: login: lhotari
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-06T04:43:14Z

Review comments:
diff_hunk: @@ -62,4 +63,16 @@ public class DeadLetterPolicy implements Serializable {
      * to be created.
      */
     private String initialSubscriptionName;
+
+    /**
+     * Function to build the producer for the retry letter topic.
+     * The input parameter is the topic name.
+     */
+    private Function<String, ProducerBuilder<byte[]>> retryLetterProducerBuilder;
+
+    /**
+     * Function to build the producer for the dead letter topic.
+     * The input parameter is the topic name.
+     */
+    private Function<String, ProducerBuilder<byte[]>> deadLetterProducerBuilder;
path: pulsar-client-api/src/main/java/org/apache/pulsar/client/api/DeadLetterPolicy.java
user: login: lhotari
body: Instead of using java.util.function.Function, I'd suggest using a richer interface so that more context parameters can be added besides the topic name. 

```suggestion
    /**
     * Customizer for configuring the producer builder for the retry letter topic.
     *
     * <p>This field holds a function that allows the caller to customize the producer builder
     * settings for the retry letter topic before the producer is created. The customization logic
     * can use the provided context (which includes input topic and subscription details) to adjust
     * configurations such as timeouts, batching, or message routing.
     */
    private ProducerBuilderCustomizer retryLetterProducerBuilder;

    /**
     * Customizer for configuring the producer builder for the dead letter topic.
     *
     * <p>This field holds a function that allows the caller to customize the producer builder
     * settings for the dead letter topic before the producer is created. Using the provided context,
     * implementations can perform specific adjustments that ensure the dead letter queue operates
     * with the appropriate configurations tailored for handling undeliverable messages.
     */
    private ProducerBuilderCustomizer deadLetterProducerBuilder;


    /**
     * Functional interface for customizing a producer builder for a specific topic.
     *
     * <p>This interface allows for customizing the producer builder configuration for either the retry letter topic
     * or the dead letter topic. The customization might include setting producer properties such as batching, timeouts,
     * or any other producer-specific configuration.
     *
     * @see ProducerBuilderContext
     */
    public interface ProducerBuilderCustomizer {
        /**
         * Customize the given producer builder with settings specific to the topic context provided.
         *
         * @param context         the context containing information about the input topic and the subscription
         * @param producerBuilder the producer builder instance to be customized
         */
        void customize(ProducerBuilderContext context, ProducerBuilder<byte[]> producerBuilder);
    }

    /**
     * Provides context information required for customizing a producer builder.
     *
     * <p>This interface supplies relevant details such as the name of the input topic and associated subscription name.
     * This contextual information helps in correctly configuring the producer for the appropriate topic.
     */
    public interface ProducerBuilderContext {
        /**
         * Returns the default name of topic for the dead letter or retry letter producer. This topic name is used
         * unless the ProducerBuilderCustomizer overrides it.
         *
         * @return a {@code String} representing the input topic name
         */
        String getDefaultTopicName();
        
        /**
         * Returns the name of the input topic for which the dead letter or retry letter producer is being configured.
         *
         * @return a {@code String} representing the input topic name
         */
        String getInputTopicName();

        /**
         * Returns the name of the subscription for which the dead letter or retry letter producer is being configured.
         *
         * @return a {@code String} representing the subscription name
         */
        String getInputTopicSubscriptionName();
    }
```
created_at: 2025-02-28T14:21:29Z
author_association: MEMBER
diff_hunk: @@ -62,4 +63,16 @@ public class DeadLetterPolicy implements Serializable {
      * to be created.
      */
     private String initialSubscriptionName;
+
+    /**
+     * Function to build the producer for the retry letter topic.
+     * The input parameter is the topic name.
+     */
+    private Function<String, ProducerBuilder<byte[]>> retryLetterProducerBuilder;
+
+    /**
+     * Function to build the producer for the dead letter topic.
+     * The input parameter is the topic name.
+     */
+    private Function<String, ProducerBuilder<byte[]>> deadLetterProducerBuilder;
path: pulsar-client-api/src/main/java/org/apache/pulsar/client/api/DeadLetterPolicy.java
user: login: thetumbled
body: Adding `ProducerBuilderContext` is good.
But `ProducerBuilderCustomizer` is unnecessary, and the grammar is incorrect.
`void customize(ProducerBuilderContext context, ProducerBuilder<byte[]> producerBuilder);`
should be changed to 
`ProducerBuilder<byte[]> customize(ProducerBuilderContext context);`

I prefer to use `Function<ProducerBuilderContext, ProducerBuilder<byte[]>>`, which reduce the burden of understanding new interface `ProducerBuilderCustomizer`. WDYT, @lhotari @BewareMyPower @nodece @dao-jun
created_at: 2025-03-03T02:42:01Z
author_association: MEMBER
diff_hunk: @@ -62,4 +63,16 @@ public class DeadLetterPolicy implements Serializable {
      * to be created.
      */
     private String initialSubscriptionName;
+
+    /**
+     * Function to build the producer for the retry letter topic.
+     * The input parameter is the topic name.
+     */
+    private Function<String, ProducerBuilder<byte[]>> retryLetterProducerBuilder;
+
+    /**
+     * Function to build the producer for the dead letter topic.
+     * The input parameter is the topic name.
+     */
+    private Function<String, ProducerBuilder<byte[]>> deadLetterProducerBuilder;
path: pulsar-client-api/src/main/java/org/apache/pulsar/client/api/DeadLetterPolicy.java
user: login: BewareMyPower
body: >  so that more context parameters can be added besides the topic name.

It's a good point.

I'm also +1 to `Function<ProducerBuilderContext, ProducerBuilder<byte[]>>`.
created_at: 2025-03-03T03:41:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -62,4 +63,16 @@ public class DeadLetterPolicy implements Serializable {
      * to be created.
      */
     private String initialSubscriptionName;
+
+    /**
+     * Function to build the producer for the retry letter topic.
+     * The input parameter is the topic name.
+     */
+    private Function<String, ProducerBuilder<byte[]>> retryLetterProducerBuilder;
+
+    /**
+     * Function to build the producer for the dead letter topic.
+     * The input parameter is the topic name.
+     */
+    private Function<String, ProducerBuilder<byte[]>> deadLetterProducerBuilder;
path: pulsar-client-api/src/main/java/org/apache/pulsar/client/api/DeadLetterPolicy.java
user: login: nodece
body: > I'm also +1 to `Function<ProducerBuilderContext, ProducerBuilder<byte[]>>`.

It makes sense to me.


created_at: 2025-03-03T03:47:13Z
author_association: MEMBER
diff_hunk: @@ -62,4 +63,16 @@ public class DeadLetterPolicy implements Serializable {
      * to be created.
      */
     private String initialSubscriptionName;
+
+    /**
+     * Function to build the producer for the retry letter topic.
+     * The input parameter is the topic name.
+     */
+    private Function<String, ProducerBuilder<byte[]>> retryLetterProducerBuilder;
+
+    /**
+     * Function to build the producer for the dead letter topic.
+     * The input parameter is the topic name.
+     */
+    private Function<String, ProducerBuilder<byte[]>> deadLetterProducerBuilder;
path: pulsar-client-api/src/main/java/org/apache/pulsar/client/api/DeadLetterPolicy.java
user: login: lhotari
body: > Adding `ProducerBuilderContext` is good.
> But `ProducerBuilderCustomizer` is unnecessary, and the grammar is incorrect.
> `void customize(ProducerBuilderContext context, ProducerBuilder<byte[]> producerBuilder);`
> should be changed to 
> `ProducerBuilder<byte[]> customize(ProducerBuilderContext context);`
> 
> I prefer to use `Function<ProducerBuilderContext, ProducerBuilder<byte[]>>`, which reduce the burden of understanding new interface `ProducerBuilderCustomizer`. WDYT, @lhotari @BewareMyPower @nodece @dao-jun

The reason why both the context and builder are passed are due to the fact that the runtime will preconfigure the builder and the customizer will only override settings.

One possibility would be to also return a builder so that the builder instance could be completely switched.

This would be a way to support different use cases. If you'd replace the builder, you could even produce to a different cluster with a producer builder created from a different pulsar client.
created_at: 2025-03-03T06:04:32Z
author_association: MEMBER
diff_hunk: @@ -62,4 +63,16 @@ public class DeadLetterPolicy implements Serializable {
      * to be created.
      */
     private String initialSubscriptionName;
+
+    /**
+     * Function to build the producer for the retry letter topic.
+     * The input parameter is the topic name.
+     */
+    private Function<String, ProducerBuilder<byte[]>> retryLetterProducerBuilder;
+
+    /**
+     * Function to build the producer for the dead letter topic.
+     * The input parameter is the topic name.
+     */
+    private Function<String, ProducerBuilder<byte[]>> deadLetterProducerBuilder;
path: pulsar-client-api/src/main/java/org/apache/pulsar/client/api/DeadLetterPolicy.java
user: login: BewareMyPower
body: > One possibility would be to also return a builder so that the builder instance could be completely switched.

That makes sense to me. Passing a `ProducerBuilder` can keep the immutable fields (`client`) still immutable.

BTW, @thetumbled once the design is updated, please update the PIP document to make it consistent with the latest design
created_at: 2025-03-03T07:03:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -62,4 +63,16 @@ public class DeadLetterPolicy implements Serializable {
      * to be created.
      */
     private String initialSubscriptionName;
+
+    /**
+     * Function to build the producer for the retry letter topic.
+     * The input parameter is the topic name.
+     */
+    private Function<String, ProducerBuilder<byte[]>> retryLetterProducerBuilder;
+
+    /**
+     * Function to build the producer for the dead letter topic.
+     * The input parameter is the topic name.
+     */
+    private Function<String, ProducerBuilder<byte[]>> deadLetterProducerBuilder;
path: pulsar-client-api/src/main/java/org/apache/pulsar/client/api/DeadLetterPolicy.java
user: login: thetumbled
body: Agree, i just updated the implementation and pip document, PTAL again, thanks for your reviews!
created_at: 2025-03-03T07:28:04Z
author_association: MEMBER
diff_hunk: @@ -33,25 +33,88 @@ retry/dead letter topic.
 
 ## Design & Implementation Details
 
-- Add two new configurations in `DeadLetterPolicy`:
+- Add `ProducerBuilderContext` and `ProducerBuilderCustomizer` interface:
+```java
+public interface ProducerBuilderContext {
+    /**
+     * Returns the default name of topic for the dead letter or retry letter producer. This topic name is used
+     * unless the ProducerBuilderCustomizer overrides it.
+     *
+     * @return a {@code String} representing the input topic name
+     */
+    String getDefaultTopicName();
+
+    /**
+     * Returns the name of the input topic for which the dead letter or retry letter producer is being configured.
+     *
+     * @return a {@code String} representing the input topic name
+     */
+    String getInputTopicName();
+
+    /**
+     * Returns the name of the subscription for which the dead letter or retry letter producer is being configured.
+     *
+     * @return a {@code String} representing the subscription name
+     */
+    String getInputTopicSubscriptionName();
+}
+
+public interface ProducerBuilderCustomizer {
+    /**
+     * Customize the given producer builder with settings specific to the topic context provided.
+     *
+     * @param context         the context containing information about the input topic and the subscription
+     * @param producerBuilder the producer builder instance to be customized
+     */
+    void customize(ProducerBuilderContext context, ProducerBuilder<byte[]> producerBuilder);
path: pip/pip-409.md
user: login: nodece
body: ```suggestion
    ProducerBuilder<byte[]> customize(ProducerBuilderContext context, ProducerBuilder<byte[]> producerBuilder);
```

Is it better? The user can provide the different producer builder. 

created_at: 2025-03-03T07:46:15Z
author_association: MEMBER
diff_hunk: @@ -33,25 +33,88 @@ retry/dead letter topic.
 
 ## Design & Implementation Details
 
-- Add two new configurations in `DeadLetterPolicy`:
+- Add `ProducerBuilderContext` and `ProducerBuilderCustomizer` interface:
+```java
+public interface ProducerBuilderContext {
+    /**
+     * Returns the default name of topic for the dead letter or retry letter producer. This topic name is used
+     * unless the ProducerBuilderCustomizer overrides it.
+     *
+     * @return a {@code String} representing the input topic name
+     */
+    String getDefaultTopicName();
+
+    /**
+     * Returns the name of the input topic for which the dead letter or retry letter producer is being configured.
+     *
+     * @return a {@code String} representing the input topic name
+     */
+    String getInputTopicName();
+
+    /**
+     * Returns the name of the subscription for which the dead letter or retry letter producer is being configured.
+     *
+     * @return a {@code String} representing the subscription name
+     */
+    String getInputTopicSubscriptionName();
+}
+
+public interface ProducerBuilderCustomizer {
+    /**
+     * Customize the given producer builder with settings specific to the topic context provided.
+     *
+     * @param context         the context containing information about the input topic and the subscription
+     * @param producerBuilder the producer builder instance to be customized
+     */
+    void customize(ProducerBuilderContext context, ProducerBuilder<byte[]> producerBuilder);
path: pip/pip-409.md
user: login: BewareMyPower
body: No. I think passing `ProducerBuilder` as the argument is to prevent from returning a different producer builder.
created_at: 2025-03-03T07:53:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -33,25 +33,88 @@ retry/dead letter topic.
 
 ## Design & Implementation Details
 
-- Add two new configurations in `DeadLetterPolicy`:
+- Add `ProducerBuilderContext` and `ProducerBuilderCustomizer` interface:
+```java
+public interface ProducerBuilderContext {
+    /**
+     * Returns the default name of topic for the dead letter or retry letter producer. This topic name is used
+     * unless the ProducerBuilderCustomizer overrides it.
+     *
+     * @return a {@code String} representing the input topic name
+     */
+    String getDefaultTopicName();
+
+    /**
+     * Returns the name of the input topic for which the dead letter or retry letter producer is being configured.
+     *
+     * @return a {@code String} representing the input topic name
+     */
+    String getInputTopicName();
+
+    /**
+     * Returns the name of the subscription for which the dead letter or retry letter producer is being configured.
+     *
+     * @return a {@code String} representing the subscription name
+     */
+    String getInputTopicSubscriptionName();
+}
+
+public interface ProducerBuilderCustomizer {
+    /**
+     * Customize the given producer builder with settings specific to the topic context provided.
+     *
+     * @param context         the context containing information about the input topic and the subscription
+     * @param producerBuilder the producer builder instance to be customized
+     */
+    void customize(ProducerBuilderContext context, ProducerBuilder<byte[]> producerBuilder);
path: pip/pip-409.md
user: login: thetumbled
body: > No. I think passing ProducerBuilder as the argument is to prevent from returning a different producer builder.

I agree. The purpose of introducing interface `ProducerBuilderCustomizer` is to retrict the creation of `ProducerBuilder`. 
If we allow returning a different producer builder created by user, there is no need for `ProducerBuilderCustomizer`.

created_at: 2025-03-03T07:57:13Z
author_association: MEMBER
diff_hunk: @@ -33,25 +33,88 @@ retry/dead letter topic.
 
 ## Design & Implementation Details
 
-- Add two new configurations in `DeadLetterPolicy`:
+- Add `ProducerBuilderContext` and `ProducerBuilderCustomizer` interface:
+```java
+public interface ProducerBuilderContext {
+    /**
+     * Returns the default name of topic for the dead letter or retry letter producer. This topic name is used
+     * unless the ProducerBuilderCustomizer overrides it.
+     *
+     * @return a {@code String} representing the input topic name
+     */
+    String getDefaultTopicName();
+
+    /**
+     * Returns the name of the input topic for which the dead letter or retry letter producer is being configured.
+     *
+     * @return a {@code String} representing the input topic name
+     */
+    String getInputTopicName();
+
+    /**
+     * Returns the name of the subscription for which the dead letter or retry letter producer is being configured.
+     *
+     * @return a {@code String} representing the subscription name
+     */
+    String getInputTopicSubscriptionName();
+}
+
+public interface ProducerBuilderCustomizer {
+    /**
+     * Customize the given producer builder with settings specific to the topic context provided.
+     *
+     * @param context         the context containing information about the input topic and the subscription
+     * @param producerBuilder the producer builder instance to be customized
+     */
+    void customize(ProducerBuilderContext context, ProducerBuilder<byte[]> producerBuilder);
path: pip/pip-409.md
user: login: lhotari
body: > I agree. The purpose of introducing interface `ProducerBuilderCustomizer` is to retrict the creation of `ProducerBuilder`. 

If there's a need to restrict the creation, the return type should be `void` in that case. However, wouldn't it be a valid use case to use a different Pulsar client instance for the producer? In that case allowing to return a different instance should be made possible.


> If we allow returning a different producer builder created by user, there is no need for `ProducerBuilderCustomizer`.
> 

That's not the case. A user might want to rely on the defaults for all other setting than what they are overriding in the customizer.
created_at: 2025-03-03T08:09:09Z
author_association: MEMBER
diff_hunk: @@ -33,25 +33,88 @@ retry/dead letter topic.
 
 ## Design & Implementation Details
 
-- Add two new configurations in `DeadLetterPolicy`:
+- Add `ProducerBuilderContext` and `ProducerBuilderCustomizer` interface:
+```java
+public interface ProducerBuilderContext {
+    /**
+     * Returns the default name of topic for the dead letter or retry letter producer. This topic name is used
+     * unless the ProducerBuilderCustomizer overrides it.
+     *
+     * @return a {@code String} representing the input topic name
+     */
+    String getDefaultTopicName();
+
+    /**
+     * Returns the name of the input topic for which the dead letter or retry letter producer is being configured.
+     *
+     * @return a {@code String} representing the input topic name
+     */
+    String getInputTopicName();
+
+    /**
+     * Returns the name of the subscription for which the dead letter or retry letter producer is being configured.
+     *
+     * @return a {@code String} representing the subscription name
+     */
+    String getInputTopicSubscriptionName();
+}
+
+public interface ProducerBuilderCustomizer {
+    /**
+     * Customize the given producer builder with settings specific to the topic context provided.
+     *
+     * @param context         the context containing information about the input topic and the subscription
+     * @param producerBuilder the producer builder instance to be customized
+     */
+    void customize(ProducerBuilderContext context, ProducerBuilder<byte[]> producerBuilder);
path: pip/pip-409.md
user: login: thetumbled
body: I don't know which kind of senerio need to use a different Pulsar client. Take the example you mentioned above:

>  This would be a way to support different use cases. If you'd replace the builder, you could even produce to a different cluster with a producer builder created from a different pulsar client.

Producing the retry messages to another cluster will break the logical design of retry topic. Let's explain it in more detail. User use `client1` to create consumer for `inputTopic` and `retryTopic`, both of them belonging to `cluster1`. But user use `client2`  belonging to `cluster2` to create the producer for `retryTopic`, resulting the message produced to `cluster2`. However, the consumer for `retryTopic` in `cluster1` can't receive any messages, which break the logic design completely. 
created_at: 2025-03-03T08:19:07Z
author_association: MEMBER
diff_hunk: @@ -33,25 +33,88 @@ retry/dead letter topic.
 
 ## Design & Implementation Details
 
-- Add two new configurations in `DeadLetterPolicy`:
+- Add `ProducerBuilderContext` and `ProducerBuilderCustomizer` interface:
+```java
+public interface ProducerBuilderContext {
+    /**
+     * Returns the default name of topic for the dead letter or retry letter producer. This topic name is used
+     * unless the ProducerBuilderCustomizer overrides it.
+     *
+     * @return a {@code String} representing the input topic name
+     */
+    String getDefaultTopicName();
+
+    /**
+     * Returns the name of the input topic for which the dead letter or retry letter producer is being configured.
+     *
+     * @return a {@code String} representing the input topic name
+     */
+    String getInputTopicName();
+
+    /**
+     * Returns the name of the subscription for which the dead letter or retry letter producer is being configured.
+     *
+     * @return a {@code String} representing the subscription name
+     */
+    String getInputTopicSubscriptionName();
+}
+
+public interface ProducerBuilderCustomizer {
+    /**
+     * Customize the given producer builder with settings specific to the topic context provided.
+     *
+     * @param context         the context containing information about the input topic and the subscription
+     * @param producerBuilder the producer builder instance to be customized
+     */
+    void customize(ProducerBuilderContext context, ProducerBuilder<byte[]> producerBuilder);
path: pip/pip-409.md
user: login: nodece
body: > However, wouldn't it be a valid use case to use a different Pulsar client instance for the producer?

This is a good case, and it is correct.

> Producing the retry messages to another cluster will break the logical design of retry topic.

When the user uses a different producer, the users know that use the new consumer to consume the message from a different cluster.
created_at: 2025-03-03T09:03:35Z
author_association: MEMBER
diff_hunk: @@ -33,25 +33,88 @@ retry/dead letter topic.
 
 ## Design & Implementation Details
 
-- Add two new configurations in `DeadLetterPolicy`:
+- Add `ProducerBuilderContext` and `ProducerBuilderCustomizer` interface:
+```java
+public interface ProducerBuilderContext {
+    /**
+     * Returns the default name of topic for the dead letter or retry letter producer. This topic name is used
+     * unless the ProducerBuilderCustomizer overrides it.
+     *
+     * @return a {@code String} representing the input topic name
+     */
+    String getDefaultTopicName();
+
+    /**
+     * Returns the name of the input topic for which the dead letter or retry letter producer is being configured.
+     *
+     * @return a {@code String} representing the input topic name
+     */
+    String getInputTopicName();
+
+    /**
+     * Returns the name of the subscription for which the dead letter or retry letter producer is being configured.
+     *
+     * @return a {@code String} representing the subscription name
+     */
+    String getInputTopicSubscriptionName();
+}
+
+public interface ProducerBuilderCustomizer {
+    /**
+     * Customize the given producer builder with settings specific to the topic context provided.
+     *
+     * @param context         the context containing information about the input topic and the subscription
+     * @param producerBuilder the producer builder instance to be customized
+     */
+    void customize(ProducerBuilderContext context, ProducerBuilder<byte[]> producerBuilder);
path: pip/pip-409.md
user: login: BewareMyPower
body: This interface can be implemented from the client side. Unnecessary flexibility might cause unexpected behaviors that are hard to diagnose. The current interceptor is such a bad example: https://github.com/apache/pulsar/blob/998bb51f5ea256c59110e5b8bef0f9daef1ac48d/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/interceptor/ProducerInterceptor.java#L83

With incorrect implementations, `beforeSend` could return `null` or a customized `Message` implementation. What's worse, if it returns a valid `MessageImpl` with broken `getValue()` implementation, it won't fail fast. Instead, a broken data could be written.

Therefore, I still suggest limiting such tricky use cases.
created_at: 2025-03-03T10:11:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -33,25 +33,88 @@ retry/dead letter topic.
 
 ## Design & Implementation Details
 
-- Add two new configurations in `DeadLetterPolicy`:
+- Add `ProducerBuilderContext` and `ProducerBuilderCustomizer` interface:
+```java
+public interface ProducerBuilderContext {
+    /**
+     * Returns the default name of topic for the dead letter or retry letter producer. This topic name is used
+     * unless the ProducerBuilderCustomizer overrides it.
+     *
+     * @return a {@code String} representing the input topic name
+     */
+    String getDefaultTopicName();
+
+    /**
+     * Returns the name of the input topic for which the dead letter or retry letter producer is being configured.
+     *
+     * @return a {@code String} representing the input topic name
+     */
+    String getInputTopicName();
+
+    /**
+     * Returns the name of the subscription for which the dead letter or retry letter producer is being configured.
+     *
+     * @return a {@code String} representing the subscription name
+     */
+    String getInputTopicSubscriptionName();
+}
+
+public interface ProducerBuilderCustomizer {
+    /**
+     * Customize the given producer builder with settings specific to the topic context provided.
+     *
+     * @param context         the context containing information about the input topic and the subscription
+     * @param producerBuilder the producer builder instance to be customized
+     */
+    void customize(ProducerBuilderContext context, ProducerBuilder<byte[]> producerBuilder);
path: pip/pip-409.md
user: login: BewareMyPower
body: The original motivation of PIP-409 is to support customizing a more efficient config of retry/dead letter topic producer. We should not make it too flexible or complicated for those cases out of that scope.
created_at: 2025-03-03T10:17:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -33,25 +33,88 @@ retry/dead letter topic.
 
 ## Design & Implementation Details
 
-- Add two new configurations in `DeadLetterPolicy`:
+- Add `ProducerBuilderContext` and `ProducerBuilderCustomizer` interface:
+```java
+public interface ProducerBuilderContext {
+    /**
+     * Returns the default name of topic for the dead letter or retry letter producer. This topic name is used
+     * unless the ProducerBuilderCustomizer overrides it.
+     *
+     * @return a {@code String} representing the input topic name
+     */
+    String getDefaultTopicName();
+
+    /**
+     * Returns the name of the input topic for which the dead letter or retry letter producer is being configured.
+     *
+     * @return a {@code String} representing the input topic name
+     */
+    String getInputTopicName();
+
+    /**
+     * Returns the name of the subscription for which the dead letter or retry letter producer is being configured.
+     *
+     * @return a {@code String} representing the subscription name
+     */
+    String getInputTopicSubscriptionName();
+}
+
+public interface ProducerBuilderCustomizer {
+    /**
+     * Customize the given producer builder with settings specific to the topic context provided.
+     *
+     * @param context         the context containing information about the input topic and the subscription
+     * @param producerBuilder the producer builder instance to be customized
+     */
+    void customize(ProducerBuilderContext context, ProducerBuilder<byte[]> producerBuilder);
path: pip/pip-409.md
user: login: thetumbled
body: WDYT @lhotari 
created_at: 2025-03-05T06:53:01Z
author_association: MEMBER
diff_hunk: @@ -62,4 +62,29 @@ public class DeadLetterPolicy implements Serializable {
      * to be created.
      */
     private String initialSubscriptionName;
+
+    /**
+     * Customizer for configuring the producer builder for the retry letter topic.
+     *
+     * <p>This field holds a function that allows the caller to customize the producer builder
+     * settings for the retry letter topic before the producer is created. The customization logic
+     * can use the provided context (which includes input topic and subscription details) to adjust
+     * configurations such as timeouts, batching, or message routing.
+     */
+    private ProducerBuilderCustomizer retryLetterProducerBuilderCustomizer = DEFAULT_PRODUCER_BUILDER_CUSTOMIZER;
+
+    /**
+     * Customizer for configuring the producer builder for the dead letter topic.
+     *
+     * <p>This field holds a function that allows the caller to customize the producer builder
+     * settings for the dead letter topic before the producer is created. Using the provided context,
+     * implementations can perform specific adjustments that ensure the dead letter queue operates
+     * with the appropriate configurations tailored for handling undeliverable messages.
+     */
+    private ProducerBuilderCustomizer deadLetterProducerBuilderCustomizer = DEFAULT_PRODUCER_BUILDER_CUSTOMIZER;
+
+    // keep compatibility with old configuration
+    private static final ProducerBuilderCustomizer DEFAULT_PRODUCER_BUILDER_CUSTOMIZER =
+            (context, builder) ->
+                    builder.enableBatching(false).enableChunking(true).blockIfQueueFull(false);
path: pulsar-client-api/src/main/java/org/apache/pulsar/client/api/DeadLetterPolicy.java
user: login: lhotari
body: Instead of having a default customizer, I'd suggest a solution where in the the runtime, the old configuration is always applied and then a user provider customizer is called if it has been set. If the user would like to change the defaults, they could do that in the customizer.
created_at: 2025-03-05T07:17:12Z
author_association: MEMBER
diff_hunk: @@ -2294,17 +2295,16 @@ private CompletableFuture<Producer<byte[]>> initDeadLetterProducerIfNeeded() {
                 p = deadLetterProducer;
                 if (p == null || p.isCompletedExceptionally()) {
                     p = createProducerWithBackOff(() -> {
-                        CompletableFuture<Producer<byte[]>> newProducer =
-                                ((ProducerBuilderImpl<byte[]>) client.newProducer(Schema.AUTO_PRODUCE_BYTES(schema)))
-                                        .initialSubscriptionName(this.deadLetterPolicy.getInitialSubscriptionName())
-                                        .topic(this.deadLetterPolicy.getDeadLetterTopic())
-                                        .producerName(
-                                                String.format("%s-%s-%s-%s-DLQ", this.topicName, this.subscription,
-                                                        this.consumerName, RandomStringUtils.randomAlphanumeric(5)))
-                                        .blockIfQueueFull(false)
-                                        .enableBatching(false)
-                                        .enableChunking(true)
path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
user: login: lhotari
body: keep these as explained in the previous comment
created_at: 2025-03-05T07:18:26Z
author_association: MEMBER
diff_hunk: @@ -2294,17 +2295,16 @@ private CompletableFuture<Producer<byte[]>> initDeadLetterProducerIfNeeded() {
                 p = deadLetterProducer;
                 if (p == null || p.isCompletedExceptionally()) {
                     p = createProducerWithBackOff(() -> {
-                        CompletableFuture<Producer<byte[]>> newProducer =
-                                ((ProducerBuilderImpl<byte[]>) client.newProducer(Schema.AUTO_PRODUCE_BYTES(schema)))
-                                        .initialSubscriptionName(this.deadLetterPolicy.getInitialSubscriptionName())
-                                        .topic(this.deadLetterPolicy.getDeadLetterTopic())
-                                        .producerName(
-                                                String.format("%s-%s-%s-%s-DLQ", this.topicName, this.subscription,
-                                                        this.consumerName, RandomStringUtils.randomAlphanumeric(5)))
-                                        .blockIfQueueFull(false)
-                                        .enableBatching(false)
-                                        .enableChunking(true)
-                                        .createAsync();
+                        ProducerBuilderImpl<byte[]> builder = (ProducerBuilderImpl<byte[]>) client
+                                .newProducer(Schema.AUTO_PRODUCE_BYTES(schema))
+                                .topic(deadLetterPolicy.getDeadLetterTopic());
+                        builder.initialSubscriptionName(deadLetterPolicy.getInitialSubscriptionName());
+                        if (deadLetterPolicy.getDeadLetterProducerBuilderCustomizer() != null) {
+                            ProducerBuilderContext context = new ProducerBuilderContextImpl(
+                                    deadLetterPolicy.getDeadLetterTopic(), topic, subscription);
+                            deadLetterPolicy.getDeadLetterProducerBuilderCustomizer().customize(context, builder);
path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
user: login: lhotari
body: only create the context instance and call the customizer if the customizer is non-null
created_at: 2025-03-05T07:19:13Z
author_association: MEMBER
diff_hunk: @@ -33,25 +33,88 @@ retry/dead letter topic.
 
 ## Design & Implementation Details
 
-- Add two new configurations in `DeadLetterPolicy`:
+- Add `ProducerBuilderContext` and `ProducerBuilderCustomizer` interface:
+```java
+public interface ProducerBuilderContext {
+    /**
+     * Returns the default name of topic for the dead letter or retry letter producer. This topic name is used
+     * unless the ProducerBuilderCustomizer overrides it.
+     *
+     * @return a {@code String} representing the input topic name
+     */
+    String getDefaultTopicName();
+
+    /**
+     * Returns the name of the input topic for which the dead letter or retry letter producer is being configured.
+     *
+     * @return a {@code String} representing the input topic name
+     */
+    String getInputTopicName();
+
+    /**
+     * Returns the name of the subscription for which the dead letter or retry letter producer is being configured.
+     *
+     * @return a {@code String} representing the subscription name
+     */
+    String getInputTopicSubscriptionName();
+}
+
+public interface ProducerBuilderCustomizer {
+    /**
+     * Customize the given producer builder with settings specific to the topic context provided.
+     *
+     * @param context         the context containing information about the input topic and the subscription
+     * @param producerBuilder the producer builder instance to be customized
+     */
+    void customize(ProducerBuilderContext context, ProducerBuilder<byte[]> producerBuilder);
path: pip/pip-409.md
user: login: lhotari
body: I'm fine with the customizer approach without allowing other use cases such as replacing the producer builder. That could be added later if needed, without breaking the API.
created_at: 2025-03-05T07:21:12Z
author_association: MEMBER
diff_hunk: @@ -62,4 +62,29 @@ public class DeadLetterPolicy implements Serializable {
      * to be created.
      */
     private String initialSubscriptionName;
+
+    /**
+     * Customizer for configuring the producer builder for the retry letter topic.
+     *
+     * <p>This field holds a function that allows the caller to customize the producer builder
+     * settings for the retry letter topic before the producer is created. The customization logic
+     * can use the provided context (which includes input topic and subscription details) to adjust
+     * configurations such as timeouts, batching, or message routing.
+     */
+    private ProducerBuilderCustomizer retryLetterProducerBuilderCustomizer = DEFAULT_PRODUCER_BUILDER_CUSTOMIZER;
+
+    /**
+     * Customizer for configuring the producer builder for the dead letter topic.
+     *
+     * <p>This field holds a function that allows the caller to customize the producer builder
+     * settings for the dead letter topic before the producer is created. Using the provided context,
+     * implementations can perform specific adjustments that ensure the dead letter queue operates
+     * with the appropriate configurations tailored for handling undeliverable messages.
+     */
+    private ProducerBuilderCustomizer deadLetterProducerBuilderCustomizer = DEFAULT_PRODUCER_BUILDER_CUSTOMIZER;
+
+    // keep compatibility with old configuration
+    private static final ProducerBuilderCustomizer DEFAULT_PRODUCER_BUILDER_CUSTOMIZER =
+            (context, builder) ->
+                    builder.enableBatching(false).enableChunking(true).blockIfQueueFull(false);
path: pulsar-client-api/src/main/java/org/apache/pulsar/client/api/DeadLetterPolicy.java
user: login: thetumbled
body: The implementation is updated, PTAL again, thanks.
created_at: 2025-03-05T07:35:37Z
author_association: MEMBER
diff_hunk: @@ -2294,17 +2295,16 @@ private CompletableFuture<Producer<byte[]>> initDeadLetterProducerIfNeeded() {
                 p = deadLetterProducer;
                 if (p == null || p.isCompletedExceptionally()) {
                     p = createProducerWithBackOff(() -> {
-                        CompletableFuture<Producer<byte[]>> newProducer =
-                                ((ProducerBuilderImpl<byte[]>) client.newProducer(Schema.AUTO_PRODUCE_BYTES(schema)))
-                                        .initialSubscriptionName(this.deadLetterPolicy.getInitialSubscriptionName())
-                                        .topic(this.deadLetterPolicy.getDeadLetterTopic())
-                                        .producerName(
-                                                String.format("%s-%s-%s-%s-DLQ", this.topicName, this.subscription,
-                                                        this.consumerName, RandomStringUtils.randomAlphanumeric(5)))
-                                        .blockIfQueueFull(false)
-                                        .enableBatching(false)
-                                        .enableChunking(true)
path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
user: login: lhotari
body: There was a slight misunderstanding regarding the `DEFAULT_PRODUCER_BUILDER_CUSTOMIZER`. Remove it completely, it's not needed when you revert the change to remove these lines above.

If the user would like to change the above settings, they could do that in the customizer. It also is important for the "customizer" abstraction that unless you do any changes in the customizer, the defaults will be use. The customizer customizes the producer builder after the runtime has configured the defaults. Makes sense?

 
created_at: 2025-03-05T08:47:31Z
author_association: MEMBER
diff_hunk: @@ -2294,17 +2295,16 @@ private CompletableFuture<Producer<byte[]>> initDeadLetterProducerIfNeeded() {
                 p = deadLetterProducer;
                 if (p == null || p.isCompletedExceptionally()) {
                     p = createProducerWithBackOff(() -> {
-                        CompletableFuture<Producer<byte[]>> newProducer =
-                                ((ProducerBuilderImpl<byte[]>) client.newProducer(Schema.AUTO_PRODUCE_BYTES(schema)))
-                                        .initialSubscriptionName(this.deadLetterPolicy.getInitialSubscriptionName())
-                                        .topic(this.deadLetterPolicy.getDeadLetterTopic())
-                                        .producerName(
-                                                String.format("%s-%s-%s-%s-DLQ", this.topicName, this.subscription,
-                                                        this.consumerName, RandomStringUtils.randomAlphanumeric(5)))
-                                        .blockIfQueueFull(false)
-                                        .enableBatching(false)
-                                        .enableChunking(true)
path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
user: login: thetumbled
body: Maybe you can suggest any code, or it is difficult to understand the subtle difference between current implementation and your expectation.
I don't get the reason why we should remove the `DEFAULT_PRODUCER_BUILDER_CUSTOMIZER`.
created_at: 2025-03-05T08:54:18Z
author_association: MEMBER
diff_hunk: @@ -2294,17 +2295,16 @@ private CompletableFuture<Producer<byte[]>> initDeadLetterProducerIfNeeded() {
                 p = deadLetterProducer;
                 if (p == null || p.isCompletedExceptionally()) {
                     p = createProducerWithBackOff(() -> {
-                        CompletableFuture<Producer<byte[]>> newProducer =
-                                ((ProducerBuilderImpl<byte[]>) client.newProducer(Schema.AUTO_PRODUCE_BYTES(schema)))
-                                        .initialSubscriptionName(this.deadLetterPolicy.getInitialSubscriptionName())
-                                        .topic(this.deadLetterPolicy.getDeadLetterTopic())
-                                        .producerName(
-                                                String.format("%s-%s-%s-%s-DLQ", this.topicName, this.subscription,
-                                                        this.consumerName, RandomStringUtils.randomAlphanumeric(5)))
-                                        .blockIfQueueFull(false)
-                                        .enableBatching(false)
-                                        .enableChunking(true)
path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
user: login: lhotari
body: @thetumbled I pushed directly to this PR, f353e02. There was another issue that the ProducerBuilderContextImpl was in the wrong package. This is a class that should be kept private in the same package as ConsumerImpl. It could even be a private inner class in ConsumerImpl, there's no value in reusing it.
created_at: 2025-03-05T09:36:40Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.api;
+
+/**
+ * Provides context information required for customizing a producer builder.
+ *
+ * <p>This interface supplies relevant details such as the name of the input topic and associated subscription name.
+ * This contextual information helps in correctly configuring the producer for the appropriate topic.
+ */
+public interface ProducerBuilderContext {
+    /**
+     * Returns the default name of topic for the dead letter or retry letter producer. This topic name is used
+     * unless the ProducerBuilderCustomizer overrides it.
+     *
+     * @return a {@code String} representing the input topic name
+     */
+    String getDefaultTopicName();
+
+    /**
+     * Returns the name of the input topic for which the dead letter or retry letter producer is being configured.
+     *
+     * @return a {@code String} representing the input topic name
+     */
+    String getInputTopicName();
+
+    /**
+     * Returns the name of the subscription for which the dead letter or retry letter producer is being configured.
+     *
+     * @return a {@code String} representing the subscription name
+     */
+    String getInputTopicSubscriptionName();
+}
path: pulsar-client-api/src/main/java/org/apache/pulsar/client/api/ProducerBuilderContext.java
user: login: lhotari
body: It could be useful to also add `getInputTopicConsumerName()`. That way, there would be the most relevant context available for the customizer.
created_at: 2025-03-05T09:46:19Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.api;
+
+/**
+ * Functional interface for customizing a producer builder for a specific topic.
+ *
+ * <p>This interface allows for customizing the producer builder configuration for either the retry letter topic
+ * or the dead letter topic. The customization might include setting producer properties such as batching, timeouts,
+ * or any other producer-specific configuration.
+ *
+ * @see ProducerBuilderContext
+ */
+public interface ProducerBuilderCustomizer {
path: pulsar-client-api/src/main/java/org/apache/pulsar/client/api/ProducerBuilderCustomizer.java
user: login: lhotari
body: The naming isn't yet fully accurate. It would be useful to have a prefix that clarifies that this applies to deadletter topic and retry letter topic producer builders. 

`DeadLetterProducerBuilderCustomizer` would be consistent with `DeadLetterPolicy` which covers also the retry letter topic. Similarly `ProducerBuilderContext` would be `DeadLetterProducerBuilderContext`.
created_at: 2025-03-05T09:50:32Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.api;
+
+/**
+ * Provides context information required for customizing a producer builder.
+ *
+ * <p>This interface supplies relevant details such as the name of the input topic and associated subscription name.
+ * This contextual information helps in correctly configuring the producer for the appropriate topic.
+ */
+public interface ProducerBuilderContext {
+    /**
+     * Returns the default name of topic for the dead letter or retry letter producer. This topic name is used
+     * unless the ProducerBuilderCustomizer overrides it.
+     *
+     * @return a {@code String} representing the input topic name
+     */
+    String getDefaultTopicName();
+
+    /**
+     * Returns the name of the input topic for which the dead letter or retry letter producer is being configured.
+     *
+     * @return a {@code String} representing the input topic name
+     */
+    String getInputTopicName();
+
+    /**
+     * Returns the name of the subscription for which the dead letter or retry letter producer is being configured.
+     *
+     * @return a {@code String} representing the subscription name
+     */
+    String getInputTopicSubscriptionName();
+}
path: pulsar-client-api/src/main/java/org/apache/pulsar/client/api/ProducerBuilderContext.java
user: login: thetumbled
body: added.
created_at: 2025-03-05T09:55:08Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.api;
+
+/**
+ * Functional interface for customizing a producer builder for a specific topic.
+ *
+ * <p>This interface allows for customizing the producer builder configuration for either the retry letter topic
+ * or the dead letter topic. The customization might include setting producer properties such as batching, timeouts,
+ * or any other producer-specific configuration.
+ *
+ * @see ProducerBuilderContext
+ */
+public interface ProducerBuilderCustomizer {
path: pulsar-client-api/src/main/java/org/apache/pulsar/client/api/ProducerBuilderCustomizer.java
user: login: thetumbled
body: fixed.
created_at: 2025-03-05T09:59:15Z
author_association: MEMBER
diff_hunk: @@ -2303,8 +2305,13 @@ private CompletableFuture<Producer<byte[]>> initDeadLetterProducerIfNeeded() {
                                                         this.consumerName, RandomStringUtils.randomAlphanumeric(5)))
                                         .blockIfQueueFull(false)
                                         .enableBatching(false)
-                                        .enableChunking(true)
-                                        .createAsync();
+                                        .enableChunking(true);
+                        if (deadLetterPolicy.getDeadLetterProducerBuilderCustomizer() != null) {
+                            DeadLetterProducerBuilderContext context = new DeadLetterProducerBuilderContextImpl(
+                                    deadLetterPolicy.getDeadLetterTopic(), topic, subscription, consumerName);
+                            deadLetterPolicy.getDeadLetterProducerBuilderCustomizer().customize(context, builder);
+                        }
path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
user: login: lhotari
body: since this code is duplicated also for the retry letter case, it would be possible to extract a private method.

```java
    private void customizeDeadLetterProducerBuilder(DeadLetterProducerBuilderCustomizer customizer,
                                                    String deadLetterOrRetryLetterTopic,
                                                    ProducerBuilder<byte[]> builder) {
        if (customizer != null) {
            DeadLetterProducerBuilderContext context = new DeadLetterProducerBuilderContextImpl(
                    deadLetterOrRetryLetterTopic, topic, subscription, consumerName);
            customizer.customize(context, builder);
        }
    }
```

could be called here with
```java
                        customizeDeadLetterProducerBuilder(deadLetterPolicy.getDeadLetterProducerBuilderCustomizer(),
                                deadLetterPolicy.getDeadLetterTopic(), builder);
```                                

and in the retry letter case

```java

customizeDeadLetterProducerBuilder(deadLetterPolicy.getRetryLetterProducerBuilderCustomizer(),
                                deadLetterPolicy.getRetryLetterTopic(), builder);

```

created_at: 2025-03-05T10:17:25Z
author_association: MEMBER
diff_hunk: @@ -33,25 +33,88 @@ retry/dead letter topic.
 
 ## Design & Implementation Details
 
-- Add two new configurations in `DeadLetterPolicy`:
+- Add `ProducerBuilderContext` and `ProducerBuilderCustomizer` interface:
path: pip/pip-409.md
user: login: lhotari
body: Update the naming also in the PIP document since it's being maintained in this PR.
created_at: 2025-03-05T10:19:01Z
author_association: MEMBER
diff_hunk: @@ -33,25 +33,88 @@ retry/dead letter topic.
 
 ## Design & Implementation Details
 
-- Add two new configurations in `DeadLetterPolicy`:
+- Add `ProducerBuilderContext` and `ProducerBuilderCustomizer` interface:
path: pip/pip-409.md
user: login: thetumbled
body: fixed.
created_at: 2025-03-06T02:19:30Z
author_association: MEMBER
diff_hunk: @@ -2303,8 +2305,13 @@ private CompletableFuture<Producer<byte[]>> initDeadLetterProducerIfNeeded() {
                                                         this.consumerName, RandomStringUtils.randomAlphanumeric(5)))
                                         .blockIfQueueFull(false)
                                         .enableBatching(false)
-                                        .enableChunking(true)
-                                        .createAsync();
+                                        .enableChunking(true);
+                        if (deadLetterPolicy.getDeadLetterProducerBuilderCustomizer() != null) {
+                            DeadLetterProducerBuilderContext context = new DeadLetterProducerBuilderContextImpl(
+                                    deadLetterPolicy.getDeadLetterTopic(), topic, subscription, consumerName);
+                            deadLetterPolicy.getDeadLetterProducerBuilderCustomizer().customize(context, builder);
+                        }
path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
user: login: thetumbled
body: updated.
created_at: 2025-03-06T02:27:09Z
author_association: MEMBER

PR comments:
user: login: thetumbled
created_at: 2025-02-28T09:39:21Z
body: The pip document has been merged, we can continue to review this pr. Thanks for your review. @BewareMyPower @nodece @dao-jun @codelipenghui @shibd @Technoboy- 
author_association: MEMBER
user: login: lhotari
created_at: 2025-02-28T14:26:04Z
body: Great work, @thetumbled. Thanks for making this happen. The PIP-409 solution will help resolve long time issues such as #19463 and #21954.
author_association: MEMBER
user: login: thetumbled
created_at: 2025-03-05T09:45:22Z
body: > LGTM, good work @thetumbled . Great to see this finally solving many use cases.

Thanks for your review and contributions!
author_association: MEMBER
user: login: thetumbled
created_at: 2025-03-05T10:00:33Z
body: > One remaining issue about naming. Commented added.

fixed, thanks.
author_association: MEMBER
user: login: codecov-commenter
created_at: 2025-03-06T08:11:12Z
body: ## [Codecov](https://app.codecov.io/gh/apache/pulsar/pull/24020?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: Patch coverage is `55.55556%` with `8 lines` in your changes missing coverage. Please review.
> Project coverage is 74.20%. Comparing base [(`bbc6224`)](https://app.codecov.io/gh/apache/pulsar/commit/bbc62245c5ddba1de4b1e7cee4ab49334bc36277?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`089cb74`)](https://app.codecov.io/gh/apache/pulsar/commit/089cb747c5c5b56d75d2c46c7d6148b34fae7753?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 951 commits behind head on master.

| [Files with missing lines](https://app.codecov.io/gh/apache/pulsar/pull/24020?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...va/org/apache/pulsar/client/impl/ConsumerImpl.java](https://app.codecov.io/gh/apache/pulsar/pull/24020?src=pr&el=tree&filepath=pulsar-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fclient%2Fimpl%2FConsumerImpl.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL0NvbnN1bWVySW1wbC5qYXZh) | 71.42% | [3 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/pulsar/pull/24020?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ent/impl/DeadLetterProducerBuilderContextImpl.java](https://app.codecov.io/gh/apache/pulsar/pull/24020?src=pr&el=tree&filepath=pulsar-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fclient%2Fimpl%2FDeadLetterProducerBuilderContextImpl.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL0RlYWRMZXR0ZXJQcm9kdWNlckJ1aWxkZXJDb250ZXh0SW1wbC5qYXZh) | 0.00% | [4 Missing :warning: ](https://app.codecov.io/gh/apache/pulsar/pull/24020?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |

<details><summary>Additional details and impacted files</summary>


[![Impacted file tree graph](https://app.codecov.io/gh/apache/pulsar/pull/24020/graphs/tree.svg?width=650&height=150&src=pr&token=acYqCpsK9J&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)](https://app.codecov.io/gh/apache/pulsar/pull/24020?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)

```diff
@@             Coverage Diff              @@
##             master   #24020      +/-   ##
============================================
+ Coverage     73.57%   74.20%   +0.62%     
+ Complexity    32624    32031     -593     
============================================
  Files          1877     1862      -15     
  Lines        139502   144225    +4723     
  Branches      15299    16433    +1134     
============================================
+ Hits         102638   107015    +4377     
+ Misses        28908    28764     -144     
- Partials       7956     8446     +490     
```

| [Flag](https://app.codecov.io/gh/apache/pulsar/pull/24020/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [inttests](https://app.codecov.io/gh/apache/pulsar/pull/24020/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `26.67% <38.88%> (+2.09%)` | :arrow_up: |
| [systests](https://app.codecov.io/gh/apache/pulsar/pull/24020/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `23.08% <0.00%> (-1.24%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pulsar/pull/24020/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.71% <55.55%> (+0.87%)` | :arrow_up: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Files with missing lines](https://app.codecov.io/gh/apache/pulsar/pull/24020?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...va/org/apache/pulsar/client/impl/ConsumerImpl.java](https://app.codecov.io/gh/apache/pulsar/pull/24020?src=pr&el=tree&filepath=pulsar-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fclient%2Fimpl%2FConsumerImpl.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL0NvbnN1bWVySW1wbC5qYXZh) | `78.85% <71.42%> (+1.28%)` | :arrow_up: |
| [...ent/impl/DeadLetterProducerBuilderContextImpl.java](https://app.codecov.io/gh/apache/pulsar/pull/24020?src=pr&el=tree&filepath=pulsar-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fclient%2Fimpl%2FDeadLetterProducerBuilderContextImpl.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL0RlYWRMZXR0ZXJQcm9kdWNlckJ1aWxkZXJDb250ZXh0SW1wbC5qYXZh) | `0.00% <0.00%> (ø)` | |

... and [1054 files with indirect coverage changes](https://app.codecov.io/gh/apache/pulsar/pull/24020/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)

</details>
<details><summary>🚀 New features to boost your workflow: </summary>

- ❄ [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- 📦 [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
author_association: NONE