Repository: apache/rocketmq (2026-07-13T07:49:42Z)

Issue:
Async request-reply RequestCallback may fire multiple times (premature onSuccess(null) + reply/timeout double callback)
**BUG REPORT**

1. Please describe the issue you observed:

The asynchronous request-reply API `DefaultMQProducer#request(Message, RequestCallback, long)` can invoke the user `RequestCallback` **more than once** for a single request. Two independent problems combine:

**(a) Premature `onSuccess(null)` on send success.**
In `DefaultMQProducerImpl#request(Message, RequestCallback, long)`, the async `SendCallback#onSuccess` calls `requestResponseFuture.executeRequestCallback()` right after the *request* message is sent:

```java
this.sendDefaultImpl(msg, CommunicationMode.ASYNC, new SendCallback() {
    @Override
    public void onSuccess(SendResult sendResult) {
        requestResponseFuture.setSendRequestOk(true);
        requestResponseFuture.executeRequestCallback();   // fires onSuccess(responseMsg==null)
    }
    ...
```

At that moment no reply has arrived, so `responseMsg` is `null` and the user receives `onSuccess(null)`. The other async overloads (`request(msg, selector, arg, callback, timeout)` and `request(msg, mq, callback, timeout)`) only set `sendRequestOk=true` here and do NOT invoke the callback, so this overload is inconsistent.

**(b) reply-vs-timeout race → double callback.**
`ClientRemotingProcessor#processReplyMessage` (reply arrival) does a non-atomic `get` + `remove`, while `RequestFutureHolder#scanExpiredRequest` (timeout scan) uses `iterator.remove()`, and `RequestResponseFuture#executeRequestCallback` has no single-shot guard. So the reply path and the timeout path can both take ownership of the same request and both run a callback.

As a result a single request may deliver `onSuccess(null)` then `onSuccess(reply)`, or `onSuccess(null)` then `onException(timeout)`.

2. Please tell us about your environment:

- RocketMQ version: develop (5.x)
- OS / JDK: Linux / JDK 21

3. Steps to reproduce:

A producer issues N async `request(msg, callback, timeout)` calls; a consumer replies via `MessageUtil.createReplyMessage(...)` after a delay near the request timeout; count callbacks per `correlationId`.

Observed on a 4-node cluster (2000 requests, 16 threads, timeout=1000ms, reply delay=200ms):

```
consumed=1312, replySent=1280
successNull=2000, successReply=284, exception=1716
doubleCallback=2000   (nullThenReply=284, successAndException=1716)
```

Every request received two callbacks.

4. What did you expect to see?

Each request delivers its `RequestCallback` **exactly once**: `onSuccess(reply)` when a reply arrives, or `onException(...)` on timeout / send failure. There should be no `onSuccess(null)` on send success.

5. What did you see instead?

Same request produced two callbacks (a premature `onSuccess(null)` followed by the real reply or a timeout), corrupting caller state.



Discussion:
**Issue Evaluation**

Category: `type/bug` | Status: **Confirmed**

The reported issue has been verified against the current codebase. Both problems are confirmed.

**Problem (a): Premature `onSuccess(null)` on send success — CONFIRMED**

In `DefaultMQProducerImpl.java:1665-1666`, the `request(Message, RequestCallback, long)` overload calls `requestResponseFuture.executeRequestCallback()` inside `SendCallback.onSuccess`, before any reply has arrived. At that point `responseMsg` is still `null`, so the user callback fires with `onSuccess(null)`. The other two async overloads (`request(msg, selector, arg, callback, timeout)` at line 1723 and `request(msg, mq, callback, timeout)` at line 1793) correctly only set `sendRequestOk=true` without invoking the callback — this overload is inconsistent.

**Problem (b): Reply-vs-timeout race → double callback — CONFIRMED**

`RequestResponseFuture.executeRequestCallback()` (line 41-49) has no single-shot guard (no AtomicBoolean, no CAS, no synchronized block). The three invocation paths (`SendCallback.onSuccess`, `ClientRemotingProcessor.processReplyMessage`, `RequestFutureHolder.scanExpiredRequest`) are not coordinated atomically:
- `processReplyMessage` (line 280-287) does a non-atomic `get` + `remove` on the ConcurrentHashMap
- `scanExpiredRequest` (line 57-67) can interleave within that window via `iterator.remove()`
- Result: both reply and timeout paths can invoke the callback for the same request

**Root Cause:** Missing single-shot guard in `executeRequestCallback()` + non-atomic get+remove in reply processing + inconsistent callback invocation across overloads.

**Impact:** `DefaultMQProducerImpl` async request-reply API
**Severity:** HIGH — Problem (a) is deterministic (every successful send fires `onSuccess(null)`); Problem (b) is a realistic TOCTOU race under load.

An automated fix proposal will be generated. Reply `/approve` to proceed with PR generation, `/revise` to request changes to the approach, or `/reject` to decline.

---
*Automated evaluation by github-manager-bot*



Patch:
diff --git a/client/src/main/java/org/apache/rocketmq/client/impl/ClientRemotingProcessor.java b/client/src/main/java/org/apache/rocketmq/client/impl/ClientRemotingProcessor.java
index 0a187751f27..567198e3a8e 100644
--- a/client/src/main/java/org/apache/rocketmq/client/impl/ClientRemotingProcessor.java
+++ b/client/src/main/java/org/apache/rocketmq/client/impl/ClientRemotingProcessor.java
@@ -272,15 +272,14 @@ private RemotingCommand receiveReplyMessage(ChannelHandlerContext ctx,
 
     private void processReplyMessage(MessageExt replyMsg) {
         final String correlationId = replyMsg.getUserProperty(MessageConst.PROPERTY_CORRELATION_ID);
-        final RequestResponseFuture requestResponseFuture = RequestFutureHolder.getInstance().getRequestFutureTable().get(correlationId);
+        // Atomically remove so that only one of the reply-arrival path and the timeout-scan path
+        // (RequestFutureHolder#scanExpiredRequest) can take ownership of this request.
+        final RequestResponseFuture requestResponseFuture = RequestFutureHolder.getInstance().getRequestFutureTable().remove(correlationId);
         if (requestResponseFuture != null) {
             requestResponseFuture.putResponseMessage(replyMsg);
-
-            RequestFutureHolder.getInstance().getRequestFutureTable().remove(correlationId);
-
-            if (requestResponseFuture.getRequestCallback() != null) {
-                requestResponseFuture.getRequestCallback().onSuccess(replyMsg);
-            }
+            // Route through executeRequestCallback so the single-callback guard applies to the
+            // reply-success path too; sync callers (null callback) are woken by putResponseMessage.
+            requestResponseFuture.executeRequestCallback();
         } else {
             String bornHost = replyMsg.getBornHostString();
             logger.warn("receive reply message, but not matched any request, CorrelationId: {} , reply from host: {}",
diff --git a/client/src/main/java/org/apache/rocketmq/client/impl/producer/DefaultMQProducerImpl.java b/client/src/main/java/org/apache/rocketmq/client/impl/producer/DefaultMQProducerImpl.java
index f68742949ff..9ad5fcef4dc 100644
--- a/client/src/main/java/org/apache/rocketmq/client/impl/producer/DefaultMQProducerImpl.java
+++ b/client/src/main/java/org/apache/rocketmq/client/impl/producer/DefaultMQProducerImpl.java
@@ -1662,8 +1662,11 @@ public void request(Message msg, final RequestCallback requestCallback, long tim
         this.sendDefaultImpl(msg, CommunicationMode.ASYNC, new SendCallback() {
             @Override
             public void onSuccess(SendResult sendResult) {
+                // Only mark the request as sent here. The user callback must fire when the reply
+                // arrives (processReplyMessage), on timeout (scanExpiredRequest), or on send
+                // failure (requestFail). Invoking it here delivers a premature onSuccess(null)
+                // and, combined with the later reply/timeout callback, causes a double callback.
                 requestResponseFuture.setSendRequestOk(true);
-                requestResponseFuture.executeRequestCallback();
             }
 
             @Override
diff --git a/client/src/main/java/org/apache/rocketmq/client/producer/RequestFutureHolder.java b/client/src/main/java/org/apache/rocketmq/client/producer/RequestFutureHolder.java
index 00f5bb6e6ea..b489873c79a 100644
--- a/client/src/main/java/org/apache/rocketmq/client/producer/RequestFutureHolder.java
+++ b/client/src/main/java/org/apache/rocketmq/client/producer/RequestFutureHolder.java
@@ -54,9 +54,13 @@ private void scanExpiredRequest() {
             RequestResponseFuture rep = next.getValue();
 
             if (rep.isTimeout()) {
-                it.remove();
-                rfList.add(rep);
-                log.warn("remove timeout request, CorrelationId={}" + rep.getCorrelationId());
+                // Atomically remove so the timeout path and the reply-arrival path in
+                // ClientRemotingProcessor#processReplyMessage cannot both handle the same request.
+                RequestResponseFuture removed = requestFutureTable.remove(next.getKey());
+                if (removed != null) {
+                    rfList.add(removed);
+                    log.warn("remove timeout request, CorrelationId={}", removed.getCorrelationId());
+                }
             }
         }
 
diff --git a/client/src/main/java/org/apache/rocketmq/client/producer/RequestResponseFuture.java b/client/src/main/java/org/apache/rocketmq/client/producer/RequestResponseFuture.java
index e66c08fdc53..8d13dc7f52f 100644
--- a/client/src/main/java/org/apache/rocketmq/client/producer/RequestResponseFuture.java
+++ b/client/src/main/java/org/apache/rocketmq/client/producer/RequestResponseFuture.java
@@ -19,6 +19,7 @@
 
 import java.util.concurrent.CountDownLatch;
 import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
 import org.apache.rocketmq.common.message.Message;
 
 public class RequestResponseFuture {
@@ -31,6 +32,7 @@ public class RequestResponseFuture {
     private volatile Message responseMsg = null;
     private volatile boolean sendRequestOk = true;
     private volatile Throwable cause = null;
+    private final AtomicBoolean executeCallbackOnlyOnce = new AtomicBoolean(false);
 
     public RequestResponseFuture(String correlationId, long timeoutMillis, RequestCallback requestCallback) {
         this.correlationId = correlationId;
@@ -39,12 +41,18 @@ public RequestResponseFuture(String correlationId, long timeoutMillis, RequestCa
     }
 
     public void executeRequestCallback() {
-        if (requestCallback != null) {
-            if (sendRequestOk && cause == null) {
-                requestCallback.onSuccess(responseMsg);
-            } else {
-                requestCallback.onException(cause);
-            }
+        if (requestCallback == null) {
+            return;
+        }
+        // Ensure the callback fires exactly once even if the reply-arrival path and the
+        // timeout-scan path race on the same request. Whoever wins the CAS runs the callback.
+        if (!this.executeCallbackOnlyOnce.compareAndSet(false, true)) {
+            return;
+        }
+        if (sendRequestOk && cause == null) {
+            requestCallback.onSuccess(responseMsg);
+        } else {
+            requestCallback.onException(cause);
         }
     }
 


Test patch:
diff --git a/client/src/test/java/org/apache/rocketmq/client/producer/RequestResponseFutureTest.java b/client/src/test/java/org/apache/rocketmq/client/producer/RequestResponseFutureTest.java
index 68615f3b394..f5c3d7709cc 100644
--- a/client/src/test/java/org/apache/rocketmq/client/producer/RequestResponseFutureTest.java
+++ b/client/src/test/java/org/apache/rocketmq/client/producer/RequestResponseFutureTest.java
@@ -18,6 +18,7 @@
 package org.apache.rocketmq.client.producer;
 
 import java.util.UUID;
+import java.util.concurrent.CountDownLatch;
 import java.util.concurrent.atomic.AtomicInteger;
 import org.apache.rocketmq.common.message.Message;
 import org.junit.Test;
@@ -44,4 +45,69 @@ public void onException(Throwable e) {
         assertThat(cc.get()).isEqualTo(1);
     }
 
+    @Test
+    public void testExecuteRequestCallbackSuccessThenTimeoutFiresOnce() {
+        final AtomicInteger success = new AtomicInteger(0);
+        final AtomicInteger exception = new AtomicInteger(0);
+        RequestResponseFuture future = new RequestResponseFuture(UUID.randomUUID().toString(), 3 * 1000L, new RequestCallback() {
+            @Override
+            public void onSuccess(Message message) {
+                success.incrementAndGet();
+            }
+
+            @Override
+            public void onException(Throwable e) {
+                exception.incrementAndGet();
+            }
+        });
+
+        // Reply-arrival path wins first (success), then the timeout-scan path tries to fire again.
+        future.setSendRequestOk(true);
+        future.executeRequestCallback();
+
+        future.setCause(new RuntimeException("request timeout, no reply message."));
+        future.executeRequestCallback();
+
+        // The same request must never deliver both a success and a timeout callback.
+        assertThat(success.get()).isEqualTo(1);
+        assertThat(exception.get()).isEqualTo(0);
+    }
+
+    @Test
+    public void testExecuteRequestCallbackConcurrentlyFiresOnce() throws Exception {
+        final AtomicInteger total = new AtomicInteger(0);
+        final RequestResponseFuture future = new RequestResponseFuture(UUID.randomUUID().toString(), 3 * 1000L, new RequestCallback() {
+            @Override
+            public void onSuccess(Message message) {
+                total.incrementAndGet();
+            }
+
+            @Override
+            public void onException(Throwable e) {
+                total.incrementAndGet();
+            }
+        });
+        future.setSendRequestOk(true);
+
+        int threads = 16;
+        final CountDownLatch start = new CountDownLatch(1);
+        final CountDownLatch done = new CountDownLatch(threads);
+        for (int i = 0; i < threads; i++) {
+            new Thread(() -> {
+                try {
+                    start.await();
+                    future.executeRequestCallback();
+                } catch (InterruptedException ignored) {
+                } finally {
+                    done.countDown();
+                }
+            }).start();
+        }
+        start.countDown();
+        done.await();
+
+        // Under concurrent contention the CAS guard still allows exactly one callback.
+        assertThat(total.get()).isEqualTo(1);
+    }
+
 }