Repository: apache/rocketmq (Java)
Date: 2024-02-05 06:22:56+00:00
Task type: Bug Fix

Issue:
[Bug] BROADCASTING DefaultMQPushConsumer can't fix illegal pull offset
### Before Creating the Bug Report

- [X] I found a bug, not just asking a question, which should be created in [GitHub Discussions](https://github.com/apache/rocketmq/discussions).

- [X] I have searched the [GitHub Issues](https://github.com/apache/rocketmq/issues) and [GitHub Discussions](https://github.com/apache/rocketmq/discussions)  of this repository and believe that this is not a duplicate.

- [X] I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ.


### Runtime platform environment

centos7

### RocketMQ version

4.9.x | 5.1.x

### JDK Version

JDK 1.8

### Describe the Bug

<img width="2409" alt="image" src="https://github.com/apache/rocketmq/assets/103550934/d2f1a0bb-b7b6-4af4-b6ce-71b6281b4a8a">

We encountered a issue that the BROADCASTING DefaultMQPushConsumer was unable to consume: As shown in the log, the client received the OFFSET_ILLEGAL response returned by the broker and kept trying to fix nextOffset but failed, resulting in the inability to consume messages.

我们遇到了一个 Push 消费者广播模式无法消费的问题：如日志所显示，客户端收到了服务端返回的 OFFSET_ILLEGAL，一直在尝试纠正 nextOffset 但始终不能成功，导致无法消费。

### Steps to Reproduce

1. Create a topic and produce certain messages.
2. Wait for message to expire (queue minOffset >0).
3. Create a subscription group and start a DefaultMQPushConsumer (MessageModel=BROADCASTING, ConsumeFromWhere=CONSUME_FROM_FIRST_OFFSET).
4. Produce more messages.

### What Did You Expect to See?

Normal message consumption.

### What Did You See Instead?

The consumer is unable to consume any messages.

### Additional Context

<img width="1670" alt="image" src="https://github.com/apache/rocketmq/assets/103550934/471d7b87-6552-4ea9-9b11-9a2002e9e7e1">
<img width="780" alt="image" src="https://github.com/apache/rocketmq/assets/103550934/faa1b6cc-d6f2-48c6-bdc7-e153b00374d7">
<img width="1261" alt="image" src="https://github.com/apache/rocketmq/assets/103550934/a29222d4-bf7e-407d-a191-61ce3b5fa35e">

Possible cause we located after reviewing code:

When the consumer handles OFFSET_ILLEGAL error, it will first set the correct nextOffset returned by the broker as the consumerOffset (in memory), and then delete the PQ corresponding to the MQ. When the next time the rebalance is triggered, it is expected to set the new consumerOffset as the nextOffset, reinitialize PQ and start pulling messages. But at this time the consumerOffset is read from the store (ReadOffsetType=READ_FROM_STORE), so it will always  get -1. As a result, the client will fall into an infinite loop of the above process.

消费端在处理 OFFSET_ILLEGAL 时，首先会将服务端返回的正确的 nextOffset 更新为消费位点（内存），接着删除 MQ 对应的 PQ。下次重平衡触发时，预期是将新的消费位点作为 nextOffset 重新初始化 PQ 并开始拉取消息。但此时是从持久化介质读取消费位点 (ReadOffsetType=READ_FROM_STORE)，因此永远只会读到 -1。于是，客户端会陷入上述过程的无限循环中。


Issue discussion:
之前我也遇到过这个问题，可以提个 patch 修复下
> 之前我也遇到过这个问题，可以提个 patch 修复下

提了一个 patch，有时间可以帮忙一起 review 下～

Solution patch:
diff --git a/client/src/main/java/org/apache/rocketmq/client/consumer/store/LocalFileOffsetStore.java b/client/src/main/java/org/apache/rocketmq/client/consumer/store/LocalFileOffsetStore.java
--- a/client/src/main/java/org/apache/rocketmq/client/consumer/store/LocalFileOffsetStore.java
+++ b/client/src/main/java/org/apache/rocketmq/client/consumer/store/LocalFileOffsetStore.java
@@ -138,10 +138,20 @@ public long readOffset(final MessageQueue mq, final ReadOffsetType type) {
 
     @Override
     public void persistAll(Set<MessageQueue> mqs) {
-        if (null == mqs || mqs.isEmpty())
+        if (null == mqs || mqs.isEmpty()) {
             return;
+        }
+        OffsetSerializeWrapper offsetSerializeWrapper = null;
+        try {
+            offsetSerializeWrapper = readLocalOffset();
+        } catch (MQClientException e) {
+            log.error("readLocalOffset exception", e);
+            return;
+        }
 
-        OffsetSerializeWrapper offsetSerializeWrapper = new OffsetSerializeWrapper();
+        if (offsetSerializeWrapper == null) {
+            offsetSerializeWrapper = new OffsetSerializeWrapper();
+        }
         for (Map.Entry<MessageQueue, ControllableOffset> entry : this.offsetTable.entrySet()) {
             if (mqs.contains(entry.getKey())) {
                 AtomicLong offset = new AtomicLong(entry.getValue().getOffset());
@@ -161,11 +171,40 @@ public void persistAll(Set<MessageQueue> mqs) {
 
     @Override
     public void persist(MessageQueue mq) {
+        if (mq == null) {
+            return;
+        }
+        ControllableOffset offset = this.offsetTable.get(mq);
+        if (offset != null) {
+            OffsetSerializeWrapper offsetSerializeWrapper = null;
+            try {
+                offsetSerializeWrapper = readLocalOffset();
+            } catch (MQClientException e) {
+                log.error("readLocalOffset exception", e);
+                return;
+            }
+            if (offsetSerializeWrapper == null) {
+                offsetSerializeWrapper = new OffsetSerializeWrapper();
+            }
+            offsetSerializeWrapper.getOffsetTable().put(mq, new AtomicLong(offset.getOffset()));
+            String jsonString = offsetSerializeWrapper.toJson(true);
+            if (jsonString != null) {
+                try {
+                    MixAll.string2File(jsonString, this.storePath);
+                } catch (IOException e) {
+                    log.error("persist consumer offset exception, " + this.storePath, e);
+                }
+            }
+        }
     }
 
     @Override
     public void removeOffset(MessageQueue mq) {
-
+        if (mq != null) {
+            this.offsetTable.remove(mq);
+            log.info("remove unnecessary messageQueue offset. group={}, mq={}, offsetTableSize={}", this.groupName, mq,
+                offsetTable.size());
+        }
     }
 
     @Override


Test patch:
diff --git a/client/src/test/java/org/apache/rocketmq/client/consumer/store/LocalFileOffsetStoreTest.java b/client/src/test/java/org/apache/rocketmq/client/consumer/store/LocalFileOffsetStoreTest.java
--- a/client/src/test/java/org/apache/rocketmq/client/consumer/store/LocalFileOffsetStoreTest.java
+++ b/client/src/test/java/org/apache/rocketmq/client/consumer/store/LocalFileOffsetStoreTest.java
@@ -17,6 +17,7 @@
 package org.apache.rocketmq.client.consumer.store;
 
 import java.io.File;
+import java.util.Arrays;
 import java.util.Collections;
 import java.util.HashSet;
 import java.util.Map;
@@ -85,4 +86,48 @@ public void testCloneOffset() throws Exception {
         assertThat(cloneOffsetTable.size()).isEqualTo(1);
         assertThat(cloneOffsetTable.get(messageQueue)).isEqualTo(1024);
     }
-}
\ No newline at end of file
+
+    @Test
+    public void testPersist() throws Exception {
+        OffsetStore offsetStore = new LocalFileOffsetStore(mQClientFactory, group);
+
+        MessageQueue messageQueue0 = new MessageQueue(topic, brokerName, 0);
+        offsetStore.updateOffset(messageQueue0, 1024, false);
+        offsetStore.persist(messageQueue0);
+        assertThat(offsetStore.readOffset(messageQueue0, ReadOffsetType.READ_FROM_STORE)).isEqualTo(1024);
+
+        MessageQueue messageQueue1 = new MessageQueue(topic, brokerName, 1);
+        assertThat(offsetStore.readOffset(messageQueue1, ReadOffsetType.READ_FROM_STORE)).isEqualTo(-1);
+    }
+
+    @Test
+    public void testPersistAll() throws Exception {
+        OffsetStore offsetStore = new LocalFileOffsetStore(mQClientFactory, group);
+
+        MessageQueue messageQueue0 = new MessageQueue(topic, brokerName, 0);
+        offsetStore.updateOffset(messageQueue0, 1024, false);
+        offsetStore.persistAll(new HashSet<MessageQueue>(Collections.singletonList(messageQueue0)));
+        assertThat(offsetStore.readOffset(messageQueue0, ReadOffsetType.READ_FROM_STORE)).isEqualTo(1024);
+
+        MessageQueue messageQueue1 = new MessageQueue(topic, brokerName, 1);
+        MessageQueue messageQueue2 = new MessageQueue(topic, brokerName, 2);
+        offsetStore.updateOffset(messageQueue1, 1025, false);
+        offsetStore.updateOffset(messageQueue2, 1026, false);
+        offsetStore.persistAll(new HashSet<MessageQueue>(Arrays.asList(messageQueue1, messageQueue2)));
+
+        assertThat(offsetStore.readOffset(messageQueue0, ReadOffsetType.READ_FROM_STORE)).isEqualTo(1024);
+        assertThat(offsetStore.readOffset(messageQueue1, ReadOffsetType.READ_FROM_STORE)).isEqualTo(1025);
+        assertThat(offsetStore.readOffset(messageQueue2, ReadOffsetType.READ_FROM_STORE)).isEqualTo(1026);
+    }
+
+    @Test
+    public void testRemoveOffset() throws Exception {
+        OffsetStore offsetStore = new LocalFileOffsetStore(mQClientFactory, group);
+        MessageQueue messageQueue = new MessageQueue(topic, brokerName, 0);
+        offsetStore.updateOffset(messageQueue, 1024, false);
+        assertThat(offsetStore.readOffset(messageQueue, ReadOffsetType.READ_FROM_MEMORY)).isEqualTo(1024);
+
+        offsetStore.removeOffset(messageQueue);
+        assertThat(offsetStore.readOffset(messageQueue, ReadOffsetType.READ_FROM_MEMORY)).isEqualTo(-1);
+    }
+}