Repository: apache/pulsar-client-go
Pull request #1340: Fix wrong result of reader.hasNext/Next after seeking by id or time
Author: login: shibd
Labels: name: break-change
Created: 2025-02-26T04:05:43Z | Merged: 2025-03-10T06:46:21Z

There are **three attributes** to handle seek/filter/subscribe for Reader.

1. `startMessageID`: The start message ID, which can be specified by the user when creating the reader. It can be set to a specific position, or to the `earliest` or `latest`. When creating a consumer, this is [set in the request](https://github.com/apache/pulsar-client-go/blob/9aeeb4e7bbfcaa4039d66ab2857888be2d6fcaf3/pulsar/consumer_partition.go#L1979), and the broker uses it to set mark delete position.
2. `lastDequeuedMsg`: The most recent message dispatched from the consumer's internal queue to the user. **If it's nil, it means no messages have been delivered to the user. If there is a value, it indicates the user has consumed up to this point.**
3. `lastMessageInBroker`: The ID of the last message in the topic on the broker.

### Question-1: How to determine hasNext?

[Refer to this code](https://github.com/apache/pulsar-client-go/blob/9aeeb4e7bbfcaa4039d66ab2857888be2d6fcaf3/pulsar/consumer_partition.go#L2224)

In simple terms, to minimize interaction with the broker:

1. If `lastDequeuedMsg` has a value, it clearly indicates consumption up to this position. Compare this `lastDequeuedMsg with lastMessageInBroker`.
2. If there is no value, compare `startMessageID with lastMessageInBroker`.

The final comparison result, **if true**, indicates there really are messages. **If false**, it doesn’t necessarily mean there are no messages, because `lastMessageInBroker` is a cache, and the topic may have more messages already written. Therefore, **lastMessageInBroker is updated from the broker again for reevaluation**

### Question-2: What happens when the consumer reconnects?

When the consumer reconnects, it clears all messages in the `receive queue` and **updates startMessageId with the first message in the queue**. This indicates that the user has been consumed up to this position.



### !!! The above logic works well without a seek operation, but once the user calls a seek operation, the situation becomes special.

### Question-3: How does the seek operation work?

**For the broker:**
When the client sends a seek request to the broker, the broker will do the following:
1. Rewind the cursor to the seek position (ID or time).
2. Send a close request to the consumer.
3. Send a respond that the seek operation is complete.

**For the client:**
1. Send a seek request. Wait for the broker to respond that the seek is complete
2. Set `lastDequeuedMsg` to null because after seeking, this value will be invalid. **(Tip: Currently missing, this PR fixes it.)**
3. Clear the queue to avoid resetting `startMessageId` to the first position of the queue upon reconnection. **(Tip: Currently missing, this PR fixes it.)**
4. If seekByID:
    - If the passed ID is a specific position or earliest, set startMessageId to seek Id. **(Tip: Currently missing, this PR fixes it.)**
    - If the passed ID is latest, obtain the topic `lastMessageId` and set startMessageId to it. **(Tip: Currently missing, a separate PR will fix this later as it [currently doesn’t support directly passing latest id](https://github.com/apache/pulsar-client-go/blob/9aeeb4e7bbfcaa4039d66ab2857888be2d6fcaf3/pulsar/reader_impl.go#L210-L214).)**
5. If seekByTime, the client doesn’t know the messageId corresponding to the time during the seek, it needs to call getLastMessageId and set markdeleteposition to startMessageId. **(Tip: Currently missing, this PR fixes it.)**
6. Seek complete, then return to the user.
7. BTW: Actually, this is just the completion of the seek operation. The real completion requires waiting for the consumer to successfully reconnect to the broker, but this happens in the background and the user doesn't need to be concerned about it. Also, **note that we don't need to wait for the consumer to reconnect to the broker before returning the seek completion to the client**, because when the broker responds that the seek is complete, **it has already reset the cursor.** After the client receives the response, it has cleared the queue, so no previous messages will come in. In extremely rare cases (**where data from the old connection's buffer re-enters the queue after clearing**), [we will filter these messages using startMessageId to filter message ID](https://github.com/apache/pulsar-client-go/blob/9aeeb4e7bbfcaa4039d66ab2857888be2d6fcaf3/pulsar/consumer_partition.go#L1475-L1486).


### Modifications

You can run tests `TestReaderWithSeekByID` and `TestReaderWithSeekByTime` to understand the user-side issues fixed by this PR. Simply put, after seeking, calling `hasNext`, `Next` again should meet expectations.



### Verifying this change
1. Add TestReaderWithSeekByID and TestReaderWithSeekByTime to cover this change.
2. I did not change any other test, other tests passed mean not introducing any break change.


### Does this pull request potentially affect one of the following parts:
- No

### Documentation
- No


Diff:
diff --git a/pulsar/consumer.go b/pulsar/consumer.go
index 7aee9645c6..d611c6916e 100644
--- a/pulsar/consumer.go
+++ b/pulsar/consumer.go
@@ -261,6 +261,7 @@ type ConsumerOptions struct {
 	SubscriptionMode SubscriptionMode
 
 	// StartMessageIDInclusive, if true, the consumer will start at the `StartMessageID`, included.
+	// Note: This configuration also affects the seek operation.
 	// Default is `false` and the consumer will start from the "next" message
 	StartMessageIDInclusive bool
 
diff --git a/pulsar/consumer_partition.go b/pulsar/consumer_partition.go
index 24ffa401f9..057cade3b3 100644
--- a/pulsar/consumer_partition.go
+++ b/pulsar/consumer_partition.go
@@ -190,9 +190,13 @@ type partitionConsumer struct {
 	backoffPolicyFunc    func() backoff.Policy
 
 	dispatcherSeekingControlCh chan struct{}
-	isSeeking                  atomic.Bool
-	ctx                        context.Context
-	cancelFunc                 context.CancelFunc
+	// handle to the dispatcher goroutine
+	isSeeking atomic.Bool
+	// After executing seekByTime, the client is unaware of the startMessageId.
+	// Use this flag to compare markDeletePosition with BrokerLastMessageId when checking hasMoreMessages.
+	hasSoughtByTime atomic.Bool
+	ctx             context.Context
+	cancelFunc      context.CancelFunc
 }
 
 // pauseDispatchMessage used to discard the message in the dispatcher goroutine.
@@ -429,11 +433,12 @@ func newPartitionConsumer(parent Consumer, client *client, options *partitionCon
 
 	startingMessageID := pc.startMessageID.get()
 	if pc.options.startMessageIDInclusive && startingMessageID != nil && startingMessageID.equal(latestMessageID) {
-		msgID, err := pc.requestGetLastMessageID()
+		msgIDResp, err := pc.requestGetLastMessageID()
 		if err != nil {
 			pc.Close()
 			return nil, err
 		}
+		msgID := convertToMessageID(msgIDResp.GetLastMessageId())
 		if msgID.entryID != noMessageEntry {
 			pc.startMessageID.set(msgID)
 
@@ -616,18 +621,27 @@ func (pc *partitionConsumer) internalUnsubscribe(unsub *unsubscribeRequest) {
 }
 
 func (pc *partitionConsumer) getLastMessageID() (*trackingMessageID, error) {
+	res, err := pc.getLastMessageIDAndMarkDeletePosition()
+	if err != nil {
+		return nil, err
+	}
+	return res.msgID, err
+}
+
+func (pc *partitionConsumer) getLastMessageIDAndMarkDeletePosition() (*getLastMsgIDResult, error) {
 	if state := pc.getConsumerState(); state == consumerClosed || state == consumerClosing {
 		pc.log.WithField("state", state).Error("Failed to getLastMessageID for the closing or closed consumer")
 		return nil, errors.New("failed to getLastMessageID for the closing or closed consumer")
 	}
 	bo := pc.backoffPolicyFunc()
-	request := func() (*trackingMessageID, error) {
+	request := func() (*getLastMsgIDResult, error) {
 		req := &getLastMsgIDRequest{doneCh: make(chan struct{})}
 		pc.eventsCh <- req
 
 		// wait for the request to complete
 		<-req.doneCh
-		return req.msgID, req.err
+		res := &getLastMsgIDResult{req.msgID, req.markDeletePosition}
+		return res, req.err
 	}
 
 	ctx, cancel := context.WithTimeout(context.Background(), pc.client.operationTimeout)
@@ -647,10 +661,16 @@ func (pc *partitionConsumer) getLastMessageID() (*trackingMessageID, error) {
 
 func (pc *partitionConsumer) internalGetLastMessageID(req *getLastMsgIDRequest) {
 	defer close(req.doneCh)
-	req.msgID, req.err = pc.requestGetLastMessageID()
+	rsp, err := pc.requestGetLastMessageID()
+	if err != nil {
+		req.err = err
+		return
+	}
+	req.msgID = convertToMessageID(rsp.GetLastMessageId())
+	req.markDeletePosition = convertToMessageID(rsp.GetConsumerMarkDeletePosition())
 }
 
-func (pc *partitionConsumer) requestGetLastMessageID() (*trackingMessageID, error) {
+func (pc *partitionConsumer) requestGetLastMessageID() (*pb.CommandGetLastMessageIdResponse, error) {
 	if state := pc.getConsumerState(); state == consumerClosed || state == consumerClosing {
 		pc.log.WithField("state", state).Error("Failed to getLastMessageID closing or closed consumer")
 		return nil, errors.New("failed to getLastMessageID closing or closed consumer")
@@ -667,8 +687,7 @@ func (pc *partitionConsumer) requestGetLastMessageID() (*trackingMessageID, erro
 		pc.log.WithError(err).Error("Failed to get last message id")
 		return nil, err
 	}
-	id := res.Response.GetLastMessageIdResponse.GetLastMessageId()
-	return convertToMessageID(id), nil
+	return res.Response.GetLastMessageIdResponse, nil
 }
 
 func (pc *partitionConsumer) sendIndividualAck(msgID MessageID) *ackRequest {
@@ -997,7 +1016,15 @@ func (pc *partitionConsumer) requestSeek(msgID *messageID) error {
 	if err := pc.requestSeekWithoutClear(msgID); err != nil {
 		return err
 	}
-	pc.clearReceiverQueue()
+	// When the seek operation is successful, it indicates:
+	// 1. The broker has reset the cursor and sent a request to close the consumer on the client side.
+	//    Since this method is in the same goroutine as the reconnectToBroker,
+	//    we can safely clear the messages in the queue (at this point, it won't contain messages after the seek).
+	// 2. The startMessageID is reset to ensure accurate judgment when calling hasNext next time.
+	//    Since the messages in the queue are cleared here reconnection won't reset startMessageId.
+	pc.lastDequeuedMsg = nil
+	pc.startMessageID.set(toTrackingMessageID(msgID))
+	pc.clearQueueAndGetNextMessage()
 	return nil
 }
 
@@ -1069,7 +1096,9 @@ func (pc *partitionConsumer) internalSeekByTime(seek *seekByTimeRequest) {
 		seek.err = err
 		return
 	}
-	pc.clearReceiverQueue()
+	pc.lastDequeuedMsg = nil
+	pc.hasSoughtByTime.Store(true)
+	pc.clearQueueAndGetNextMessage()
 }
 
 func (pc *partitionConsumer) internalAck(req *ackRequest) {
@@ -1451,10 +1480,6 @@ func (pc *partitionConsumer) messageShouldBeDiscarded(msgID *trackingMessageID)
 	if pc.startMessageID.get() == nil {
 		return false
 	}
-	// if we start at latest message, we should never discard
-	if pc.options.startMessageID != nil && pc.options.startMessageID.equal(latestMessageID) {
-		return false
-	}
 
 	if pc.options.startMessageIDInclusive {
 		return pc.startMessageID.get().greater(msgID.messageID)
@@ -1709,9 +1734,15 @@ type redeliveryRequest struct {
 }
 
 type getLastMsgIDRequest struct {
-	doneCh chan struct{}
-	msgID  *trackingMessageID
-	err    error
+	doneCh             chan struct{}
+	msgID              *trackingMessageID
+	markDeletePosition *trackingMessageID
+	err                error
+}
+
+type getLastMsgIDResult struct {
+	msgID              *trackingMessageID
+	markDeletePosition *trackingMessageID
 }
 
 type seekRequest struct {
@@ -2195,6 +2226,25 @@ func (pc *partitionConsumer) discardCorruptedMessage(msgID *pb.MessageIdData,
 }
 
 func (pc *partitionConsumer) hasNext() bool {
+
+	// If a seek by time has been performed, then the `startMessageId` becomes irrelevant.
+	// We need to compare `markDeletePosition` and `lastMessageId`,
+	// and then reset `startMessageID` to `markDeletePosition`.
+	if pc.lastDequeuedMsg == nil && pc.hasSoughtByTime.CompareAndSwap(true, false) {
+		res, err := pc.getLastMessageIDAndMarkDeletePosition()
+		if err != nil {
+			pc.log.WithError(err).Error("Failed to get last message id")
+			pc.hasSoughtByTime.CompareAndSwap(false, true)
+			return false
+		}
+		pc.lastMessageInBroker = res.msgID
+		pc.startMessageID.set(res.markDeletePosition)
+		// We only care about comparing ledger ids and entry ids as mark delete position
+		// doesn't have other ids such as batch index
+		compareResult := pc.lastMessageInBroker.messageID.compareLedgerAndEntryID(pc.startMessageID.get().messageID)
+		return compareResult > 0 || (pc.options.startMessageIDInclusive && compareResult == 0)
+	}
+
 	if pc.lastMessageInBroker != nil && pc.hasMoreMessages() {
 		return true
 	}
@@ -2256,12 +2306,14 @@ func convertToMessageID(id *pb.MessageIdData) *trackingMessageID {
 
 	msgID := &trackingMessageID{
 		messageID: &messageID{
-			ledgerID: int64(*id.LedgerId),
-			entryID:  int64(*id.EntryId),
+			ledgerID:  int64(id.GetLedgerId()),
+			entryID:   int64(id.GetEntryId()),
+			batchIdx:  id.GetBatchIndex(),
+			batchSize: id.GetBatchSize(),
 		},
 	}
-	if id.BatchIndex != nil {
-		msgID.batchIdx = *id.BatchIndex
+	if msgID.batchIdx == -1 {
+		msgID.batchIdx = 0
 	}
 
 	return msgID
diff --git a/pulsar/consumer_test.go b/pulsar/consumer_test.go
index 33e4d057ee..8e5c2b84e2 100644
--- a/pulsar/consumer_test.go
+++ b/pulsar/consumer_test.go
@@ -1262,8 +1262,9 @@ func TestConsumerSeek(t *testing.T) {
 	defer producer.Close()
 
 	consumer, err := client.Subscribe(ConsumerOptions{
-		Topic:            topicName,
-		SubscriptionName: "sub-1",
+		Topic:                   topicName,
+		SubscriptionName:        "sub-1",
+		StartMessageIDInclusive: true,
 	})
 	assert.Nil(t, err)
 	defer consumer.Close()
diff --git a/pulsar/consumer_zero_queue_test.go b/pulsar/consumer_zero_queue_test.go
index 34e9df9f20..2e7c4b275b 100644
--- a/pulsar/consumer_zero_queue_test.go
+++ b/pulsar/consumer_zero_queue_test.go
@@ -474,6 +474,7 @@ func TestZeroQueueConsumer_Seek(t *testing.T) {
 		Topic:                   topicName,
 		EnableZeroQueueConsumer: true,
 		SubscriptionName:        "sub-1",
+		StartMessageIDInclusive: true,
 	})
 	assert.Nil(t, err)
 	_, ok := consumer.(*zeroQueueConsumer)
diff --git a/pulsar/impl_message.go b/pulsar/impl_message.go
index 0acd782b80..f32cc7fc61 100644
--- a/pulsar/impl_message.go
+++ b/pulsar/impl_message.go
@@ -18,6 +18,7 @@
 package pulsar
 
 import (
+	"cmp"
 	"errors"
 	"fmt"
 	"math"
@@ -147,6 +148,13 @@ func (id *messageID) equal(other *messageID) bool {
 		id.batchIdx == other.batchIdx
 }
 
+func (id *messageID) compareLedgerAndEntryID(other *messageID) int {
+	if result := cmp.Compare(id.ledgerID, other.ledgerID); result != 0 {
+		return result
+	}
+	return cmp.Compare(id.entryID, other.entryID)
+}
+
 func (id *messageID) greaterEqual(other *messageID) bool {
 	return id.equal(other) || id.greater(other)
 }
diff --git a/pulsar/reader.go b/pulsar/reader.go
index 98bde4e395..95aed0e567 100644
--- a/pulsar/reader.go
+++ b/pulsar/reader.go
@@ -53,6 +53,7 @@ type ReaderOptions struct {
 	StartMessageID MessageID
 
 	// StartMessageIDInclusive, if true, the reader will start at the `StartMessageID`, included.
+	// Note: This configuration also affects the seek operation.
 	// Default is `false` and the reader will start from the "next" message
 	StartMessageIDInclusive bool
 
diff --git a/pulsar/reader_impl.go b/pulsar/reader_impl.go
index f76255e2e8..55b05037f7 100644
--- a/pulsar/reader_impl.go
+++ b/pulsar/reader_impl.go
@@ -196,19 +196,6 @@ func (r *reader) Close() {
 	r.metrics.ReadersClosed.Inc()
 }
 
-func (r *reader) messageID(msgID MessageID) *trackingMessageID {
-	mid := toTrackingMessageID(msgID)
-
-	partition := int(mid.partitionIdx)
-	// did we receive a valid partition index?
-	if partition < 0 {
-		r.log.Warnf("invalid partition index %d expected", partition)
-		return nil
-	}
-
-	return mid
-}
-
 func (r *reader) Seek(msgID MessageID) error {
 	r.Lock()
 	defer r.Unlock()
@@ -218,9 +205,12 @@ func (r *reader) Seek(msgID MessageID) error {
 		return fmt.Errorf("invalid message id type %T", msgID)
 	}
 
-	mid := r.messageID(msgID)
-	if mid == nil {
-		return nil
+	mid := toTrackingMessageID(msgID)
+
+	partition := int(mid.partitionIdx)
+	if partition < 0 {
+		r.log.Warnf("invalid partition index %d expected", partition)
+		return fmt.Errorf("seek msgId must include partitoinIndex")
 	}
 
 	return r.c.Seek(mid)
diff --git a/pulsar/reader_test.go b/pulsar/reader_test.go
index 836535704b..2fbe89e458 100644
--- a/pulsar/reader_test.go
+++ b/pulsar/reader_test.go
@@ -1070,3 +1070,170 @@ func TestReaderNextReturnsOnClosedConsumer(t *testing.T) {
 	assert.ErrorAs(t, err, &e)
 	assert.Equal(t, ConsumerClosed, e.Result())
 }
+
+func testReaderSeekByIDWithHasNext(t *testing.T, startMessageID MessageID, startMessageIDInclusive bool) {
+	client, err := NewClient(ClientOptions{
+		URL: lookupURL,
+	})
+
+	assert.Nil(t, err)
+	defer client.Close()
+
+	topic := newTopicName()
+	ctx := context.Background()
+
+	// create producer
+	producer, err := client.CreateProducer(ProducerOptions{
+		Topic:           topic,
+		DisableBatching: true,
+	})
+	assert.Nil(t, err)
+	defer producer.Close()
+
+	// send 100 messages
+	var lastMsgID MessageID
+	for i := 0; i < 10; i++ {
+		lastMsgID, err = producer.Send(ctx, &ProducerMessage{
+			Payload: []byte(fmt.Sprintf("hello-%d", i)),
+		})
+		assert.NoError(t, err)
+		assert.NotNil(t, lastMsgID)
+	}
+
+	reader, err := client.CreateReader(ReaderOptions{
+		Topic:                   topic,
+		StartMessageID:          startMessageID,
+		StartMessageIDInclusive: startMessageIDInclusive,
+	})
+	assert.Nil(t, err)
+	defer reader.Close()
+
+	// Seek to last message ID
+	err = reader.Seek(lastMsgID)
+	assert.NoError(t, err)
+
+	if startMessageIDInclusive {
+		assert.True(t, reader.HasNext())
+		ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
+		msg, err := reader.Next(ctx)
+		assert.NoError(t, err)
+		assert.NotNil(t, msg)
+		assert.True(t, messageIDCompare(lastMsgID, msg.ID()) == 0)
+		cancel()
+	} else {
+		assert.False(t, reader.HasNext())
+		ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
+		msg, err := reader.Next(ctx)
+		assert.Error(t, err)
+		assert.Nil(t, msg)
+		cancel()
+	}
+
+}
+
+func TestReaderWithSeekByID(t *testing.T) {
+	params := []struct {
+		messageID               MessageID
+		startMessageIDInclusive bool
+	}{
+		{EarliestMessageID(), false},
+		{EarliestMessageID(), true},
+		{LatestMessageID(), false},
+		{LatestMessageID(), true},
+	}
+
+	for _, c := range params {
+		t.Run(fmt.Sprintf("TestReaderSeekByID_%v_%v", c.messageID, c.startMessageIDInclusive),
+			func(t *testing.T) {
+				testReaderSeekByIDWithHasNext(t, c.messageID, c.startMessageIDInclusive)
+			})
+	}
+}
+
+func testReaderSeekByTimeWithHasNext(t *testing.T, startMessageID MessageID) {
+	client, err := NewClient(ClientOptions{
+		URL: lookupURL,
+	})
+
+	assert.Nil(t, err)
+	defer client.Close()
+
+	topic := newTopicName()
+	ctx := context.Background()
+
+	// create producer
+	producer, err := client.CreateProducer(ProducerOptions{
+		Topic:           topic,
+		DisableBatching: true,
+	})
+	assert.Nil(t, err)
+	defer producer.Close()
+
+	// 1. send 10 messages
+	var lastMsgID MessageID
+	for i := 0; i < 10; i++ {
+		lastMsgID, err = producer.Send(ctx, &ProducerMessage{
+			Payload: []byte(fmt.Sprintf("hello-%d", i)),
+		})
+		assert.NoError(t, err)
+
+		assert.NotNil(t, lastMsgID)
+	}
+
+	// 2. create reader
+	reader, err := client.CreateReader(ReaderOptions{
+		Topic:                   topic,
+		StartMessageID:          startMessageID,
+		StartMessageIDInclusive: false,
+	})
+	assert.Nil(t, err)
+	defer reader.Close()
+
+	// 3. Seek time to now
+	reader.SeekByTime(time.Now())
+
+	// 4. Should not receive msg
+	{
+		assert.False(t, reader.HasNext())
+		timeoutCtx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
+		msg, err := reader.Next(timeoutCtx)
+		assert.Error(t, err)
+		assert.Nil(t, msg)
+		cancel()
+	}
+
+	// 5. send more 10 messages
+	for i := 0; i < 10; i++ {
+		lastMsgID, err = producer.Send(ctx, &ProducerMessage{
+			Payload: []byte(fmt.Sprintf("hello2-%d", i)),
+		})
+		assert.NoError(t, err)
+		assert.NotNil(t, lastMsgID)
+	}
+
+	// 6. Assert these messages are received
+	for i := 0; i < 10; i++ {
+		assert.True(t, reader.HasNext())
+		msg, err := reader.Next(context.Background())
+		assert.NoError(t, err)
+		assert.Equal(t, fmt.Sprintf("hello2-%d", i), string(msg.Payload()))
+	}
+
+	// assert not more msg
+	{
+		assert.False(t, reader.HasNext())
+		timeoutCtx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
+		msg, err := reader.Next(timeoutCtx)
+		assert.Error(t, err)
+		assert.Nil(t, msg)
+		cancel()
+	}
+}
+func TestReaderWithSeekByTime(t *testing.T) {
+	startMessageIDs := []MessageID{EarliestMessageID(), LatestMessageID()}
+	for _, startMsgID := range startMessageIDs {
+		t.Run(fmt.Sprintf("TestReaderSeekByTime_%v", startMsgID), func(t *testing.T) {
+			testReaderSeekByTimeWithHasNext(t, startMsgID)
+		})
+	}
+}


Reviews:
user: login: BewareMyPower
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-27T04:29:52Z
user: login: BewareMyPower
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-27T04:45:39Z
user: login: BewareMyPower
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-27T05:00:10Z
user: login: BewareMyPower
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-27T05:39:29Z
user: login: BewareMyPower
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-27T06:04:36Z
user: login: shibd
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-27T07:07:53Z
user: login: shibd
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-27T07:10:06Z
user: login: shibd
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-27T07:11:52Z
user: login: shibd
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-27T07:15:36Z
user: login: shibd
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-27T07:15:41Z
user: login: BewareMyPower
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T03:33:04Z
user: login: BewareMyPower
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T03:39:15Z
user: login: BewareMyPower
body: If any user depends on the existing `seek` behavior (inclusive by default), he/she might treat it as a "message lost" issue.

This behavior change is beyond the scope of this PR. I also think it needs a discussion at the mail list.  We also need to document the change and tell existing users to set `startMessageInclusive` with true to keep the old behavior. But it's still a breaking change because if `startMessageInclusive` is true, the behavior of subscribing also changes.

Therefore, my opinion is to keep the existing seek behavior not changed for now. The "fix" could be delayed to next release.
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T03:44:18Z
user: login: BewareMyPower
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T03:47:36Z
user: login: shibd
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-06T12:11:25Z
user: login: shibd
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-06T12:14:10Z
user: login: BewareMyPower
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T08:56:04Z
user: login: nodece
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-07T16:48:22Z
user: login: BewareMyPower
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-08T15:03:14Z
user: login: shibd
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-09T03:10:07Z
user: login: nodece
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-09T15:54:39Z
user: login: nodece
body: Please address the batch index issue.
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-03-09T15:55:32Z
user: login: shibd
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-10T00:05:21Z
user: login: nodece
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-10T04:40:32Z
user: login: nodece
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-10T06:43:47Z
user: login: nodece
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-10T06:46:06Z

Review comments:
diff_hunk: @@ -1070,3 +1070,159 @@ func TestReaderNextReturnsOnClosedConsumer(t *testing.T) {
 	assert.ErrorAs(t, err, &e)
 	assert.Equal(t, ConsumerClosed, e.Result())
 }
+
+func testReaderSeekByIDWithHasNext(t *testing.T, startMessageID MessageID, startMessageIDInclusive bool) {
+	client, err := NewClient(ClientOptions{
+		URL: lookupURL,
+	})
+
+	assert.Nil(t, err)
+	defer client.Close()
+
+	topic := newTopicName()
+	ctx := context.Background()
+
+	// create producer
+	producer, err := client.CreateProducer(ProducerOptions{
+		Topic:           topic,
+		DisableBatching: true,
+	})
+	assert.Nil(t, err)
+	defer producer.Close()
+
+	// send 100 messages
+	var lastMsgID MessageID
+	for i := 0; i < 10; i++ {
+		lastMsgID, err = producer.Send(ctx, &ProducerMessage{
+			Payload: []byte(fmt.Sprintf("hello-%d", i)),
+		})
+		fmt.Println(lastMsgID.String())
path: pulsar/reader_test.go
user: login: BewareMyPower
body: Don't use `fmt.Println` in tests
created_at: 2025-02-27T04:29:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1070,3 +1070,159 @@ func TestReaderNextReturnsOnClosedConsumer(t *testing.T) {
 	assert.ErrorAs(t, err, &e)
 	assert.Equal(t, ConsumerClosed, e.Result())
 }
+
+func testReaderSeekByIDWithHasNext(t *testing.T, startMessageID MessageID, startMessageIDInclusive bool) {
+	client, err := NewClient(ClientOptions{
+		URL: lookupURL,
+	})
+
+	assert.Nil(t, err)
+	defer client.Close()
+
+	topic := newTopicName()
+	ctx := context.Background()
+
+	// create producer
+	producer, err := client.CreateProducer(ProducerOptions{
+		Topic:           topic,
+		DisableBatching: true,
+	})
+	assert.Nil(t, err)
+	defer producer.Close()
+
+	// send 100 messages
+	var lastMsgID MessageID
+	for i := 0; i < 10; i++ {
+		lastMsgID, err = producer.Send(ctx, &ProducerMessage{
+			Payload: []byte(fmt.Sprintf("hello-%d", i)),
+		})
+		fmt.Println(lastMsgID.String())
+		assert.NoError(t, err)
+		assert.NotNil(t, lastMsgID)
+	}
+
+	reader, err := client.CreateReader(ReaderOptions{
+		Topic:                   topic,
+		StartMessageID:          startMessageID,
+		StartMessageIDInclusive: startMessageIDInclusive,
+	})
+	assert.Nil(t, err)
+	defer reader.Close()
+
+	// Seek to last message ID
+	err = reader.Seek(lastMsgID)
+	assert.NoError(t, err)
+
+	if startMessageIDInclusive {
+		assert.True(t, reader.HasNext())
+		ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
+		msg, err := reader.Next(ctx)
+		assert.NoError(t, err)
+		assert.NotNil(t, msg)
+		assert.True(t, messageIDCompare(lastMsgID, msg.ID()) == 0)
+		cancel()
+	} else {
+		assert.False(t, reader.HasNext())
+		ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
+		msg, err := reader.Next(ctx)
+		assert.Error(t, err)
+		assert.Nil(t, msg)
+		cancel()
+	}
+
+}
+
+func TestReaderWithSeekByID(t *testing.T) {
+	testReaderSeekByIDWithHasNext(t, EarliestMessageID(), false)
+	testReaderSeekByIDWithHasNext(t, EarliestMessageID(), true)
+	testReaderSeekByIDWithHasNext(t, LatestMessageID(), false)
+	testReaderSeekByIDWithHasNext(t, LatestMessageID(), true)
+}
+
+func testReaderSeekByTimeWithHasNext(t *testing.T, startMessageID MessageID) {
+	client, err := NewClient(ClientOptions{
+		URL: lookupURL,
+	})
+
+	assert.Nil(t, err)
+	defer client.Close()
+
+	topic := newTopicName()
+	ctx := context.Background()
+
+	// create producer
+	producer, err := client.CreateProducer(ProducerOptions{
+		Topic:           topic,
+		DisableBatching: true,
+	})
+	assert.Nil(t, err)
+	defer producer.Close()
+
+	// 1. send 10 messages
+	var lastMsgID MessageID
+	for i := 0; i < 10; i++ {
+		lastMsgID, err = producer.Send(ctx, &ProducerMessage{
+			Payload: []byte(fmt.Sprintf("hello-%d", i)),
+		})
+		fmt.Println(lastMsgID.String())
+		assert.NoError(t, err)
+
+		assert.NotNil(t, lastMsgID)
+	}
+
+	// 2. create reader
+	reader, err := client.CreateReader(ReaderOptions{
+		Topic:                   topic,
+		StartMessageID:          startMessageID,
+		StartMessageIDInclusive: false,
+	})
+	assert.Nil(t, err)
+	defer reader.Close()
+
+	// 3. Seek time to now
+	reader.SeekByTime(time.Now())
+
+	// 4. Should not receive msg
+	{
+		assert.False(t, reader.HasNext())
+		timeoutCtx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
+		msg, err := reader.Next(timeoutCtx)
+		assert.Error(t, err)
+		assert.Nil(t, msg)
+		cancel()
+	}
+
+	// 5. send more 10 messages
+	for i := 0; i < 10; i++ {
+		lastMsgID, err = producer.Send(ctx, &ProducerMessage{
+			Payload: []byte(fmt.Sprintf("hello2-%d", i)),
+		})
+		fmt.Println(lastMsgID.String())
+		assert.NoError(t, err)
+		assert.NotNil(t, lastMsgID)
+	}
+
+	// 6. Assert these messages are received
+	for i := 0; i < 10; i++ {
+		assert.True(t, reader.HasNext())
+		msg, err := reader.Next(context.Background())
+		assert.NoError(t, err)
+		assert.Equal(t, fmt.Sprintf("hello2-%d", i), string(msg.Payload()))
+	}
+
+	// assert not more msg
+	{
+		assert.False(t, reader.HasNext())
+		timeoutCtx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
+		msg, err := reader.Next(timeoutCtx)
+		assert.Error(t, err)
+		assert.Nil(t, msg)
+		cancel()
+	}
+}
+func TestReaderWithSeekByTime(t *testing.T) {
+	testReaderSeekByTimeWithHasNext(t, EarliestMessageID())
+	testReaderSeekByTimeWithHasNext(t, EarliestMessageID())
+	testReaderSeekByTimeWithHasNext(t, LatestMessageID())
+	testReaderSeekByTimeWithHasNext(t, LatestMessageID())
path: pulsar/reader_test.go
user: login: BewareMyPower
body: I don't get why did you run twice for each call? BTW, for parameterized tests, you can call `T.Run` so that each single sub-test can have its own test name and fail independently. See example here https://github.com/apache/pulsar-client-go/blob/020b55c1dce4475fe8781f4fb6f858de903844b8/pulsar/consumer_test.go#L4760
created_at: 2025-02-27T04:45:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1262,8 +1262,9 @@ func TestConsumerSeek(t *testing.T) {
 	defer producer.Close()
 
 	consumer, err := client.Subscribe(ConsumerOptions{
-		Topic:            topicName,
-		SubscriptionName: "sub-1",
+		Topic:                   topicName,
+		SubscriptionName:        "sub-1",
+		StartMessageIDInclusive: true,
path: pulsar/consumer_test.go
user: login: BewareMyPower
body: This change seems irrelevant?
created_at: 2025-02-27T04:59:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -474,6 +474,7 @@ func TestZeroQueueConsumer_Seek(t *testing.T) {
 		Topic:                   topicName,
 		EnableZeroQueueConsumer: true,
 		SubscriptionName:        "sub-1",
+		StartMessageIDInclusive: true,
path: pulsar/consumer_zero_queue_test.go
user: login: BewareMyPower
body: irrelevant change again
created_at: 2025-02-27T05:00:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -997,7 +1016,15 @@ func (pc *partitionConsumer) requestSeek(msgID *messageID) error {
 	if err := pc.requestSeekWithoutClear(msgID); err != nil {
 		return err
 	}
-	pc.clearReceiverQueue()
+	// When the seek operation is successful, it indicates:
+	// 1. The broker has reset the cursor and sent a request to close the consumer on the client side.
+	//    Since this method is in the same goroutine as the reconnectToBroker,
+	//    we can safely clear the messages in the queue (at this point, it won't contain messages after the seek).
+	// 2. The startMessageID is reset to ensure accurate judgment when calling hasNext next time.
+	//    Since the messages in the queue are cleared here reconnection won't reset startMessageId.
+	pc.lastDequeuedMsg = nil
path: pulsar/consumer_partition.go
user: login: BewareMyPower
body: I have a concern that if there is a race here. Before this change, `lastDequeuedMsg` is only updated by `setLastDequeuedMsg`, which is only called by the `Next` method called in user's goroutine. However, after that, `lastDequeuedMsg` could be reset with nil in the event loop goroutine (`runEventsLoop -> internalSeek -> requestSeek`)
created_at: 2025-02-27T05:39:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2195,6 +2226,24 @@ func (pc *partitionConsumer) discardCorruptedMessage(msgID *pb.MessageIdData,
 }
 
 func (pc *partitionConsumer) hasNext() bool {
+
+	// If a seek by time has been performed, then the `startMessageId` becomes irrelevant.
+	// We need to compare `markDeletePosition` and `lastMessageId`,
+	// and then reset `startMessageID` to `markDeletePosition`.
+	if pc.hasSoughtByTime.CompareAndSwap(true, false) {
path: pulsar/consumer_partition.go
user: login: BewareMyPower
body: This CAS operation seems confusing.

1. `hasSoughtByTime` is set with true in `internalSeekByTime`, so concurrent seek could happen. In Java client, it's prevented by `seekStatus.compareAndSet(SeekStatus.NOT_STARTED, SeekStatus.IN_PROGRESS)`.
2. Here it's changed from true to false, which means next time `hasNext` is called, it will skip the code block here.
created_at: 2025-02-27T06:04:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -997,7 +1016,15 @@ func (pc *partitionConsumer) requestSeek(msgID *messageID) error {
 	if err := pc.requestSeekWithoutClear(msgID); err != nil {
 		return err
 	}
-	pc.clearReceiverQueue()
+	// When the seek operation is successful, it indicates:
+	// 1. The broker has reset the cursor and sent a request to close the consumer on the client side.
+	//    Since this method is in the same goroutine as the reconnectToBroker,
+	//    we can safely clear the messages in the queue (at this point, it won't contain messages after the seek).
+	// 2. The startMessageID is reset to ensure accurate judgment when calling hasNext next time.
+	//    Since the messages in the queue are cleared here reconnection won't reset startMessageId.
+	pc.lastDequeuedMsg = nil
path: pulsar/consumer_partition.go
user: login: shibd
body: When executing seek, will clear `messageChan` first and then `pauseDispatchMessage` message until the seek operator is complete(reconnect success)

Refer to this PR:

https://github.com/apache/pulsar-client-go/pull/1265
created_at: 2025-02-27T07:07:53Z
author_association: MEMBER
diff_hunk: @@ -2195,6 +2226,24 @@ func (pc *partitionConsumer) discardCorruptedMessage(msgID *pb.MessageIdData,
 }
 
 func (pc *partitionConsumer) hasNext() bool {
+
+	// If a seek by time has been performed, then the `startMessageId` becomes irrelevant.
+	// We need to compare `markDeletePosition` and `lastMessageId`,
+	// and then reset `startMessageID` to `markDeletePosition`.
+	if pc.hasSoughtByTime.CompareAndSwap(true, false) {
path: pulsar/consumer_partition.go
user: login: shibd
body: > Here it's changed from true to false, which means next time hasNext is called, it will skip the code block here.

Yes, After executing `seekByTime`, we only need to enter this method once, as it updates startMessageID to markDeletePosition. Now, it clearly knows what the messageId was from the last execution of seekByTime
created_at: 2025-02-27T07:10:06Z
author_association: MEMBER
diff_hunk: @@ -2195,6 +2226,24 @@ func (pc *partitionConsumer) discardCorruptedMessage(msgID *pb.MessageIdData,
 }
 
 func (pc *partitionConsumer) hasNext() bool {
+
+	// If a seek by time has been performed, then the `startMessageId` becomes irrelevant.
+	// We need to compare `markDeletePosition` and `lastMessageId`,
+	// and then reset `startMessageID` to `markDeletePosition`.
+	if pc.hasSoughtByTime.CompareAndSwap(true, false) {
path: pulsar/consumer_partition.go
user: login: shibd
body: However, we need to add a new condition: when `&& lastDequeuedMsg == nil`. This is because the user might have received the message but hasn't called this `HasNext`.
created_at: 2025-02-27T07:11:52Z
author_association: MEMBER
diff_hunk: @@ -474,6 +474,7 @@ func TestZeroQueueConsumer_Seek(t *testing.T) {
 		Topic:                   topicName,
 		EnableZeroQueueConsumer: true,
 		SubscriptionName:        "sub-1",
+		StartMessageIDInclusive: true,
path: pulsar/consumer_zero_queue_test.go
user: login: shibd
body: This is necessary, Because this test seeks to N−5 and expects to receive this message next time. 

In fact, StartMessageIDInclusive should also apply to the seek operation.
created_at: 2025-02-27T07:15:36Z
author_association: MEMBER
diff_hunk: @@ -1262,8 +1262,9 @@ func TestConsumerSeek(t *testing.T) {
 	defer producer.Close()
 
 	consumer, err := client.Subscribe(ConsumerOptions{
-		Topic:            topicName,
-		SubscriptionName: "sub-1",
+		Topic:                   topicName,
+		SubscriptionName:        "sub-1",
+		StartMessageIDInclusive: true,
path: pulsar/consumer_test.go
user: login: shibd
body: This is necessary, Because this test seeks to N−50 and expects to receive this message next time.

In fact, StartMessageIDInclusive should also apply to the seek operation.


created_at: 2025-02-27T07:15:41Z
author_association: MEMBER
diff_hunk: @@ -2195,6 +2226,24 @@ func (pc *partitionConsumer) discardCorruptedMessage(msgID *pb.MessageIdData,
 }
 
 func (pc *partitionConsumer) hasNext() bool {
+
+	// If a seek by time has been performed, then the `startMessageId` becomes irrelevant.
+	// We need to compare `markDeletePosition` and `lastMessageId`,
+	// and then reset `startMessageID` to `markDeletePosition`.
+	if pc.lastDequeuedMsg == nil && pc.hasSoughtByTime.CompareAndSwap(true, false) {
+		res, err := pc.getLastMessageIDAndMarkDeletePosition()
+		if err != nil {
+			pc.log.WithError(err).Error("Failed to get last message id")
path: pulsar/consumer_partition.go
user: login: BewareMyPower
body: In this case, `hasSoughtByTime` will still be `false`. Should we restore it with true here?
created_at: 2025-03-04T03:32:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1262,8 +1262,9 @@ func TestConsumerSeek(t *testing.T) {
 	defer producer.Close()
 
 	consumer, err := client.Subscribe(ConsumerOptions{
-		Topic:            topicName,
-		SubscriptionName: "sub-1",
+		Topic:                   topicName,
+		SubscriptionName:        "sub-1",
+		StartMessageIDInclusive: true,
path: pulsar/consumer_test.go
user: login: BewareMyPower
body: Anyway, this is a breaking change. Though the existing behavior might not be expected, users might already depend on this behavior.

Unlike `hasNext()`, the existing behavior is that if a consumer seeks to the message id, it will still consume that message. While this PR's main purpose is to resolve the wrong result of `hasNext()`. I think you should use a different PR for changing the `seek` behavior.
created_at: 2025-03-04T03:39:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1262,8 +1262,9 @@ func TestConsumerSeek(t *testing.T) {
 	defer producer.Close()
 
 	consumer, err := client.Subscribe(ConsumerOptions{
-		Topic:            topicName,
-		SubscriptionName: "sub-1",
+		Topic:                   topicName,
+		SubscriptionName:        "sub-1",
+		StartMessageIDInclusive: true,
path: pulsar/consumer_test.go
user: login: BewareMyPower
body: Unlike `hasNext()`, if it returns true and there is no more message, the application will be stuck at `Next`.

However, the existing behavior of `seek` won't cause such serious result. Indeed, some users might think it's a special behavior of Pulsar Go client and set the message id with the inclusive semantics.
created_at: 2025-03-04T03:47:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1262,8 +1262,9 @@ func TestConsumerSeek(t *testing.T) {
 	defer producer.Close()
 
 	consumer, err := client.Subscribe(ConsumerOptions{
-		Topic:            topicName,
-		SubscriptionName: "sub-1",
+		Topic:                   topicName,
+		SubscriptionName:        "sub-1",
+		StartMessageIDInclusive: true,
path: pulsar/consumer_test.go
user: login: shibd
body: I do not agree that this is a break change; it is more like a behavior fix or bug fix.

After this PR, it will keep the same behavior with the java client. Refer to(peek to id-5 and receive msg from id-6):
https://github.com/apache/pulsar/blob/a1f3322ed358ab6841f0d3e43f2afcc54788b887/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/SubscriptionSeekTest.java#L142-L165



created_at: 2025-03-06T12:11:24Z
author_association: MEMBER
diff_hunk: @@ -2195,6 +2226,24 @@ func (pc *partitionConsumer) discardCorruptedMessage(msgID *pb.MessageIdData,
 }
 
 func (pc *partitionConsumer) hasNext() bool {
+
+	// If a seek by time has been performed, then the `startMessageId` becomes irrelevant.
+	// We need to compare `markDeletePosition` and `lastMessageId`,
+	// and then reset `startMessageID` to `markDeletePosition`.
+	if pc.lastDequeuedMsg == nil && pc.hasSoughtByTime.CompareAndSwap(true, false) {
+		res, err := pc.getLastMessageIDAndMarkDeletePosition()
+		if err != nil {
+			pc.log.WithError(err).Error("Failed to get last message id")
path: pulsar/consumer_partition.go
user: login: shibd
body: Yes, it should actually return an error, but the API wasn't designed that way. Of course, I'll set it to true and let the next call continue with this logic. 
created_at: 2025-03-06T12:14:10Z
author_association: MEMBER
diff_hunk: @@ -2256,12 +2306,14 @@ func convertToMessageID(id *pb.MessageIdData) *trackingMessageID {
 
 	msgID := &trackingMessageID{
 		messageID: &messageID{
-			ledgerID: int64(*id.LedgerId),
-			entryID:  int64(*id.EntryId),
+			ledgerID:  int64(id.GetLedgerId()),
+			entryID:   int64(id.GetEntryId()),
+			batchIdx:  id.GetBatchIndex(),
+			batchSize: id.GetBatchSize(),
 		},
 	}
-	if id.BatchIndex != nil {
-		msgID.batchIdx = *id.BatchIndex
+	if msgID.batchIdx == -1 {
+		msgID.batchIdx = 0
path: pulsar/consumer_partition.go
user: login: nodece
body: Why reset this value to 0? -1 is not the batch message, 0 is batch.
created_at: 2025-03-07T16:48:22Z
author_association: MEMBER
diff_hunk: @@ -2256,12 +2306,14 @@ func convertToMessageID(id *pb.MessageIdData) *trackingMessageID {
 
 	msgID := &trackingMessageID{
 		messageID: &messageID{
-			ledgerID: int64(*id.LedgerId),
-			entryID:  int64(*id.EntryId),
+			ledgerID:  int64(id.GetLedgerId()),
+			entryID:   int64(id.GetEntryId()),
+			batchIdx:  id.GetBatchIndex(),
+			batchSize: id.GetBatchSize(),
 		},
 	}
-	if id.BatchIndex != nil {
-		msgID.batchIdx = *id.BatchIndex
+	if msgID.batchIdx == -1 {
+		msgID.batchIdx = 0
path: pulsar/consumer_partition.go
user: login: BewareMyPower
body: Sounds like a concern. Could you provide a test case to show this change might not work?
created_at: 2025-03-08T15:03:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2256,12 +2306,14 @@ func convertToMessageID(id *pb.MessageIdData) *trackingMessageID {
 
 	msgID := &trackingMessageID{
 		messageID: &messageID{
-			ledgerID: int64(*id.LedgerId),
-			entryID:  int64(*id.EntryId),
+			ledgerID:  int64(id.GetLedgerId()),
+			entryID:   int64(id.GetEntryId()),
+			batchIdx:  id.GetBatchIndex(),
+			batchSize: id.GetBatchSize(),
 		},
 	}
-	if id.BatchIndex != nil {
-		msgID.batchIdx = *id.BatchIndex
+	if msgID.batchIdx == -1 {
+		msgID.batchIdx = 0
path: pulsar/consumer_partition.go
user: login: shibd
body: In the Go Client, it's not like this. You can run the following code to test:

Need change msg id String method first.
``` go
func (id *messageID) String() string {
	return fmt.Sprintf("%d:%d:%d:%d", id.ledgerID, id.entryID, id.batchIdx, id.partitionIdx)
}
```


``` go
func TestMessageID_BatchIdx(t *testing.T) {
	client, err := NewClient(ClientOptions{
		URL: lookupURL,
	})

	assert.Nil(t, err)
	defer client.Close()

	topic := newTopicName()
	ctx := context.Background()

	// create producer
	producer, err := client.CreateProducer(ProducerOptions{
		Topic:           topic,
		DisableBatching: true,
	})
	assert.Nil(t, err)
	defer producer.Close()

	// create consumer
	consumer, err := client.Subscribe(ConsumerOptions{
		Topic:             topic,
		SubscriptionName:  "my-sub",
		Type:              Exclusive,
		ReceiverQueueSize: 10,
	})
	assert.Nil(t, err)
	defer consumer.Close()

	// send 10 messages with no batch
	var lastMsgID MessageID
	for i := 0; i < 10; i++ {
		lastMsgID, err = producer.Send(ctx, &ProducerMessage{
			Payload: []byte(fmt.Sprintf("hello-%d", i)),
		})
		t.Logf("Sended message msgId: %#v \n", lastMsgID.String())
		assert.NoError(t, err)
		assert.NotNil(t, lastMsgID)
	}

	// Receive 10 message with no batch
	for i := 0; i < 10; i++ {
		msg, err := consumer.Receive(ctx)
		assert.Nil(t, err)
		t.Logf("Received message msgId: %#v \n", msg.ID().String())
		consumer.Ack(msg)
	}
}

output:
   consumer_test.go:572: Sended message msgId: "3141:0:0:0" 
    consumer_test.go:572: Sended message msgId: "3141:1:0:0" 
    consumer_test.go:572: Sended message msgId: "3141:2:0:0" 
    consumer_test.go:572: Sended message msgId: "3141:3:0:0" 
    consumer_test.go:572: Sended message msgId: "3141:4:0:0" 
    consumer_test.go:572: Sended message msgId: "3141:5:0:0" 
    consumer_test.go:572: Sended message msgId: "3141:6:0:0" 
    consumer_test.go:572: Sended message msgId: "3141:7:0:0" 
    consumer_test.go:572: Sended message msgId: "3141:8:0:0" 
    consumer_test.go:572: Sended message msgId: "3141:9:0:0" 
    consumer_test.go:581: Received message msgId: "3141:0:0:0" 
    consumer_test.go:581: Received message msgId: "3141:1:0:0" 
    consumer_test.go:581: Received message msgId: "3141:2:0:0" 
    consumer_test.go:581: Received message msgId: "3141:3:0:0" 
    consumer_test.go:581: Received message msgId: "3141:4:0:0" 
    consumer_test.go:581: Received message msgId: "3141:5:0:0" 
    consumer_test.go:581: Received message msgId: "3141:6:0:0" 
    consumer_test.go:581: Received message msgId: "3141:7:0:0" 
    consumer_test.go:581: Received message msgId: "3141:8:0:0" 
    consumer_test.go:581: Received message msgId: "3141:9:0:0" 
```

In Java, it will be a different implementation class, so there's no batchIndex field when disabled batch.

If we want to keep it consistent, we can change it later.
created_at: 2025-03-09T03:10:07Z
author_association: MEMBER
diff_hunk: @@ -2256,12 +2306,14 @@ func convertToMessageID(id *pb.MessageIdData) *trackingMessageID {
 
 	msgID := &trackingMessageID{
 		messageID: &messageID{
-			ledgerID: int64(*id.LedgerId),
-			entryID:  int64(*id.EntryId),
+			ledgerID:  int64(id.GetLedgerId()),
+			entryID:   int64(id.GetEntryId()),
+			batchIdx:  id.GetBatchIndex(),
+			batchSize: id.GetBatchSize(),
 		},
 	}
-	if id.BatchIndex != nil {
-		msgID.batchIdx = *id.BatchIndex
+	if msgID.batchIdx == -1 {
+		msgID.batchIdx = 0
path: pulsar/consumer_partition.go
user: login: nodece
body: I see, because the batchIndex type is `int32`, not point.

I think you should remove this change. When the batch index is -1, do not change the value to 0.

created_at: 2025-03-09T15:54:39Z
author_association: MEMBER
diff_hunk: @@ -2256,12 +2306,14 @@ func convertToMessageID(id *pb.MessageIdData) *trackingMessageID {
 
 	msgID := &trackingMessageID{
 		messageID: &messageID{
-			ledgerID: int64(*id.LedgerId),
-			entryID:  int64(*id.EntryId),
+			ledgerID:  int64(id.GetLedgerId()),
+			entryID:   int64(id.GetEntryId()),
+			batchIdx:  id.GetBatchIndex(),
+			batchSize: id.GetBatchSize(),
 		},
 	}
-	if id.BatchIndex != nil {
-		msgID.batchIdx = *id.BatchIndex
+	if msgID.batchIdx == -1 {
+		msgID.batchIdx = 0
path: pulsar/consumer_partition.go
user: login: shibd
body: If remove this, when seek to a messageId(no batch), let say`(3141:1:0:0)`, When we compare, there will be unexpected behavior because the `lastBrokerMessageId` we get from the broker and convert to : `3141:1:-1:0`.

What are your main concerns? Currently, this method is only used for comparison in `hasNext`.

I think we need to maintain consistency because now the non-batch message ID returned by the producer and consumer is `batchIndex=0`.
created_at: 2025-03-10T00:05:21Z
author_association: MEMBER
diff_hunk: @@ -2256,12 +2306,14 @@ func convertToMessageID(id *pb.MessageIdData) *trackingMessageID {
 
 	msgID := &trackingMessageID{
 		messageID: &messageID{
-			ledgerID: int64(*id.LedgerId),
-			entryID:  int64(*id.EntryId),
+			ledgerID:  int64(id.GetLedgerId()),
+			entryID:   int64(id.GetEntryId()),
+			batchIdx:  id.GetBatchIndex(),
+			batchSize: id.GetBatchSize(),
 		},
 	}
-	if id.BatchIndex != nil {
-		msgID.batchIdx = *id.BatchIndex
+	if msgID.batchIdx == -1 {
+		msgID.batchIdx = 0
path: pulsar/consumer_partition.go
user: login: nodece
body: > What are your main concerns? Currently, this method is only used for comparison in `hasNext`.

I need to consider the IDs of batch messages and non-batch messages.

If `3141:1:0` is a non-batch message, the correct message ID should be `3141:1:0:-1`, not `3141:1:0:0`.

The batch index is greater than -1, which means this message is from the batch messages.
created_at: 2025-03-10T04:40:32Z
author_association: MEMBER
diff_hunk: @@ -2256,12 +2306,14 @@ func convertToMessageID(id *pb.MessageIdData) *trackingMessageID {
 
 	msgID := &trackingMessageID{
 		messageID: &messageID{
-			ledgerID: int64(*id.LedgerId),
-			entryID:  int64(*id.EntryId),
+			ledgerID:  int64(id.GetLedgerId()),
+			entryID:   int64(id.GetEntryId()),
+			batchIdx:  id.GetBatchIndex(),
+			batchSize: id.GetBatchSize(),
 		},
 	}
-	if id.BatchIndex != nil {
-		msgID.batchIdx = *id.BatchIndex
+	if msgID.batchIdx == -1 {
+		msgID.batchIdx = 0
path: pulsar/consumer_partition.go
user: login: nodece
body: The PR didn't change the original behavior.

BTW, I think this is a worthy point of improvement.
created_at: 2025-03-10T06:46:06Z
author_association: MEMBER

PR comments:
user: login: shibd
created_at: 2025-03-06T12:26:22Z
body: > This behavior change is beyond the scope of this PR.

@BewareMyPower I changed the title. I think my intention was to clarify the issue with `hasNext` or `receive` after `seek`.

I believe email discussions are pointless because this is a behavior fix. If we need to inform users, we should clearly explain it in the release notes for the next release.



author_association: MEMBER
user: login: shibd
created_at: 2025-03-06T14:17:51Z
body: Start discuss on thread: https://lists.apache.org/thread/knxsq637pyhltthn77859mpxb5wcxglj
author_association: MEMBER
user: login: geniusjoe
created_at: 2025-03-07T11:20:17Z
body: > If any user depends on the existing `seek` behavior (inclusive by default), he/she might treat it as a "message lost" issue.
> 
> This behavior change is beyond the scope of this PR. I also think it needs a discussion at the mail list. We also need to document the change and tell existing users to set `startMessageInclusive` with true to keep the old behavior. But it's still a breaking change because if `startMessageInclusive` is true, the behavior of subscribing also changes.
> 
> Therefore, my opinion is to keep the existing seek behavior not changed for now. The "fix" could be delayed to next release.

I'm not very sure what do you mean "message lost"? 
Does previous behavior continue to consume messages regardless of `seek()` happened, or just continue to consume messages in client cache and then jump to consume the position which `seek()` defined? Would you mind explaining more specific which part of messages are lost compared to the current fixes.
author_association: CONTRIBUTOR
user: login: BewareMyPower
created_at: 2025-03-07T14:47:35Z
body: @geniusjoe

```go
func TestReaderSeek2(t *testing.T) {
        client, err := NewClient(ClientOptions{
                URL: lookupURL,
        })
        assert.Nil(t, err)
        defer client.Close()

        topicName := newTopicName()
        ctx := context.Background()
        producer, err := client.CreateProducer(ProducerOptions{
                Topic: topicName,
        })
        ids := make([]MessageID, 0)
        assert.Nil(t, err)
        for i := 0; i < 10; i++ {
                id, err := producer.Send(ctx, &ProducerMessage{
                        Payload: []byte(fmt.Sprintf("msg-%d", i)),
                })
                assert.Nil(t, err)
                ids = append(ids, id)
        }
        reader, err := client.CreateReader(ReaderOptions{
                Topic:          topicName,
                StartMessageID: EarliestMessageID(),
        })
        // Seek to id[3] so that reader can continue reading from "msg-3"
        reader.Seek(ids[3])
        msg, err := reader.Next(ctx)
        assert.Nil(t, err)
        t.Log(string(msg.Payload()))
}
```

Users seek to `ids[3]` (the position of "msg-3") to expect consuming from "msg-3". However, after the change, "msg-3" is unexpectedly skipped, the test code above will print "msg-4".
author_association: CONTRIBUTOR