Repository: coze-dev/coze-loop (2026-03-18T06:54:03Z)

Issue:
[Bug] dataset_version.snapshot_progress stops updating after the first batch in RunSnapshotItemJob
### 📋 CheckList

- [x] I have searched existing issues to avoid duplicates
- [x] I am using a recently maintained version of Coze Loop
- [x] I have provided all required information
- [x] I understand this is a bug report and not a feature request
- [x] I have submitted this report in English (otherwise it will not be processed)

### 🐛 Bug Description

I discovered a bug in the RunSnapshotItemJob method within item_snapshot.go. When a dataset snapshot is processed in multiple batches (e.g., 100 batches), **the snapshot_progress field in the dataset_version table is only updated after the first batch.**

For all subsequent batches (from batch 2 onwards), the snapshot_progress in the database remains stuck at the value of the first batch.

This behavior breaks the "resume from breakpoint" mechanism. The initProgressCtx method relies on snapshot_progress to determine where to resume a job. Since the database holds an outdated progress value (from the first batch), **if a job fails mid-way (e.g., at batch 50) and restarts, it will incorrectly resume from the position recorded after batch 1, effectively rendering the checkpoint logic useless.**

### 🔄 Steps to Reproduce

Prepare Data: Create or use a dataset with enough items to trigger the snapshot job to run in multiple batches (ensure the data volume exceeds the single-batch limit so that at least 2 batches are generated).

Trigger Action: Publish the dataset version to start the RunSnapshotItemJob.

Observation: Monitor the dataset_version table in the database while the job is running.

Result: You will consistently see that snapshot_progress is updated only after the first batch completes. It fails to update for all subsequent batches, regardless of how many batches follow.

### ✅ Expected Behavior

The snapshot_progress field in the dataset_version table should be updated after every batch is successfully processed, ensuring the checkpoint is always up-to-date.

### ❌ Actual Behavior

When a dataset snapshot is processed in multiple batches (e.g., 100 batches), the snapshot_progress field in the dataset_version table is only updated after the first batch.

### 🚨 Severity

Low - Cosmetic issue or minor inconvenience

### 🔧 Component

Other

### 💻 Environment

_No response_

### 🔧 Go Environment

_No response_

### 📋 Logs

```shell

```

### 📝 Additional Context

**I have already located the issue and plan to submit a PR to fix it. Please assign this to me.**


Discussion:
Thanks for the feedback, and we'll check it out. cc @Celoria 
Hi, I've been looking into this and traced the root cause.

**Proposed approach:** Fix commitProgress() in backend/modules/data/domain/dataset/service/item_snapshot.go. Currently, snapshot_progress (cursor) is only persisted to DB on the first batch (SnapshotStatusUnstarted→InProgress) and the last batch (→Completed). Middle batches update cursor in memory only, breaking resume-from-breakpoint. Fix: remove the SnapshotStatusUnstarted guard so progress is persisted after every batch, or add an explicit intermediate progress update path. Add tests verifying cursor persistence across multiple batches.

I'll open a draft PR shortly. Happy to adjust based on your preference.



Patch:
diff --git a/backend/modules/data/domain/dataset/service/item_snapshot.go b/backend/modules/data/domain/dataset/service/item_snapshot.go
index 2ed2a2257..d32ae4ada 100644
--- a/backend/modules/data/domain/dataset/service/item_snapshot.go
+++ b/backend/modules/data/domain/dataset/service/item_snapshot.go
@@ -123,10 +123,8 @@ func (s *DatasetServiceImpl) commitProgress(ctx context.Context, processCtx *sna
 		}
 		return nil
 	}
-	if processCtx.version.SnapshotStatus == entity.SnapshotStatusUnstarted {
-		if err := s.commitToInProgress(ctx, processCtx); err != nil {
-			return err
-		}
+	if err := s.commitToInProgress(ctx, processCtx); err != nil {
+		return err
 	}
 	return nil
 }


Test patch:
diff --git a/backend/modules/data/domain/dataset/service/item_snapshot_test.go b/backend/modules/data/domain/dataset/service/item_snapshot_test.go
index ac5729881..d2a8c4643 100644
--- a/backend/modules/data/domain/dataset/service/item_snapshot_test.go
+++ b/backend/modules/data/domain/dataset/service/item_snapshot_test.go
@@ -189,6 +189,68 @@ func TestDatasetServiceImpl_commitToInProgress(t *testing.T) {
 	}
 }
 
+func TestDatasetServiceImpl_commitProgress_MultiBatch(t *testing.T) {
+	ctrl := gomock.NewController(t)
+	defer ctrl.Finish()
+
+	mockRepo := mock_repo.NewMockIDatasetAPI(ctrl)
+	service := &DatasetServiceImpl{
+		repo: mockRepo,
+	}
+
+	t.Run("PatchVersion called for every batch with hasMore=true", func(t *testing.T) {
+		processCtx := &snapshotContext{
+			spaceID:   1,
+			versionID: 1,
+			version: &entity.DatasetVersion{
+				ID:               1,
+				SpaceID:          1,
+				SnapshotStatus:   entity.SnapshotStatusUnstarted,
+				SnapshotProgress: &entity.SnapshotProgress{},
+				UpdateVersion:    0,
+			},
+		}
+
+		// Batch 1: hasMore=true, status transitions from Unstarted to InProgress
+		mockRepo.EXPECT().PatchVersion(gomock.Any(), gomock.Any(), gomock.Any()).Return(nil)
+		err := service.commitProgress(context.Background(), processCtx, true)
+		if err != nil {
+			t.Fatalf("batch 1: commitProgress() error = %v", err)
+		}
+		if processCtx.version.SnapshotStatus != entity.SnapshotStatusInProgress {
+			t.Errorf("batch 1: expected status %v, got %v", entity.SnapshotStatusInProgress, processCtx.version.SnapshotStatus)
+		}
+		if processCtx.version.UpdateVersion != 1 {
+			t.Errorf("batch 1: expected UpdateVersion 1, got %v", processCtx.version.UpdateVersion)
+		}
+
+		// Batch 2: hasMore=true, status already InProgress — must still call PatchVersion
+		processCtx.version.SnapshotProgress.Cursor = "cursor_batch2"
+		mockRepo.EXPECT().PatchVersion(gomock.Any(), gomock.Any(), gomock.Any()).Return(nil)
+		err = service.commitProgress(context.Background(), processCtx, true)
+		if err != nil {
+			t.Fatalf("batch 2: commitProgress() error = %v", err)
+		}
+		if processCtx.version.SnapshotStatus != entity.SnapshotStatusInProgress {
+			t.Errorf("batch 2: expected status %v, got %v", entity.SnapshotStatusInProgress, processCtx.version.SnapshotStatus)
+		}
+		if processCtx.version.UpdateVersion != 2 {
+			t.Errorf("batch 2: expected UpdateVersion 2, got %v", processCtx.version.UpdateVersion)
+		}
+
+		// Batch 3: hasMore=true, third batch — must still call PatchVersion
+		processCtx.version.SnapshotProgress.Cursor = "cursor_batch3"
+		mockRepo.EXPECT().PatchVersion(gomock.Any(), gomock.Any(), gomock.Any()).Return(nil)
+		err = service.commitProgress(context.Background(), processCtx, true)
+		if err != nil {
+			t.Fatalf("batch 3: commitProgress() error = %v", err)
+		}
+		if processCtx.version.UpdateVersion != 3 {
+			t.Errorf("batch 3: expected UpdateVersion 3, got %v", processCtx.version.UpdateVersion)
+		}
+	})
+}
+
 func TestDatasetServiceImpl_commitToFailed(t *testing.T) {
 	ctrl := gomock.NewController(t)
 	defer ctrl.Finish()