Repository: dotnetcore/CAP

Issue #1429: Duplicated messages in Kafka during delayed publishing large messges
Created: 2023-11-13T07:33:47Z
We are seeing incorrect behavior of publishing delayed messages in production so I replicated the issue on local with quite simple scenario.
using DotNetCore.CAP;
using KafkaTest;

WebApplicationBuilder builder = WebApplication.CreateBuilder(args);

builder.Services.AddTransient<ISubscriberService, SubscriberService>();

builder.Services.AddCap(options =>
{
    options.UseDashboard();
    options.UsePostgreSql("Server=localhost;User ID=postgres;Password=1234;Database=CapTest");
    options.UseKafka(kafkaOptions => { kafkaOptions.Servers = "localhost:9092"; });

    options.UseStorageLock = true;
});

WebApplication app = builder.Build();

app.MapGet("/test", async (ICapPublisher eventBus) =>
{
    for (int i = 0; i < 10000; i++)
    {
        await eventBus.PublishDelayAsync(TimeSpan.FromMinutes(5), "cap-test", Guid.NewGuid().ToString().ToLower());
    }
});

app.Run();
After startup of this project 10k messages should be published in Kafka in 5 minutes, but the result is that it`s always more than 10k (sometimes 13k, sometimes more). Below you can find the info when 22862 messages were produced.
GUID duplicated at least 3 times

All the headers are exactly the same


No messages have Retries > 0

22862 produced to the topic (confirmed by making separate consumer)

All of the below messages are duplicates

Packages used:
DotNetCore.CAP.Dashboard Version 7.2.2
DotNetCore.CAP.Kafka Version 7.2.2
DotNetCore.CAP.PostgreSql Version 7.2.2

Duplicate issue #1515: Scheduling freezes on large number of delayed messages
Created: 2024-04-12T01:36:09Z
Setup: CAP 8.1 RabbitMq + Postgres
Given: A large number of delayed messages (>20000 in my case)
When IDataStorage.ScheduleMessagesOfDelayedAsync fetches all messages from the DB and schedules them for processing, entire operation is executed inside a DB transaction and with FOR UPDATE SKIP LOCKED instruction that locks all selected rows for the duration of the transaction.
Next, the following happens:

Transaction (T1) starts in ScheduleMessagesOfDelayedAsync
Pending messages are read and enqueued via _dispatcher.EnqueueToScheduler() (IProcessor.Delayed.cs)
Messages are added to the _schedulerQueue priority queue for async processing by a background task. (Dispatcher.cs)
The BG task picks the next available message from the priority queue: _schedulerQueue.TryPeek(out _, out _nextSendTime)
The message is sent to transport (await _sender.SendAsync(_schedulerQueue.Dequeue());) and SetSuccessfulState is called (IMessageSender.Default.cs)
An update statement is executed in ChangeMessageStateAsync() (IDataStorage.PostgreSql.cs) against the DB while the T1 transaction is still not committed. Thus, the update statement times out and fails as the affected row is still locked by T1.
The while (_schedulerQueue.TryPeek(out _, out _nextSendTime)) loop freezes forever. (IDispatcher.Default.cs)

Closing comment (2024-04-12T02:16:58Z): It looks similar to #1429. It is caused by ScheduleMessagesOfDelayedAsync triggering again during the execution of T1 transaction. Please refer to the solution in #1429. If there are a large number of delayed messages at the same time, you may need to override the QueuedMessageFetchTime method.