Repository: conduitio-labs/conduit-connector-rabbitmq
Pull request #75: Implement publisher confirmations
Author: login: alarbada
Created: 2025-05-15T15:14:37Z | Merged: 2025-05-19T12:49:26Z

### Description

Implements publisher confirmations. Fixes #73
 
### Quick checks:

- [x] There is no other [pull request](https://github.com/conduitio-labs/conduit-connector-rabbitmq/pulls) for the same update/change.
- [ ] I have written unit tests.
- [x] I have made sure that the PR is of reasonable size and can be easily reviewed.


Linked issue #73: Feature: Publisher confirmations
### Feature description

Currently, destination is implemented to call `channel.PublishWithContext` which [does not guarantee](https://github.com/rabbitmq/amqp091-go/blob/566601c8e5debdfbc715aa1ef9ca8220beee997a/channel.go#L1511) delivery of the message, meaning messages will be lost if the broker restarts or network fails.

Destination should use Confirm mode and wait for producer acks.
Linked issue comments: user: login: lovromazgon
created_at: 2025-05-13T16:05:17Z
body: @alarbada that sounds quite important. Can you please have a look at this? Thanks!
author_association: CONTRIBUTOR
user: login: rauanmayemir
created_at: 2025-05-13T16:08:52Z
body: I could send a PR to solve it in a per-message sync manner, but sending a batch and then awaiting for all confirmations would be the best.
author_association: CONTRIBUTOR
user: login: alarbada
created_at: 2025-05-15T14:09:10Z
body: I will take a look at it, thanks for the ping @lovromazgon 

author_association: CONTRIBUTOR

Diff:
diff --git a/connector.yaml b/connector.yaml
index 5fe797a..47876f7 100644
--- a/connector.yaml
+++ b/connector.yaml
@@ -3,7 +3,7 @@ specification:
   name: rabbitmq
   summary: A RabbitMQ source and destination plugin for Conduit, written in Go.
   description: A RabbitMQ source and destination plugin for Conduit, written in Go.
-  version: v0.3.0
+  version: v0.4.0
   author: Meroxa, Inc.
   source:
     parameters:
diff --git a/destination.go b/destination.go
index 794ba0b..75fd62c 100644
--- a/destination.go
+++ b/destination.go
@@ -66,6 +66,13 @@ func (d *Destination) Open(ctx context.Context) (err error) {
 	}
 	sdk.Logger(ctx).Debug().Msgf("opened channel")
 
+	// We setup the channel in confirm mode, so that we ensure the "written"
+	// total of records that we return is correct.
+	noWait := false
+	if err := d.ch.Confirm(noWait); err != nil {
+		return fmt.Errorf("failed to set channel on confirm mode: %w", err)
+	}
+
 	err = d.declareQueue(ctx)
 	if err != nil {
 		return err
@@ -79,28 +86,65 @@ func (d *Destination) Open(ctx context.Context) (err error) {
 	return nil
 }
 
+func (d *Destination) createMessage(record opencdc.Record, msgID string) amqp091.Publishing {
+	return amqp091.Publishing{
+		ContentType:     d.config.Delivery.ContentType,
+		ContentEncoding: d.config.Delivery.ContentEncoding,
+		DeliveryMode:    d.config.Delivery.DeliveryMode,
+		Priority:        d.config.Delivery.Priority,
+		CorrelationId:   d.config.Delivery.CorrelationID,
+		ReplyTo:         d.config.Delivery.ReplyTo,
+
+		MessageId: msgID,
+		Type:      d.config.Delivery.MessageTypeName,
+		UserId:    d.config.Delivery.UserID,
+		AppId:     d.config.Delivery.AppID,
+		Headers:   metadataToHeaders(record.Metadata),
+		Body:      record.Bytes(),
+
+		Expiration: d.config.Delivery.Expiration,
+	}
+}
+
+func (d *Destination) createConfirmation(
+	ctx context.Context,
+	confirmation *amqp091.DeferredConfirmation,
+	msgID string,
+	routingKey string,
+	msg amqp091.Publishing,
+) func() error {
+	return func() error {
+		confirmed, err := confirmation.WaitContext(ctx)
+		if err != nil {
+			return fmt.Errorf("failed to wait for publish confirmation: %w", err)
+		}
+
+		if !confirmed {
+			return fmt.Errorf(
+				"message was not confirmed: id=%s type=%s userId=%s appId=%s",
+				msgID, msg.Type, msg.UserId, msg.AppId,
+			)
+		}
+
+		sdk.Logger(ctx).Trace().
+			Str("messageID", msgID).
+			Str("routingKey", routingKey).
+			Bool("mandatoryDelivery", d.config.Delivery.Mandatory).
+			Bool("immediateDelivery", d.config.Delivery.Immediate).
+			Msg("published message")
+
+		return nil
+	}
+}
+
 func (d *Destination) Write(ctx context.Context, records []opencdc.Record) (int, error) {
-	for i, record := range records {
+	confirmations := make([]func() error, 0, len(records))
+
+	for _, record := range records {
 		msgID := string(record.Position)
-		msg := amqp091.Publishing{
-			ContentType:     d.config.Delivery.ContentType,
-			ContentEncoding: d.config.Delivery.ContentEncoding,
-			DeliveryMode:    d.config.Delivery.DeliveryMode,
-			Priority:        d.config.Delivery.Priority,
-			CorrelationId:   d.config.Delivery.CorrelationID,
-			ReplyTo:         d.config.Delivery.ReplyTo,
-
-			MessageId: msgID,
-			Type:      d.config.Delivery.MessageTypeName,
-			UserId:    d.config.Delivery.UserID,
-			AppId:     d.config.Delivery.AppID,
-			Headers:   metadataToHeaders(record.Metadata),
-			Body:      record.Bytes(),
-
-			Expiration: d.config.Delivery.Expiration,
-		}
+		msg := d.createMessage(record, msgID)
 
-		if createdAt, err := record.Metadata.GetCreatedAt(); err != nil {
+		if createdAt, err := record.Metadata.GetCreatedAt(); err == nil {
 			msg.Timestamp = createdAt
 		}
 
@@ -109,12 +153,11 @@ func (d *Destination) Write(ctx context.Context, records []opencdc.Record) (int,
 			var err error
 			routingKey, err = d.getRoutingKey(record)
 			if err != nil {
-				return i, fmt.Errorf("could not get routingKey: %w", err)
+				return 0, fmt.Errorf("could not get routingKey: %w", err)
 			}
 		}
 
-		err := d.ch.PublishWithContext(
-			ctx,
+		confirmation, err := d.ch.PublishWithDeferredConfirm(
 			d.config.Exchange.Name,
 			routingKey,
 			d.config.Delivery.Mandatory,
@@ -122,15 +165,17 @@ func (d *Destination) Write(ctx context.Context, records []opencdc.Record) (int,
 			msg,
 		)
 		if err != nil {
-			return i, fmt.Errorf("failed to publish: %w", err)
+			return 0, fmt.Errorf("failed to publish: %w", err)
 		}
 
-		sdk.Logger(ctx).Trace().
-			Str("messageID", msgID).
-			Str("routingKey", routingKey).
-			Bool("mandatoryDelivery", d.config.Delivery.Mandatory).
-			Bool("immediateDelivery", d.config.Delivery.Immediate).
-			Msg("published message")
+		confirmationFn := d.createConfirmation(ctx, confirmation, msgID, routingKey, msg)
+		confirmations = append(confirmations, confirmationFn)
+	}
+
+	for i, confirmation := range confirmations {
+		if err := confirmation(); err != nil {
+			return i, err
+		}
 	}
 
 	return len(records), nil


Reviews:
user: login: raulb
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T08:01:49Z
user: login: raulb
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T11:20:53Z
user: login: raulb
body: LGTM
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T11:23:08Z
user: login: raulb
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T11:23:38Z
user: login: rauanmayemir
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T11:28:33Z
user: login: alarbada
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T11:58:41Z
user: login: alarbada
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T12:08:47Z
user: login: rauanmayemir
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T12:20:16Z
user: login: rauanmayemir
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T12:21:53Z

Review comments:
diff_hunk: @@ -66,6 +66,13 @@ func (d *Destination) Open(ctx context.Context) (err error) {
 	}
 	sdk.Logger(ctx).Debug().Msgf("opened channel")
 
+	// We setup the channel in confirm mode, so that we ensure the "written"
+	// total of records that we return is correct.
+	noWait := false
+	if err := d.ch.Confirm(noWait); err != nil {
path: destination.go
user: login: raulb
body: Why not sending `d.ch.Confirm(false)`
created_at: 2025-05-19T08:01:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -66,6 +66,13 @@ func (d *Destination) Open(ctx context.Context) (err error) {
 	}
 	sdk.Logger(ctx).Debug().Msgf("opened channel")
 
+	// We setup the channel in confirm mode, so that we ensure the "written"
+	// total of records that we return is correct.
+	noWait := false
+	if err := d.ch.Confirm(noWait); err != nil {
path: destination.go
user: login: raulb
body: Never mind. I think it reads better as it is since the API is not very intuitive with the double negation.
created_at: 2025-05-19T11:20:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -79,28 +86,65 @@ func (d *Destination) Open(ctx context.Context) (err error) {
 	return nil
 }
 
+func (d *Destination) createMessage(record opencdc.Record, msgID string) amqp091.Publishing {
+	return amqp091.Publishing{
+		ContentType:     d.config.Delivery.ContentType,
+		ContentEncoding: d.config.Delivery.ContentEncoding,
+		DeliveryMode:    d.config.Delivery.DeliveryMode,
+		Priority:        d.config.Delivery.Priority,
+		CorrelationId:   d.config.Delivery.CorrelationID,
+		ReplyTo:         d.config.Delivery.ReplyTo,
+
+		MessageId: msgID,
+		Type:      d.config.Delivery.MessageTypeName,
+		UserId:    d.config.Delivery.UserID,
+		AppId:     d.config.Delivery.AppID,
+		Headers:   metadataToHeaders(record.Metadata),
+		Body:      record.Bytes(),
+
+		Expiration: d.config.Delivery.Expiration,
+	}
+}
+
+func (d *Destination) createConfirmation(
+	ctx context.Context,
+	confirmation *amqp091.DeferredConfirmation,
+	msgID string,
+	routingKey string,
+	msg amqp091.Publishing,
+) func() error {
+	return func() error {
+		confirmed, err := confirmation.WaitContext(ctx)
+		if err != nil {
+			return fmt.Errorf("failed to wait for publish confirmation: %w", err)
+		}
+
+		if !confirmed {
+			return fmt.Errorf(
+				"message was not confirmed: id=%s type=%s userId=%s appId=%s",
+				msgID, msg.Type, msg.UserId, msg.AppId,
+			)
+		}
+
+		sdk.Logger(ctx).Trace().
+			Str("messageID", msgID).
+			Str("routingKey", routingKey).
+			Bool("mandatoryDelivery", d.config.Delivery.Mandatory).
+			Bool("immediateDelivery", d.config.Delivery.Immediate).
+			Msg("published message")
+
+		return nil
+	}
+}
+
 func (d *Destination) Write(ctx context.Context, records []opencdc.Record) (int, error) {
-	for i, record := range records {
+	confirmations := make([]func() error, 0, len(records))
+
+	for _, record := range records {
 		msgID := string(record.Position)
-		msg := amqp091.Publishing{
-			ContentType:     d.config.Delivery.ContentType,
-			ContentEncoding: d.config.Delivery.ContentEncoding,
-			DeliveryMode:    d.config.Delivery.DeliveryMode,
-			Priority:        d.config.Delivery.Priority,
-			CorrelationId:   d.config.Delivery.CorrelationID,
-			ReplyTo:         d.config.Delivery.ReplyTo,
-
-			MessageId: msgID,
-			Type:      d.config.Delivery.MessageTypeName,
-			UserId:    d.config.Delivery.UserID,
-			AppId:     d.config.Delivery.AppID,
-			Headers:   metadataToHeaders(record.Metadata),
-			Body:      record.Bytes(),
-
-			Expiration: d.config.Delivery.Expiration,
-		}
+		msg := d.createMessage(record, msgID)
 
-		if createdAt, err := record.Metadata.GetCreatedAt(); err != nil {
+		if createdAt, err := record.Metadata.GetCreatedAt(); err == nil {
path: destination.go
user: login: raulb
body: good catch
created_at: 2025-05-19T11:21:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,28 +153,29 @@ func (d *Destination) Write(ctx context.Context, records []opencdc.Record) (int,
 			var err error
 			routingKey, err = d.getRoutingKey(record)
 			if err != nil {
-				return i, fmt.Errorf("could not get routingKey: %w", err)
+				return 0, fmt.Errorf("could not get routingKey: %w", err)
 			}
 		}
 
-		err := d.ch.PublishWithContext(
-			ctx,
+		confirmation, err := d.ch.PublishWithDeferredConfirm(
 			d.config.Exchange.Name,
 			routingKey,
 			d.config.Delivery.Mandatory,
 			d.config.Delivery.Immediate,
 			msg,
 		)
 		if err != nil {
-			return i, fmt.Errorf("failed to publish: %w", err)
+			return 0, fmt.Errorf("failed to publish: %w", err)
 		}
 
-		sdk.Logger(ctx).Trace().
-			Str("messageID", msgID).
-			Str("routingKey", routingKey).
-			Bool("mandatoryDelivery", d.config.Delivery.Mandatory).
-			Bool("immediateDelivery", d.config.Delivery.Immediate).
-			Msg("published message")
+		confirmationFn := d.createConfirmation(ctx, confirmation, msgID, routingKey, msg)
+		confirmations = append(confirmations, confirmationFn)
+	}
+
+	for i, confirmation := range confirmations {
path: destination.go
user: login: rauanmayemir
body: I don't understand what's happening here. Are we technically blocking and waiting on each message?
created_at: 2025-05-19T11:28:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,28 +153,29 @@ func (d *Destination) Write(ctx context.Context, records []opencdc.Record) (int,
 			var err error
 			routingKey, err = d.getRoutingKey(record)
 			if err != nil {
-				return i, fmt.Errorf("could not get routingKey: %w", err)
+				return 0, fmt.Errorf("could not get routingKey: %w", err)
 			}
 		}
 
-		err := d.ch.PublishWithContext(
-			ctx,
+		confirmation, err := d.ch.PublishWithDeferredConfirm(
 			d.config.Exchange.Name,
 			routingKey,
 			d.config.Delivery.Mandatory,
 			d.config.Delivery.Immediate,
 			msg,
 		)
 		if err != nil {
-			return i, fmt.Errorf("failed to publish: %w", err)
+			return 0, fmt.Errorf("failed to publish: %w", err)
 		}
 
-		sdk.Logger(ctx).Trace().
-			Str("messageID", msgID).
-			Str("routingKey", routingKey).
-			Bool("mandatoryDelivery", d.config.Delivery.Mandatory).
-			Bool("immediateDelivery", d.config.Delivery.Immediate).
-			Msg("published message")
+		confirmationFn := d.createConfirmation(ctx, confirmation, msgID, routingKey, msg)
+		confirmations = append(confirmations, confirmationFn)
+	}
+
+	for i, confirmation := range confirmations {
path: destination.go
user: login: alarbada
body: Npe not really @rauanmayemir . You can see in the for loop that first we create all the confirmations. The messages are published and sent by `PublishWithDeferredConfirm`, as you can see [here](https://github.com/rabbitmq/amqp091-go/blob/566601c8e5debdfbc715aa1ef9ca8220beee997a/channel.go#L1549C1-L1576C1).
When we wait for each confirmation in the second loop, all the following confirmations have been submitted, so we are actually waiting for all of them instead of individually.

Does this make sense?
created_at: 2025-05-19T11:58:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,28 +153,29 @@ func (d *Destination) Write(ctx context.Context, records []opencdc.Record) (int,
 			var err error
 			routingKey, err = d.getRoutingKey(record)
 			if err != nil {
-				return i, fmt.Errorf("could not get routingKey: %w", err)
+				return 0, fmt.Errorf("could not get routingKey: %w", err)
 			}
 		}
 
-		err := d.ch.PublishWithContext(
-			ctx,
+		confirmation, err := d.ch.PublishWithDeferredConfirm(
 			d.config.Exchange.Name,
 			routingKey,
 			d.config.Delivery.Mandatory,
 			d.config.Delivery.Immediate,
 			msg,
 		)
 		if err != nil {
-			return i, fmt.Errorf("failed to publish: %w", err)
+			return 0, fmt.Errorf("failed to publish: %w", err)
 		}
 
-		sdk.Logger(ctx).Trace().
-			Str("messageID", msgID).
-			Str("routingKey", routingKey).
-			Bool("mandatoryDelivery", d.config.Delivery.Mandatory).
-			Bool("immediateDelivery", d.config.Delivery.Immediate).
-			Msg("published message")
+		confirmationFn := d.createConfirmation(ctx, confirmation, msgID, routingKey, msg)
+		confirmations = append(confirmations, confirmationFn)
+	}
+
+	for i, confirmation := range confirmations {
path: destination.go
user: login: alarbada
body: ![image](https://github.com/user-attachments/assets/39d6904b-9234-499d-a1ae-c7fb2b3f0fef)

If I'm not mistaken, basically this.
created_at: 2025-05-19T12:08:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,28 +153,29 @@ func (d *Destination) Write(ctx context.Context, records []opencdc.Record) (int,
 			var err error
 			routingKey, err = d.getRoutingKey(record)
 			if err != nil {
-				return i, fmt.Errorf("could not get routingKey: %w", err)
+				return 0, fmt.Errorf("could not get routingKey: %w", err)
 			}
 		}
 
-		err := d.ch.PublishWithContext(
-			ctx,
+		confirmation, err := d.ch.PublishWithDeferredConfirm(
 			d.config.Exchange.Name,
 			routingKey,
 			d.config.Delivery.Mandatory,
 			d.config.Delivery.Immediate,
 			msg,
 		)
 		if err != nil {
-			return i, fmt.Errorf("failed to publish: %w", err)
+			return 0, fmt.Errorf("failed to publish: %w", err)
 		}
 
-		sdk.Logger(ctx).Trace().
-			Str("messageID", msgID).
-			Str("routingKey", routingKey).
-			Bool("mandatoryDelivery", d.config.Delivery.Mandatory).
-			Bool("immediateDelivery", d.config.Delivery.Immediate).
-			Msg("published message")
+		confirmationFn := d.createConfirmation(ctx, confirmation, msgID, routingKey, msg)
+		confirmations = append(confirmations, confirmationFn)
+	}
+
+	for i, confirmation := range confirmations {
path: destination.go
user: login: rauanmayemir
body: I kinda understand that actual blocking happens only on the couple initial messages and then the rest will likely be fulfilled.
created_at: 2025-05-19T12:20:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,28 +153,29 @@ func (d *Destination) Write(ctx context.Context, records []opencdc.Record) (int,
 			var err error
 			routingKey, err = d.getRoutingKey(record)
 			if err != nil {
-				return i, fmt.Errorf("could not get routingKey: %w", err)
+				return 0, fmt.Errorf("could not get routingKey: %w", err)
 			}
 		}
 
-		err := d.ch.PublishWithContext(
-			ctx,
+		confirmation, err := d.ch.PublishWithDeferredConfirm(
 			d.config.Exchange.Name,
 			routingKey,
 			d.config.Delivery.Mandatory,
 			d.config.Delivery.Immediate,
 			msg,
 		)
 		if err != nil {
-			return i, fmt.Errorf("failed to publish: %w", err)
+			return 0, fmt.Errorf("failed to publish: %w", err)
 		}
 
-		sdk.Logger(ctx).Trace().
-			Str("messageID", msgID).
-			Str("routingKey", routingKey).
-			Bool("mandatoryDelivery", d.config.Delivery.Mandatory).
-			Bool("immediateDelivery", d.config.Delivery.Immediate).
-			Msg("published message")
+		confirmationFn := d.createConfirmation(ctx, confirmation, msgID, routingKey, msg)
+		confirmations = append(confirmations, confirmationFn)
+	}
+
+	for i, confirmation := range confirmations {
path: destination.go
user: login: rauanmayemir
body: Yeah, indeed, we’re on the same page. Still seems unorthodox to me. 😄 
created_at: 2025-05-19T12:21:53Z
author_association: CONTRIBUTOR

PR comments:
user: login: alarbada
created_at: 2025-05-15T15:25:02Z
body: So, the fix is ready, but I have some doubts on how to implement the batching proposed by @rauanmayemir:

>  sending a batch and then awaiting for all confirmations would be the best.

I could first publish all messages, collect all the `deferredConfirmation`s and then wait in another loop for each of them. The issue I have though with that is what do we return as `written` when one record fails to publish. Do we just return i? or 0 instead, as we don't know now whether the message has been successfully written or not?

author_association: CONTRIBUTOR
user: login: alarbada
created_at: 2025-05-15T15:27:56Z
body: It's been a while since my last queue connector. For batched writes I might want to just look how do we do it in other connectors such as kafka.
author_association: CONTRIBUTOR
user: login: rauanmayemir
created_at: 2025-05-15T18:07:49Z
body: Yeah, I wish Write would optionally allow returning an array of indexes and errors.
author_association: CONTRIBUTOR
user: login: hariso
created_at: 2025-05-16T09:51:31Z
body: Conduit's destination connector return `n, err`, where `n` is the number of consecutive records from the beginning of the batch that were successfully written. For example, if the destination is given 10 records and first 5 were written, then 2 weren't, then 3 were, the connector should return `5, "error for the 6th or other records"`.

The reason has to do with ordering (and ordering is important because an event might depend on previous events). We can't acknowledge the 8th record in the source, because the source will then understand that as records 1-7 were successfully written, and when the source starts again, it will just skip 6 and 7. 


author_association: CONTRIBUTOR
user: login: alarbada
created_at: 2025-05-19T12:59:28Z
body: @rauanmayemir just released [v0.4.0](https://github.com/conduitio-labs/conduit-connector-rabbitmq/releases/tag/v0.4.0), please let us know if you encounter any more problems.
author_association: CONTRIBUTOR
user: login: rauanmayemir
created_at: 2025-05-20T09:05:34Z
body: Thanks, deployed in test env, will be observing.
author_association: CONTRIBUTOR