Repository: open-telemetry/opentelemetry-collector-contrib (2026-04-07T08:04:48Z)

Issue:
Make control flow configurable
### Component(s)

receiver/googlecloudpubsub

### Is your feature request related to a problem? Please describe.

Currently users don't have a way to control parameters of the streaming pull loop and the serverside Pub/Sub parameters for the stream.

### Describe the solution you'd like

Add flow control configuration, giving advanced users more control over the parameters of the streaming pull control loop.

### Describe alternatives you've considered

Only rebuilding and custom code is an alternative.

### Additional context

I'm codeowner and the PR is ready

### Tip

<sub>[React](https://github.blog/news-insights/product-news/add-reactions-to-pull-requests-issues-and-comments/) with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding `+1` or `me too`, to help us triage it. Learn more [here](https://opentelemetry.io/community/end-user/issue-participation/).</sub>


Discussion:
Pinging code owners:
- receiver/googlecloudpubsub: @alexvanboxel

 See [Adding Labels via Comments](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#adding-labels-via-comments) if you do not have permissions to add labels yourself.

Draft https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/44806
Only missing documentation that will be added today.
This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see [Adding Labels via Comments](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#adding-labels-via-comments), or if you are unsure of which component this issue relates to, please ping `@open-telemetry/collector-contrib-triagers`. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:
- receiver/googlecloudpubsubreceiver: @alexvanboxel

See [Adding Labels via Comments](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#adding-labels-via-comments) if you do not have permissions to add labels yourself.

Is not stale... I will make time for it again.
Hi @alexvanboxel , thanks for keeping this moving! I wanted to add my support for this feature. We are currently hitting limitations with the default streaming pull parameters in our environment and having these settings configurable would be a huge help for our production stability



Patch:
diff --git a/.chloggen/googlepubsub_receiver_flow_control_config.yaml b/.chloggen/googlepubsub_receiver_flow_control_config.yaml
new file mode 100644
index 0000000000000..e5a0e7196e8e4
--- /dev/null
+++ b/.chloggen/googlepubsub_receiver_flow_control_config.yaml
@@ -0,0 +1,29 @@
+# Use this changelog template to create an entry for release notes.
+
+# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
+change_type: enhancement
+
+# The name of the component, or a single word describing the area of concern, (e.g. receiver/filelog)
+component: receiver/googlecloudpubsub
+
+# A brief description of the change.  Surround your text with quotes ("") if it needs to start with a backtick (`).
+note: Add flow control configuration
+
+# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
+issues: [44804]
+
+# (Optional) One or more lines of additional information to render under the primary note.
+# These lines will be padded with 2 spaces and then inserted directly into the document.
+# Use pipe (|) for multiline entries.
+subtext: |
+  Add flow control configuration, giving advanced users more control over the parameters of the streaming 
+  pull control loop.
+
+# If your change doesn't affect end users or the exported elements of any package,
+# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
+# Optional: The change log or logs in which this entry should be included.
+# e.g. '[user]' or '[user, api]'
+# Include 'user' if the change is relevant to end users.
+# Include 'api' if there is a change to a library API.
+# Default: '[user]'
+change_logs: [user]
\ No newline at end of file
diff --git a/receiver/googlecloudpubsubreceiver/README.md b/receiver/googlecloudpubsubreceiver/README.md
index 26b95c208e9a3..a2bd527cefab2 100644
--- a/receiver/googlecloudpubsubreceiver/README.md
+++ b/receiver/googlecloudpubsubreceiver/README.md
@@ -35,6 +35,8 @@ The following configuration options are supported:
 * `ignore_encoding_error` (Optional): Ignore errors when the configured encoder fails to decoding a PubSub messages.
   It's advised to set this to `true` when using a custom encoder, and use `receiver.googlecloudpubsub.encoding_error`
   metric to monitor the number of errors. Ignoring the error will cause the receiver to drop the message.
+* `flow_control` (Optional): Allows to fine tune the flow control settings for the subscription stream. See
+  [Flow control](#flow-control) for more details.
 
 ```yaml
 receivers:
@@ -99,6 +101,34 @@ service account and give the subscription `Pub/Sub Subscriber` permission.
 
 The subscription should also be of delivery type `Pull`.
 
+### Flow control
+
+The flow control allows to fine tune the flow control settings for the subscription stream.
+
+* `trigger_ack_batch_duration` (Optional): Part of the control loop, the time between each message
+  acknowledge batch.
+* `stream_ack_deadline` (Optional): The ack deadline to use for the stream. The minimum deadline you can specify
+  is 10 seconds. The maximum deadline you can specify is 600 seconds (10 minutes).
+* `max_outstanding_messages` (Optional): Flow control settings for the maximum number of outstanding messages. When
+  there are max_outstanding_messages currently sent to the streaming pull client that have not yet been acked or 
+  nacked, the server stops sending more messages. The sending of messages resumes once the number of outstanding
+  messages is less than this value. If the value is <= 0, there is no limit to the number of outstanding messages.
+* `max_outstanding_bytes` (Optional): Flow control settings for the maximum number of outstanding bytes. When there
+  are max_outstanding_bytes or more worth of messages currently sent to the streaming pull client that have not yet
+  been acked or nacked, the server will stop sending more messages. The sending of messages resumes once the number
+  of outstanding bytes is less than this value. If the value is <= 0, there is no limit to the number of outstanding
+  bytes.
+
+```yaml
+flow_control:
+  trigger_ack_batch_duration: 1s
+  stream_ack_deadline: 60s
+  max_outstanding_messages: 1000000
+  max_outstanding_bytes: 1000000
+```
+
+More details can be found at [Pub/Sub gRPC StreamingPullRequest](https://docs.cloud.google.com/pubsub/docs/reference/rpc/google.pubsub.v1#streamingpullrequest)
+
 ### Filtering
 
 When the messages on the subscription are accompanied by the correct attributes and you only need a specific
diff --git a/receiver/googlecloudpubsubreceiver/config.go b/receiver/googlecloudpubsubreceiver/config.go
index 2dde9cc9156e6..d1030dbd34a8a 100644
--- a/receiver/googlecloudpubsubreceiver/config.go
+++ b/receiver/googlecloudpubsubreceiver/config.go
@@ -6,8 +6,11 @@ package googlecloudpubsubreceiver // import "github.com/open-telemetry/opentelem
 import (
 	"fmt"
 	"regexp"
+	"time"
 
 	"go.opentelemetry.io/collector/exporter/exporterhelper"
+
+	"github.com/open-telemetry/opentelemetry-collector-contrib/receiver/googlecloudpubsubreceiver/internal"
 )
 
 var subscriptionMatcher = regexp.MustCompile(`projects/[a-z][a-z0-9\-]*(:[a-z0-9\-]+)?/subscriptions/`)
@@ -36,6 +39,33 @@ type Config struct {
 
 	// The client id that will be used by Pubsub to make load balancing decisions
 	ClientID string `mapstructure:"client_id"`
+
+	FlowControlConfig FlowControlConfig `mapstructure:"flow_control"`
+}
+
+// FlowControlConfig defines the flow control configuration for the receiver. This is used to
+// tune the internal flow control implementation, along with the Pub/Sub flow control settings
+// documented at https://cloud.google.com/pubsub/docs/flow-control and
+// https://cloud.google.com/pubsub/docs/reference/rpc/google.pubsub.v1#streamingpullrequest
+type FlowControlConfig struct {
+	// The maximum duration the acknowledgement loop waits before sending the acknowledgements.
+	TriggerAckBatchDuration time.Duration `mapstructure:"trigger_ack_batch_duration"`
+
+	// The ack deadline to use for the Pub/Sub stream.
+	StreamAckDeadline time.Duration `mapstructure:"stream_ack_deadline"`
+	// Pub/Sub flow control settings for the maximum number of outstanding messages.
+	MaxOutstandingMessages int64 `mapstructure:"max_outstanding_messages"`
+	// Pub/Sub flow control settings for the maximum number of outstanding bytes.
+	MaxOutstandingBytes int64 `mapstructure:"max_outstanding_bytes"`
+}
+
+func (fcc *FlowControlConfig) getInternalConfig() *internal.FlowControlConfig {
+	return &internal.FlowControlConfig{
+		TriggerAckBatchDuration: fcc.TriggerAckBatchDuration,
+		StreamAckDeadline:       fcc.StreamAckDeadline,
+		MaxOutstandingMessages:  fcc.MaxOutstandingMessages,
+		MaxOutstandingBytes:     fcc.MaxOutstandingBytes,
+	}
 }
 
 func (config *Config) validate() error {
diff --git a/receiver/googlecloudpubsubreceiver/config.schema.yaml b/receiver/googlecloudpubsubreceiver/config.schema.yaml
index 43a70d25ea816..c81a0f3df9a47 100644
--- a/receiver/googlecloudpubsubreceiver/config.schema.yaml
+++ b/receiver/googlecloudpubsubreceiver/config.schema.yaml
@@ -1,3 +1,24 @@
+$defs:
+  flow_control_config:
+    description: FlowControlConfig defines the flow control configuration for the receiver. This is used to tune the internal flow control implementation, along with the Pub/Sub flow control settings documented at https://cloud.google.com/pubsub/docs/flow-control and https://cloud.google.com/pubsub/docs/reference/rpc/google.pubsub.v1#streamingpullrequest
+    type: object
+    properties:
+      max_outstanding_bytes:
+        description: Pub/Sub flow control settings for the maximum number of outstanding bytes.
+        type: integer
+        x-customType: int64
+      max_outstanding_messages:
+        description: Pub/Sub flow control settings for the maximum number of outstanding messages.
+        type: integer
+        x-customType: int64
+      stream_ack_deadline:
+        description: The ack deadline to use for the Pub/Sub stream.
+        type: string
+        format: duration
+      trigger_ack_batch_duration:
+        description: The maximum duration the acknowledgement loop waits before sending the acknowledgements.
+        type: string
+        format: duration
 type: object
 properties:
   client_id:
@@ -12,6 +33,8 @@ properties:
   endpoint:
     description: Override of the Pubsub Endpoint, leave empty for the default endpoint
     type: string
+  flow_control:
+    $ref: flow_control_config
   ignore_encoding_error:
     description: Ignore errors when the configured encoder fails to decoding a PubSub messages
     type: boolean
diff --git a/receiver/googlecloudpubsubreceiver/factory.go b/receiver/googlecloudpubsubreceiver/factory.go
index 47b3e3b69967a..28c0fba5330f8 100644
--- a/receiver/googlecloudpubsubreceiver/factory.go
+++ b/receiver/googlecloudpubsubreceiver/factory.go
@@ -12,6 +12,7 @@ import (
 	"go.opentelemetry.io/collector/receiver"
 	"go.opentelemetry.io/collector/receiver/receiverhelper"
 
+	"github.com/open-telemetry/opentelemetry-collector-contrib/receiver/googlecloudpubsubreceiver/internal"
 	"github.com/open-telemetry/opentelemetry-collector-contrib/receiver/googlecloudpubsubreceiver/internal/metadata"
 )
 
@@ -38,7 +39,15 @@ type pubsubReceiverFactory struct {
 }
 
 func (*pubsubReceiverFactory) CreateDefaultConfig() component.Config {
-	return &Config{}
+	fcc := internal.NewDefaultFlowControlConfig()
+	return &Config{
+		FlowControlConfig: FlowControlConfig{
+			TriggerAckBatchDuration: fcc.TriggerAckBatchDuration,
+			StreamAckDeadline:       fcc.StreamAckDeadline,
+			MaxOutstandingMessages:  fcc.MaxOutstandingMessages,
+			MaxOutstandingBytes:     fcc.MaxOutstandingBytes,
+		},
+	}
 }
 
 func (factory *pubsubReceiverFactory) ensureReceiver(settings receiver.Settings, config component.Config) (*pubsubReceiver, error) {
diff --git a/receiver/googlecloudpubsubreceiver/internal/flow_control_config.go b/receiver/googlecloudpubsubreceiver/internal/flow_control_config.go
new file mode 100644
index 0000000000000..396e688d0c8b1
--- /dev/null
+++ b/receiver/googlecloudpubsubreceiver/internal/flow_control_config.go
@@ -0,0 +1,26 @@
+// Copyright The OpenTelemetry Authors
+// SPDX-License-Identifier: Apache-2.0
+
+package internal // import "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/googlecloudpubsubreceiver/internal"
+
+import "time"
+
+type FlowControlConfig struct {
+	// The maximum duration the acknowledgement loop waits before sending the acknowledgements.
+	TriggerAckBatchDuration time.Duration
+	// The ack deadline to use for the Pub/Sub stream.
+	StreamAckDeadline time.Duration
+	// Pub/Sub flow control settings for the maximum number of outstanding messages.
+	MaxOutstandingMessages int64
+	// Pub/Sub flow control settings for the maximum number of outstanding bytes.
+	MaxOutstandingBytes int64
+}
+
+func NewDefaultFlowControlConfig() *FlowControlConfig {
+	return &FlowControlConfig{
+		TriggerAckBatchDuration: 10 * time.Second,
+		StreamAckDeadline:       60 * time.Second,
+		MaxOutstandingMessages:  0,
+		MaxOutstandingBytes:     0,
+	}
+}
diff --git a/receiver/googlecloudpubsubreceiver/internal/handler.go b/receiver/googlecloudpubsubreceiver/internal/handler.go
index a395210368409..b2afed2db03ac 100644
--- a/receiver/googlecloudpubsubreceiver/internal/handler.go
+++ b/receiver/googlecloudpubsubreceiver/internal/handler.go
@@ -42,8 +42,9 @@ type StreamHandler struct {
 	handlerWaitGroup sync.WaitGroup
 	settings         receiver.Settings
 	telemetryBuilder *metadata.TelemetryBuilder
-	// time that acknowledge loop waits before acknowledging messages
-	ackBatchWait time.Duration
+
+	// flow control settings, like max durations, counts and triggers
+	flowControlConfig *FlowControlConfig
 
 	isRunning    atomic.Bool
 	retryAttempt int
@@ -63,16 +64,20 @@ func NewHandler(
 	client SubscriberClient,
 	clientID string,
 	subscription string,
+	config *FlowControlConfig,
 	callback func(ctx context.Context, message *pubsubpb.ReceivedMessage) error,
 ) (*StreamHandler, error) {
+	if config == nil {
+		config = NewDefaultFlowControlConfig()
+	}
 	handler := StreamHandler{
-		settings:         settings,
-		telemetryBuilder: telemetryBuilder,
-		client:           client,
-		clientID:         clientID,
-		subscription:     subscription,
-		pushMessage:      callback,
-		ackBatchWait:     10 * time.Second,
+		settings:          settings,
+		telemetryBuilder:  telemetryBuilder,
+		client:            client,
+		clientID:          clientID,
+		subscription:      subscription,
+		pushMessage:       callback,
+		flowControlConfig: config,
 	}
 	return &handler, handler.initStream(ctx)
 }
@@ -89,8 +94,10 @@ func (handler *StreamHandler) initStream(ctx context.Context) error {
 
 	request := pubsubpb.StreamingPullRequest{
 		Subscription:             handler.subscription,
-		StreamAckDeadlineSeconds: 60,
+		StreamAckDeadlineSeconds: int32(handler.flowControlConfig.StreamAckDeadline.Seconds()),
 		ClientId:                 handler.clientID,
+		MaxOutstandingMessages:   handler.flowControlConfig.MaxOutstandingMessages,
+		MaxOutstandingBytes:      handler.flowControlConfig.MaxOutstandingBytes,
 		AckIds:                   handler.acks,
 	}
 	if err := handler.stream.Send(&request); err != nil {
@@ -183,7 +190,7 @@ func (handler *StreamHandler) acknowledgeMessages() error {
 // a stream got restarted, the messages that still needed to be acknowledged are acknowledged at the start
 // of the new stream, so we don't need to start with an acknowledgeMessages.
 func (handler *StreamHandler) requestStream(ctx context.Context, cancel context.CancelFunc) {
-	timer := time.NewTimer(handler.ackBatchWait)
+	timer := time.NewTimer(handler.flowControlConfig.TriggerAckBatchDuration)
 	for {
 		select {
 		case <-ctx.Done():
@@ -203,7 +210,7 @@ func (handler *StreamHandler) requestStream(ctx context.Context, cancel context.
 		if errors.Is(ctx.Err(), context.Canceled) {
 			break
 		}
-		timer.Reset(handler.ackBatchWait)
+		timer.Reset(handler.flowControlConfig.TriggerAckBatchDuration)
 	}
 	timer.Stop()
 	cancel()
diff --git a/receiver/googlecloudpubsubreceiver/receiver.go b/receiver/googlecloudpubsubreceiver/receiver.go
index 88a344b186553..dbb58b727d779 100644
--- a/receiver/googlecloudpubsubreceiver/receiver.go
+++ b/receiver/googlecloudpubsubreceiver/receiver.go
@@ -355,6 +355,7 @@ func (receiver *pubsubReceiver) createMultiplexingReceiverHandler(ctx context.Co
 		receiver.client,
 		receiver.config.ClientID,
 		receiver.config.Subscription,
+		receiver.config.FlowControlConfig.getInternalConfig(),
 		func(ctx context.Context, message *pubsubpb.ReceivedMessage) error {
 			payload := message.Message.Data
 			encoding, compression := receiver.detectEncoding(message.Message.Attributes)
@@ -417,6 +418,7 @@ func (receiver *pubsubReceiver) createReceiverHandler(ctx context.Context) error
 		receiver.client,
 		receiver.config.ClientID,
 		receiver.config.Subscription,
+		receiver.config.FlowControlConfig.getInternalConfig(),
 		handlerFn)
 	if err != nil {
 		return err


Test patch:
diff --git a/receiver/googlecloudpubsubreceiver/config_test.go b/receiver/googlecloudpubsubreceiver/config_test.go
index 23e29f3d0fd0a..ce7043379faea 100644
--- a/receiver/googlecloudpubsubreceiver/config_test.go
+++ b/receiver/googlecloudpubsubreceiver/config_test.go
@@ -30,8 +30,13 @@ func TestLoadConfig(t *testing.T) {
 		expectedErr error
 	}{
 		{
-			id:       component.NewIDWithName(metadata.Type, ""),
-			expected: &Config{},
+			id: component.NewIDWithName(metadata.Type, ""),
+			expected: &Config{
+				FlowControlConfig: FlowControlConfig{
+					TriggerAckBatchDuration: 10 * time.Second,
+					StreamAckDeadline:       60 * time.Second,
+				},
+			},
 		},
 		{
 			id: component.NewIDWithName(metadata.Type, "customname"),
@@ -42,6 +47,10 @@ func TestLoadConfig(t *testing.T) {
 					Timeout: 20 * time.Second,
 				},
 				Subscription: "projects/my-project/subscriptions/otlp-subscription",
+				FlowControlConfig: FlowControlConfig{
+					TriggerAckBatchDuration: 10 * time.Second,
+					StreamAckDeadline:       60 * time.Second,
+				},
 			},
 		},
 	}
diff --git a/receiver/googlecloudpubsubreceiver/internal/handler_test.go b/receiver/googlecloudpubsubreceiver/internal/handler_test.go
index 71903a9ce2454..76ae375d31b12 100644
--- a/receiver/googlecloudpubsubreceiver/internal/handler_test.go
+++ b/receiver/googlecloudpubsubreceiver/internal/handler_test.go
@@ -22,17 +22,19 @@ import (
 	"github.com/open-telemetry/opentelemetry-collector-contrib/receiver/googlecloudpubsubreceiver/internal/metadata"
 )
 
-func TestCancelStream(t *testing.T) {
-	ctx, cancel := context.WithCancel(t.Context())
-	defer cancel()
-	srv := pstest.NewServer()
-	defer srv.Close()
+func createHandler(ctx context.Context, t *testing.T) (cleanupFn func(), srv *pstest.Server, handler *StreamHandler) {
+	srv = pstest.NewServer()
 
 	var copts []option.ClientOption
 	var dialOpts []grpc.DialOption
 	conn, err := grpc.NewClient(srv.Addr, append(dialOpts, grpc.WithTransportCredentials(insecure.NewCredentials()))...)
 	assert.NoError(t, err)
-	defer func() { assert.NoError(t, conn.Close()) }()
+
+	cleanupFn = func() {
+		assert.NoError(t, srv.Close())
+		assert.NoError(t, conn.Close())
+	}
+
 	copts = append(copts, option.WithGRPCConn(conn))
 	_, err = srv.GServer.CreateTopic(ctx, &pubsubpb.Topic{
 		Name: "projects/my-project/topics/otlp",
@@ -50,13 +52,20 @@ func TestCancelStream(t *testing.T) {
 
 	client, err := pubsub.NewSubscriptionAdminClient(ctx, copts...)
 	assert.NoError(t, err)
-
-	handler, err := NewHandler(ctx, settings, telemetryBuilder, client, "client-id", "projects/my-project/subscriptions/otlp",
-		func(context.Context, *pubsubpb.ReceivedMessage) error {
+	handler, err = NewHandler(ctx, settings, telemetryBuilder, client, "client-id", "projects/my-project/subscriptions/otlp",
+		nil, func(context.Context, *pubsubpb.ReceivedMessage) error {
 			return nil
 		})
-	handler.ackBatchWait = 10 * time.Millisecond
 	assert.NoError(t, err)
+	return cleanupFn, srv, handler
+}
+
+func TestCancelStream(t *testing.T) {
+	ctx, cancel := context.WithCancel(t.Context())
+	defer cancel()
+	cleanupFn, srv, handler := createHandler(ctx, t)
+	defer cleanupFn()
+
 	srv.Publish("projects/my-project/topics/otlp", []byte{}, map[string]string{
 		"ce-type":      "org.opentelemetry.otlp.traces.v1",
 		"content-type": "application/protobuf",